VM14K: First Vietnamese Medical Benchmark

Dat Nguyen; Duc Nguyen; Kien Tran; Long Nguyen; Minh Dang; Minh Tran; Quan H. Nguyen; Thai Dao; Thong Nguyen

REVIEW 3 major objections 6 minor 38 references

VM14K: First Vietnamese Medical Benchmark

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces VM14K, the first Vietnamese medical benchmark: 14,000 expert-verified multiple-choice questions across 34 specialties and four difficulty levels, built by a scalable, open-source pipeline.

desk verdict A genuinely new Vietnamese medical QA resource, but the 'expert-verified' claim needs numbers, not adjectives. read the letter →

arxiv 2506.01305 v2 pith:BHCDCMV4 submitted 2025-06-02 cs.CL

Thong Nguyen , Duc Nguyen , Minh Dang , Thai Dao , Long Nguyen , Quan H. Nguyen , Dat Nguyen , Kien Tran

show 1 more author

Minh Tran

This is my paper

classification cs.CL

keywords Vietnamesemedicalbenchmarklargelanguagemodelsquestionansweringmultilingualevaluationexpertannotationdifficultylevelsconstructiondatacurationpipeline

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a trustworthy medical benchmark for Vietnamese can be built from locally verified educational sources, rather than by translating English tests. It introduces VM14K, 14,000 expert-verified multiple-choice questions spanning 34 medical specialties and four difficulty levels, and argues this is the first benchmark of its kind for Vietnamese. The authors also demonstrate the benchmark's use by evaluating a range of general and medical-specialized language models, finding that the best performer, DeepSeek R1, still leaves clear room for improvement and that model rankings on VM14K differ from rankings on English medical benchmarks. This matters because, without native-language benchmarks, medical AI for Vietnamese speakers cannot be measured fairly, and the open-source pipeline offers a route to build similar benchmarks for other underserved languages.

What carries the argument

The load-bearing mechanism is the data-construction pipeline and the benchmark schema it feeds. A Python crawler pulls raw material from Vietnamese medical textbooks, exams, online quizzes, and clinical records, with a PostgreSQL hash table tracking duplicates; Spark transforms the text to JSON; PDF and Word extraction tools handle documents; then GPT-4o and Gemini 2.0 Flash convert raw question text into a structured field set and assign each item a difficulty level and one or more of 34 medical topics. Deduplication layers hashing with Levenshtein-distance clustering to absorb accent variants and minor character differences. Verification brings together three sources of truth—the original source answer, foundation-model answers, and medical-expert answers—and the annotation queue is ordered by mutual disagreement and difficulty, so the easiest disputed items are checked first. The four-tier difficulty rubric and the 34-specialty taxonomy are the scaffold that makes scores interpretable by breadth and depth, and the whole pipeline is released to be reusable in other languages.

What would settle it

Run an independent, double-blind re-annotation of a random sample of VM14K questions by Vietnamese medical specialists and check agreement with the released answers and difficulty labels; if agreement is at or below the noise level of the original three-source verification, the benchmark cannot serve as ground truth for leaderboard comparisons.

Watch

Extended reading notes

Core claim

The central claim is that VM14K is the first Vietnamese medical question benchmark and that it offers a valid, reusable measure of medical knowledge for language models serving Vietnamese speakers. To establish this, the authors collected about 100,000 raw questions from Vietnamese medical textbooks, exams, online quizzes, and clinical records; used GPT-4o and Gemini 2.0 Flash to extract structured questions and assign difficulty and topic labels; removed duplicates with hashing and Levenshtein-distance clustering; and had medical experts verify answers, prioritizing questions where the source answer and the model answer disagreed. From the verified pool they sampled 14,000 questions balanced across 34 specialties and four difficulty levels, released as a 4,000-question sample set, a 10,000-question public set, and a 2,000-question private leaderboard set. Evaluations of general and medical-specialized language models show DeepSeek R1 with the highest accuracy, and the authors read the divergence from English-benchmark rankings as evidence that VM14K measures Vietnamese-specific medical language and knowledge, not just recycled English material.

Load-bearing premise

The benchmark's validity as a gold standard rests on the expert verification step being consistent and correct, but the paper does not report the number of experts, their agreement rates, or how disagreements with source and model answers were resolved.

Editorial extensions

If this is right

Vietnamese-language medical AI can be evaluated directly on native clinical language rather than through translated English tests, avoiding the terminology and cultural mismatches that translation introduces.
Model developers get a concrete target from the public and private splits: the best current model scores about 78% pass@1 on VM14K, so a model approaching 90% or above would be a clear advance for Vietnamese medical applications.
The ensemble results imply that a single pass understates model capability, so future evaluations should consider shuffled-choice voting as a standard protocol.
The difficulty levels let buyers and researchers see where a model breaks: strong recall on Easy items with a sharp drop on Challenging and Hard items identifies a model that cannot perform clinical reasoning in Vietnamese.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the ranking inversion against English benchmarks implies that a hospital or product team selecting a model for Vietnamese patients using English medical scores could pick a substantially weaker model; VM14K should be part of procurement decisions.
Editorial inference: publishing inter-annotator agreement and per-question confidence would turn VM14K from a one-shot resource into a reusable measurement instrument; without those numbers, difficulty-label effects could partly reflect annotation noise.
Editorial inference: the 100,000 raw questions that were not expert-verified are a natural training resource; using them for fine-tuning while holding the 14,000 verified questions for evaluation would be a direct test of the pipeline's value beyond benchmarking.
Editorial inference: the private 2,000-question set is the only guard against benchmark contamination; its integrity depends on the authors never releasing it, and the community should treat any public appearance of those questions as a contamination event.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is the first Vietnamese medical QA benchmark, and that alone makes it worth a serious look. The dataset (14k MCQs, 34 specialties, four difficulty tiers) is new, built from Vietnamese exams and textbooks rather than translated English items, and they release a private test split for leaderboard evaluation. The pipeline is open-source and could transfer to other languages. Credit where due: the paper identifies a real gap and makes a concrete attempt to fill it.

The soft spots are the usual ones for resource papers, but they matter more here because the central claim is 'expert-verified.' Section 3.2.2 describes a three-source verification (source, LLM, human), but never says how many experts looked at each question, what their qualifications were, how disagreements were adjudicated, or whether every one of the 14,000 questions actually passed through human review. The line about prioritizing easy questions where source and LLM answers differ is worrying: it reads as though items with source–LLM agreement may not have been human-checked at all. If that's the case, the gold standard is partly LLM-confirmed, not expert-verified. Also, the abstract says 'medical experts' while the introduction says 'medical students' — not the same thing, and the paper should be consistent.

The difficulty and topic labels are assigned by GPT-4o and Gemini, and Section 4 then analyzes model performance by those very labels. That's a mild circularity, not a fatal one, but it should be acknowledged as a limitation. The evaluation tables report accuracy without error bars or significance tests; a few percentage points difference between models may not be real.

The citation pattern looks fine, and the related-work coverage of non-English medical benchmarks is reasonable. No suspicious references.

Bottom line: a useful new resource with a load-bearing documentation gap. The right fix is not to throw the dataset away but to make the verification process transparent and public. I'd send this to review with a request for major revision — the potential value for Vietnamese medical NLP is real.

Referee Report

3 major / 6 minor

Summary. VM14K is a proposed benchmark for evaluating large language models on Vietnamese medical knowledge. The paper describes a three-stage construction pipeline: crawling and extracting Vietnamese medical exams, textbooks, and public medical records; using GPT-4o and Gemini 2.0 Flash to parse raw text into structured multiple-choice fields (question, options, correct answer, difficulty, medical topic); and a verification pipeline that compares source answers, LLM answers, and human annotation. The final resource contains 14,000 multiple-choice questions across 34 medical specialties and four difficulty levels, split into a 4k sample public set, a 10k full public set, and a 2k private leaderboard set. The paper reports pass@k and ensemble accuracies for several general-purpose and medical-specialized models, together with a cost-performance analysis and breakdowns by difficulty and medical topic.

Significance. If the verification claims are adequately substantiated, VM14K would fill a genuine gap: it is a native Vietnamese medical benchmark rather than a translated English one, it is substantially larger than most existing non-English medical QA resources, and its three-way split with a private test set supports controlled evaluation. The open-source release of the pipeline and the stated intention to support scalability to other languages are also strengths. However, the current manuscript does not provide enough evidence for the 'expert-verified' label that appears in the abstract, and the difficulty and topic labels used in the analyses are generated by LLMs without reported validation. The resource's value as a gold standard therefore remains to be demonstrated.

major comments (3)

[Section 3.2.2] The verification process is not described in enough detail to support the central claim that all 14,000 questions are expert-verified. The text says that the first questions to be verified by human experts are the easiest ones where the original source answers and the foundation model answers differ, which implies that questions with source-LLM agreement may have bypassed human review. The manuscript does not state how many human annotators participated, whether they were medical students or licensed physicians, whether each question received at least one independent human review, how disagreements were adjudicated, or what the inter-annotator agreement was. These details are load-bearing because every model ranking in Section 4 assumes that the gold answers are correct. Please report the full verification protocol and per-item human review counts, or revise the 'expert-verified' characterization accordingly.
[Sections 1, 5] There is an unresolved inconsistency in who created the labels. The Introduction says the benchmark was 'eventually annotated by medical students,' while the Abstract and Conclusion call it 'expert-verified' and 'expert-annotated.' Medical students under qualified supervision may be a perfectly acceptable annotation workforce, but the qualification and oversight process must be stated explicitly. Please unify the terminology and describe the expertise required of the annotators, because this directly affects the benchmark's credibility as a gold standard.
[Section 3.2.2, Tables 2-4] The difficulty levels and medical topics are assigned by GPT-4o and Gemini 2.0 Flash, as described in Section 3.2.2 and Table 1, yet Sections 4.1 and 4.2 interpret model performance by these labels (Figure 3, Table 4). Without any validation of these automatic labels on a human-annotated subset, the difficulty and topic analyses should be explicitly framed as analyses of LLM-assigned labels rather than as intrinsic properties of the benchmark. Additionally, Table 1 defines the extracted field correctOption as 'The correct answer according to GPT-4o'; please clarify how this field relates to the final gold answer after the verification step.

minor comments (6)

[Abstract] The abstract contains a typo and grammatical issue in 'communities,therefore help ensuring the quality'; please revise the sentence.
[Section 3.2.2, Table 3] Model names are inconsistent: 'Geimini Flash' appears in Section 3.2.2 and 'Gemin 2.0 Flash' appears in Table 3; use the same spelling throughout.
[Table 2] Table 2 lists 'Deepseek-R1[?]' with a missing citation placeholder; the reference list has no DeepSeek entry.
[Section 4] The metric description says pass@k with k=3 and k=1, but several models in Table 2 have no pass@3 entry; please clarify which models were evaluated with pass@3 and the reason for the omission.
[Figure 4] The cost-performance analysis does not state the pricing snapshot, the API used, or whether reasoning-token overhead for models like DeepSeek-R1 and o3-mini is included; please specify these details for comparability.
[Appendix A] Table 5 lists 34 medical categories but the paper does not report per-category question counts; a distribution table would help readers assess the 'balanced representation' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark is built from external medical sources and evaluated models are scored against independent gold answers.

full rationale

The central construction chain is self-contained against the evaluated models. VM14K is assembled from external medical exams, textbooks, and clinical records (Section 3.2.1), with answers sourced from the original materials and checked by a three-source verification step that includes human experts (Section 3.2.2). Model performance in Section 4 is measured against these pre-existing answers, not against outputs of the evaluated models, so no score is forced by construction. The only overlapping element is that GPT-4o and Gemini 2.0 Flash were used to assign difficulty and topic labels (Section 3.2.2, Table 1), and the same two models are later reported by these strata (Tables 2–3, Figure 3). That is a stratification choice, not a reduction: the labels do not enter the accuracy computation, and overall scores are independent of the label source. The paper's 'first Vietnamese medical benchmark' claim is an external, falsifiable historical claim, not derived from its own definitions. The expert-verification description is qualitative and under-specified, which is a transparency and validity concern rather than a circularity concern; unless un-reviewed LLM-generated answers were admitted as gold, the benchmark's correctness does not reduce to the models being evaluated. No self-citation chain or imported uniqueness theorem is load-bearing; cited prior benchmarks are used only for comparison. Hence the derivation chain does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark construction relies on the integrity of external source materials and on the accuracy of LLM-based extraction and labeling, which are not independently verified in the paper. No numerical parameters are fitted to data, and no new theoretical entities are introduced.

assumptions (3)

domain assumption The source materials (medical exams, textbooks, clinical records) are accurate and representative of Vietnamese medical knowledge.
The entire benchmark is built on these sources, and any systemic error in the source would propagate to the questions and answers. This assumption is invoked throughout Section 3.
domain assumption GPT-4o and Gemini 2.0 Flash extract and label questions without systematic errors.
These models are used for question extraction, difficulty labeling, and topic labeling (Section 3.2.2). If they introduce errors, the benchmark's labels and question content could be corrupted.
domain assumption The expert annotators are sufficiently competent and consistent.
The paper relies on medical experts to verify answers, but it does not report their qualifications in detail or any measures of agreement (Section 3.2.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of VM14K: First Vietnamese Medical Benchmark." pith.science (2026). https://pith.science/paper/BHCDCMV4

@misc{pith2026250601305,
  author       = {Pith},
  title        = {Pith review of: VM14K: First Vietnamese Medical Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BHCDCMV4}},
  note         = {Machine review of arXiv:2506.01305}
}

read the original abstract

Medical benchmarks are indispensable for evaluating the capabilities of language models in healthcare for non-English-speaking communities,therefore help ensuring the quality of real-life applications. However, not every community has sufficient resources and standardized methods to effectively build and design such benchmark, and available non-English medical data is normally fragmented and difficult to verify. We developed an approach to tackle this problem and applied it to create the first Vietnamese medical question benchmark, featuring 14,000 multiple-choice questions across 34 medical specialties. Our benchmark was constructed using various verifiable sources, including carefully curated medical exams and clinical records, and eventually annotated by medical experts. The benchmark includes four difficulty levels, ranging from foundational biological knowledge commonly found in textbooks to typical clinical case studies that require advanced reasoning. This design enables assessment of both the breadth and depth of language models' medical understanding in the target language thanks to its extensive coverage and in-depth subject-specific expertise. We release the benchmark in three parts: a sample public set (4k questions), a full public set (10k questions), and a private set (2k questions) used for leaderboard evaluation. Each set contains all medical subfields and difficulty levels. Our approach is scalable to other languages, and we open-source our data construction pipeline to support the development of future multilingual benchmarks in the medical domain.

Figures

Figures reproduced from arXiv: 2506.01305 by the authors.

**Figure 2.** Duplication removal process Extract content We used docx2python 2 to extract text from Microsoft Words and pymupdf 3 to extract text from PDF. The raw text of questions (mostly multiple-choice questions and their options) is saved in the same output file. Extract questions We utilized OpenAI’s GPT-4o[21] and Gemini 2.0 Flash[9] to extract questions to structured output 1. Since our full raw dataset included 100,000 … view at source ↗

**Figure 3.** Performance of general models on the benchmark by difficulty levels. Challenging and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Cost versus performance analysis Hard is less severe for larger models, except for HuatuoGPT-o1-8B: despite being 8B, it performs at the top, likely due to specialized training or data focused on medical reasoning. However, the rest of the 8B models are consistently th…

**Figure 5.** Figure 5: Difficulty level distribution of VM14K {question} Choose the correct option from these answers: A. {optionA} B. {optionB} ... Only response with 1 character Example: A [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Prompt used in experiment 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 24 canonical work pages

[1]

Hewett, Mojan Javaheripi, Piero Kauffmann, James R

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

2024
[2]

The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

work page 2025
[3]

Mohammed Al-Garadi, Tushar Mungle, Abdulaziz Ahmed, Abeed Sarker, Zhuqi Miao, and Michael E. Matheny. Large language models in healthcare, 2025

work page 2025
[4]

Claude 3.5 sonnet, 2024

Anthropic. Claude 3.5 sonnet, 2024

2024
[5]

Overview of the medical question answering task at trec 2017 liveqa

Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. Overview of the medical question answering task at trec 2017 liveqa. InTREC 2017, 2017

work page 2017
[6]

Huatuogpt-o1, towards medical complex reasoning with llms, 2024

Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms, 2024

2024
[7]

Meditron- 70b: Scaling medical pretraining for large language models, 2023

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron- 70b...

work page 2023
[8]

The llama 3 herd of models, 07 2024

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and Zhiwei Zhao. The llama 3 herd of models, 07 2024

work page 2024

Show all 38 references

[9]

Introducing gemini 2.0: our new ai model for the agentic era, 2024

Google. Introducing gemini 2.0: our new ai model for the agentic era, 2024

2024
[10]

Impact of translation on biomedical information extraction from real-life clinical notes, 2023

Christel Gérardin, Yuhan Xiong, Perceval Wajsbürt, Fabrice Carrat, and Xavier Tannier. Impact of translation on biomedical information extraction from real-life clinical notes, 2023

2023
[11]

Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

2021
[12]

Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

2021
[13]

What disease does this patient have? a large-scale open domain question answering dataset from medical exams.arXiv preprint arXiv:2009.13081, 2020

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.arXiv preprint arXiv:2009.13081, 2020

2009 arXiv
[14]

Pubmedqa: A dataset for biomedical research question answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

2019
[15]

Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries, 2023

Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries, 2023

2023
[16]

Evaluating gpt-4 and chatgpt on japanese medical licensing examinations, 2023

Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations, 2023

2023
[17]

Kormedmcqa: Multi-choice question answering benchmark for korean healthcare professional licensing examinations, 2024

Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. Kormedmcqa: Multi-choice question answering benchmark for korean healthcare professional licensing examinations, 2024. 10

2024
[18]

Andre Lamurias, Diana Sousa, and Francisco M. Couto. Generating biomedical question answering corpora from q&a forums.IEEE Access, 8:161042–161051, 2020

2020
[19]

Benchmarking large language models on cmexam – a comprehensive chinese medical exam dataset, 2023

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. Benchmarking large language models on cmexam – a comprehensive chinese medical exam dataset, 2023

2023
[20]

A survey on medical large language models: Technology, application, trustworthiness, and future directions, 2024

Lei Liu, Xiaoyan Yang, Junchi Lei, Yue Shen, Jian Wang, Peng Wei, Zhixuan Chu, Zhan Qin, and Kui Ren. A survey on medical large language models: Technology, application, trustworthiness, and future directions, 2024

2024
[21]

Hello GPT-4o, 2024

OpenAI. Hello GPT-4o, 2024

2024
[22]

Openai o3-mini pushing the frontier of cost-effective reasoning, 2025

OpenAI. Openai o3-mini pushing the frontier of cost-effective reasoning, 2025

2025
[23]

Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors,Proceedings of the Conferenc...

2022
[24]

Assessing the research landscape and clinical utility of large language models: a scoping review.BMC Medical Informatics and Decision Making, 24(72), 2024

Yu Jin Park, Aishwarya Pillai, Jing Deng, and et al. Assessing the research landscape and clinical utility of large language models: a scoping review.BMC Medical Informatics and Decision Making, 24(72), 2024

2024
[25]

Opportunities and challenges for large language models in primary health care.J Prim Care Community Health, 16:21501319241312571, Jan-Dec 2025

Hai Qin and Yi Tong. Opportunities and challenges for large language models in primary health care.J Prim Care Community Health, 16:21501319241312571, Jan-Dec 2025

2025
[26]

Towards building multilingual language model for medicine, 2024

Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine, 2024

2024
[27]

Holistic evaluation of large language models for medical applications

Nigam Shah, Mike Pfeffer, and Percy Liang. Holistic evaluation of large language models for medical applications. Oct 2024. Accessed: 2025-05-15

2024
[28]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Ag...

2022
[29]

Gemma 3 technical report, 03 2025

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouil- lard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon...

2025
[30]

ChiMed: A Chinese medical corpus for question answering

Yuanhe Tian, Weicheng Ma, Fei Xia, and Yan Song. ChiMed: A Chinese medical corpus for question answering. In Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii, editors,Proceedings of the 18th BioNLP Workshop and Shared Task, pages 250–260, Floren...

2019
[31]

Medqa benchmark, 2025

ValsAI. Medqa benchmark, 2025

2025
[32]

Bitterman, Ling Pan, Ching-Yu Cheng, James Zou, and Dianbo Liu

Xiaoye Wang, Nicole Xi Zhang, Hongyu He, Trang Nguyen, Kun-Hsing Yu, Hao Deng, Cynthia Brandt, Danielle S. Bitterman, Ling Pan, Ching-Yu Cheng, James Zou, and Dianbo Liu. Safety challenges of ai in medicine in the era of large language models, 2025

2025
[33]

Qwen3 technical report, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, 11 Jianxin Yang, ...

2025
[34]

Tcmd: A traditional chinese medicine qa dataset for evaluating large language models, 2024

Ping Yu, Kaitao Song, Fengchen He, Ming Chen, and Jianfeng Lu. Tcmd: A traditional chinese medicine qa dataset for evaluating large language models, 2024

2024
[35]

A survey of datasets in medicine for large language models.Intelligence & Robotics, 4(4), 2024

Deshiwei Zhang, Xiaojuan Xue, Peng Gao, Zhijuan Jin, Menghan Hu, Yue Wu, and Xiayang Ying. A survey of datasets in medicine for large language models.Intelligence & Robotics, 4(4), 2024

2024
[36]

Ultramedical: Building specialized generalists in biomedicine, 2024

Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, and Bowen Zhou. Ultramedical: Building specialized generalists in biomedicine, 2024

2024
[37]

Zhang, X

S. Zhang, X. Zhang, H. Wang, L. Guo, and S. Liu. Multi-scale attentive interaction networks for chinese medical question answer selection.IEEE Access, 6:74061–74071, 2018

2018
[38]

Critical care studies using large language models based on electronic healthcare records: A technical note.J Intensive Med, 5(2):137–150, 2024

Z Zhang and H Ni. Critical care studies using large language models based on electronic healthcare records: A technical note.J Intensive Med, 5(2):137–150, 2024. 12 A Detail of dataset Table 5: Medical categories and descriptions Category Description Allergy and Immunology Dia...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Hewett, Mojan Javaheripi, Piero Kauffmann, James R

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

2024

[2] [2]

The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

work page 2025

[3] [3]

Mohammed Al-Garadi, Tushar Mungle, Abdulaziz Ahmed, Abeed Sarker, Zhuqi Miao, and Michael E. Matheny. Large language models in healthcare, 2025

work page 2025

[4] [4]

Claude 3.5 sonnet, 2024

Anthropic. Claude 3.5 sonnet, 2024

2024

[5] [5]

Overview of the medical question answering task at trec 2017 liveqa

Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. Overview of the medical question answering task at trec 2017 liveqa. InTREC 2017, 2017

work page 2017

[6] [6]

Huatuogpt-o1, towards medical complex reasoning with llms, 2024

Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms, 2024

2024

[7] [7]

Meditron- 70b: Scaling medical pretraining for large language models, 2023

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron- 70b...

work page 2023

[8] [8]

The llama 3 herd of models, 07 2024

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and Zhiwei Zhao. The llama 3 herd of models, 07 2024

work page 2024