REVIEW 5 major objections 6 minor 82 references
ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Commercial LLMs score well on HPB exams but miss most concurrent diagnoses in real inpatient cases, and medical LLMs fail to generalize to HPB diseases.
desk verdict A serious HPB benchmark worth refereeing, but the headline degradation numbers are probably overstated by non-inferable reference diagnoses and the lack of a human baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
ClinBench-HPB is the central object: 3,535 multiple-choice questions (2,000 Chinese, 1,535 English) drawn from public exam datasets plus synthesized items, and 337 open-ended clinical cases from journals, case-sharing websites, and a collaborating hospital, all mapped to the 33 main and 465 subcategories of ICD-10 HPB diseases. The scoring mechanism is a two-stage LLM-as-a-judge, with DeepSeek-V3-0324 as the strict evaluator deciding whether a prediction includes each reference disease on six dimensions—anatomical specificity, pathological mechanism, etiological origin, temporal characteristics, test identification, and treatment strategy—and Claude-3.7-Sonnet as the check model to catch false negatives from rigid matching. Performance is reported as disease-level recall and patient-level recall, where patient-level recall requires naming every concurrent diagnosis; four distinct prompts and circular option shifts are used to smooth over input sensitivity.
What would settle it
If re-scoring the 50 hospital cases using only each patient's primary diagnosis, dropping incidental comorbidities such as pulmonary bullae or coronary atherosclerosis, erased most of the gap between MCQ accuracy and patient-level recall, the claim that LLMs fail at real HPB diagnosis would be seriously weakened. A second check would be to have experienced clinicians independently list all diagnoses from the same case text and compare their recall to the LLMs' recall.
Extended reading notes
Core claim
The paper's central claim is that LLM performance on HPB medicine is substantially overestimated by closed-ended medical-exam benchmarks. In the benchmark's MCQ portion, leading commercial models reach average accuracies around 0.84, which would suggest strong mastery of HPB knowledge; in the open-ended diagnosis portion, disease-level recall is moderate but patient-level recall collapses, and on the 50 hospital-collected inpatient cases even the best commercial model (DeepSeekV3-1226) captures the full spectrum of concurrent diagnoses in only 22% of patients. The paper further argues that medical LLMs fine-tuned on exam-style data degrade on HPB cases outside their training distribution, with all four HuatuoGPT-o1 models underperforming their base models on website and hospital subsets, while reasoning-enhanced LLMs show the most transfer to diagnosis as they scale. The benchmark's design is the evidence: cases are organized by ICD-10 categories, each reference diagnosis is checked against the model's prediction along six clinical dimensions, and results are averaged over four prompt variants.
Load-bearing premise
The benchmark's central result depends on the reference diagnosis list for each case being complete, correct, and derivable from the text a model sees, so that a model failing to name any listed comorbidity is counted as a diagnostic error.
Editorial extensions
If this is right
- Medical-exam accuracy should not be read as clinical diagnostic competence, because the same models that reach about 0.84 average accuracy on MCQs drop to low patient-level recall on inpatient HPB cases.
- Hospital-style HPB cases with multiple comorbidities are the hard setting where current LLMs miss diagnoses that can change management, so deployment in that setting needs additional safeguards.
- Fine-tuning medical LLMs on exam-style data does not guarantee transfer, and the HuatuoGPT-o1 models underperforming their general-purpose bases on website and hospital cases shows a concrete out-of-distribution risk.
- Reasoning-enhanced LLMs trained on math and code transfer some capability to diagnosis as they scale, suggesting reasoning training as a plausible path when medical data are scarce.
- The released benchmark gives a standardized, ICD-10-organized test for future HPB LLMs, covering all 33 main categories and 465 subcategories of HPB disease.
Reading between the lines
- A test worth running is whether experienced clinicians, given the same case text, also fail to list every comorbidity; if they do, the patient-level recall gap reflects task ambiguity more than model deficiency.
- The two-model judge could be audited by physicians on all 337 cases rather than sampled subsets; the reported 97.5–97.7% agreement may not hold uniformly for rare comorbidities.
- Because the hospital cases are text-only, adding the original imaging could change results in either direction, since the CT and MRI impressions already contain some disease names.
- The same benchmark construction could be applied to other organ systems to see whether the MCQ-to-clinical gap is a general property of LLMs or specific to HPB complexity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ClinBench-HPB, a benchmark for hepato-pancreato-biliary (HPB) diseases containing 3,535 multiple-choice questions (CN-QA: 2,000 Chinese; EN-QA: 1,535 English) and 337 open-ended clinical cases (120 from journals, 167 from case-sharing websites, 50 from a collaborating hospital). The authors evaluate 26 LLMs across commercial, open-source general-purpose, medical, and reasoning-enhanced categories. The central claim is that LLMs achieve high MCQ accuracy (up to 0.84) but exhibit substantial performance degradation on clinical case diagnosis, particularly on inpatient cases with an average of 7.86 diagnoses per case (patient-level recall 0.09–0.22). The paper also claims that medical LLMs underperform general-purpose models on HPB diagnosis, indicating limited generalizability.
Significance. If the central claim holds, ClinBench-HPB would be a valuable resource: it provides broad ICD-10 coverage, a large set of clinically sourced cases, a two-stage LLM-as-judge evaluation with physician agreement, and a systematic comparison of 26 models. The construction pipeline is described in detail, and the authors report physician validation of 500 synthetic MCQs (96% quality) and judge agreement of 97.5% and 97.7% on sampled diseases. However, the benchmark's main conclusion rests on the assumption that the reference diagnosis lists are complete, correct, and inferable from the input text. The example in Figure 1 raises a concrete concern, and the lack of a human baseline is a critical omission.
major comments (5)
- [Section 3.1, Figure 1, Section 3.2] The reference 'Final Diagnosis' lists comorbidities that are not inferable from the provided patient information. In the Figure 1 example, the reference list includes coronary atherosclerosis and duodenal diverticulum, but the patient information, laboratory tests, and imaging findings shown do not provide evidence for these conditions. Since the disease-level and patient-level recall metrics in Section 3.2 count every reference disease, a model that omits non-inferable items is penalized as if it made a diagnostic error. With an average of 7.86 diagnoses per Hospital case, the reported patient-level recalls of 0.09–0.22 and the contrast with MCQ accuracy near 0.8 conflate true diagnostic misses with impossible-to-derive reference labels. The paper should either restrict reference lists to conditions that are actually supported by the input text, or report a human-physician baseline showing how often clinicians can recover the same list from the same input.
- [Section 3.2, Figure 6] The diagnostic prompts instruct the model to 'list all major diseases,' while the reference lists include incidental findings and chronic comorbidities such as pulmonary bullae, coronary atherosclerosis, and duodenal diverticulum (Figure 1). This is a mismatch between the task instruction and the evaluation criterion. A model that deliberately focuses on clinically significant HPB conditions is penalized for not listing incidental imaging findings. The authors should align the reference construction with the prompt (e.g., by scoring only diseases that are clinically relevant to the HPB presentation or that are explicitly inferable from the input) or change the prompt to ask for all documented conditions.
- [Section 3.2, Appendix A.1.2] The reported agreement rates (97.5% on 393 Chinese diseases and 97.7% on 132 English diseases) validate the automated judge against physician judgments on the judge's decisions, but they do not validate the completeness or correctness of the reference lists themselves. The sample selection (which cases, how many physicians, inter-rater agreement) is not described. Moreover, DeepSeek-V3-0324 is both the strict evaluation model and one of the evaluated models in Table 3; while the check model (Claude-3.7-Sonnet) and the physician agreement mitigate self-favoritism, the paper should report whether the model's relative ranking changes when an independent judge is used.
- [Table 3, Section 4.2] The combined metric Avg = (Avgq + Avgp)/2 gives equal weight to MCQ accuracy and patient-level clinical recall, and the claims that open-source models are 'approaching commercial models' and that reasoning-enhanced LLMs 'show promise' are based on this arbitrary weighting. For instance, Llama3.1-70B (Avg 0.601) is reported as outperforming GPT-4o (Avg 0.595), but GPT-4o is superior on both MCQ subsets (Avgq 0.722 vs 0.809). The paper should present task-specific results and, if a combined metric is used, justify the weighting and report statistical significance.
- [Section 3.1] The paper claims that the benchmark encompasses all 33 main categories and 465 subcategories of HPB diseases defined in ICD-10, but no coverage mapping is provided. The construction process (keyword filtering, LLM-based selection, synthetic generation) is described qualitatively, and Table 2 gives only aggregate counts per subset. Without a table or figure mapping the 3,535 MCQs and 337 cases to the ICD-10 codes (or at least to the 33 Level-1 categories), this central claim is unverifiable. The authors should include a coverage analysis as part of the benchmark release.
minor comments (6)
- [Throughout] There are numerous typos, including 'eatablish' (abstract), 'evalauting' (abstract), 'diganostic' (Section 1), 'multi-scource' (Section 1), 'improvment' (Section 4.2), 'Hostipal' (Table 3), 'evalaution' (Section 5), and 'inherents' (Section 5). The title and abstract should also use a consistent spelling of the benchmark name ('ClinBench-HPB' vs 'ClinBench-HBP').
- [Section 3.1] The number of synthesized questions is reported as 'approximately 1,400' and later as '1,362 questions generated.' These numbers should be reconciled.
- [Section 3.1] The EN-QA subset is 1,535 items, described as 173 items from public datasets and 1,362 synthetic items, but no per-source counts are given for the public datasets (MedQA-English, MedMCQA, PubMedQA, MMLU-Pro). Providing a breakdown would improve transparency.
- [Table 3, Appendix Tables 4-7] The tables are hard to read because columns are packed without visual separation and the header row is ambiguous. In Table 7, the OpenAI-o1 row shows identical values (Pt=0.180, Dis=0.710) across all four prompts with zero standard deviation; the authors should explain why the prompt variability had no effect for this model.
- [Section 4.1, Appendix A.2] The collection of 50 inpatient cases from a collaborating hospital is described, but no ethics approval or data-governance statement is provided; only manual PHI removal is mentioned. A statement on institutional review and consent would be expected for a clinical benchmark.
- [Benchmark Release] The benchmark release is described only as a URL (https://clinbench-hpb.github.io). The paper should specify the license, data format, and whether evaluation scripts and full model outputs will be released to enable reproducibility.
Circularity Check
No significant circularity: the benchmark is grounded in external case sources and physician-validated labels, and the reported comparisons are measurements rather than constructions.
full rationale
The paper's central claim is empirical: LLMs score well on MCQ items but show lower recall on open-ended HPB case diagnosis. The clinical-case reference labels come from external sources (NEJM, JAMA, CNKI, case-sharing websites, and a collaborating hospital) and are reviewed by four licensed physicians, so the main diagnostic scores are anchored outside the paper's own construction pipeline. The MCQ subset draws from public examination datasets, with LLM-based filtering and synthesis used for coverage; however, the reported accuracies are not defined by those filters in any equation-level way, and the paper's main finding is not that any specific model must score at a particular value by construction. The use of DeepSeek-V3-0324 as the strict evaluation model while also being one of the evaluated models is a self-evaluation overlap, but it is not a reduction: the judge's decisions are based on a six-dimensional rubric with physician agreement rates of 97.5% and 97.7%, and the same judge gives itself a low Hospital patient-level recall (0.175), so there is no evidence that the scoring rule forces the reported degradation. The only explicit limitations stated by the authors—text-only modality and stochasticity/expense of LLM-based assessment—concern reliability and scope rather than circularity. The concern that some reference comorbidities may not be inferable from the provided text is a label-validity question, not a circularity of derivation: the evaluation would be miscalibrated if true, but it would not be equivalent to its inputs by definition. Overall, the paper is self-contained against external benchmarks and physician review, so no circular step can be exhibited under the required standard.
Assumptions & free parameters
free parameters (3)
- synthetic MCQ retention threshold =
scores 1-2 excluded, 3 revised, 4-5 retained
- option permutation count =
min(4, n-1) circular shifts
- reasoning-model temperature =
0.6 for reasoning-enhanced models and DeepSeek-R1; greedy for others
assumptions (5)
- domain assumption ICD-10 HPB disease coding (33 main categories, 465 subcategories) is a complete and valid taxonomy for HPB disease coverage.
- domain assumption Reference diagnoses in clinical cases are complete, correct, and inferable from the text given to the model.
- domain assumption The two-stage LLM judge, DeepSeek-V3-0324 as strict evaluator and Claude-3.7-Sonnet as check model, is a valid proxy for physician assessment of diagnostic coverage.
- ad hoc to paper LLM-synthesized multiple-choice questions preserve clinical validity and do not introduce leakage or biased content.
- domain assumption Public exam datasets (MedQA, MedMCQA, CMB, PubMedQA, MMLU-Pro) measure HPB knowledge meaningfully.
Cite this review
Pith. "Pith review of ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases." pith.science (2026). https://pith.science/paper/BMDHWZPO
@misc{pith2026250600095,
author = {Pith},
title = {Pith review of: ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases},
year = {2026},
howpublished = {\url{https://pith.science/paper/BMDHWZPO}},
note = {Machine review of arXiv:2506.00095}
}
read the original abstract
Hepato-pancreato-biliary (HPB) disorders represent a global public health challenge due to their high morbidity and mortality. Although large language models (LLMs) have shown promising performance in general medical question-answering tasks, the current evaluation benchmarks are mostly derived from standardized examinations or manually designed questions, lacking HPB coverage and clinical cases. To address these issues, we systematically eatablish an HPB disease evaluation benchmark comprising 3,535 closed-ended multiple-choice questions and 337 open-ended real diagnosis cases, which encompasses all the 33 main categories and 465 subcategories of HPB diseases defined in the International Statistical Classification of Diseases, 10th Revision (ICD-10). The multiple-choice questions are curated from public datasets and synthesized data, and the clinical cases are collected from prestigious medical journals, case-sharing platforms, and collaborating hospitals. By evalauting commercial and open-source general and medical LLMs on our established benchmark, namely ClinBench-HBP, we find that while commercial LLMs perform competently on medical exam questions, they exhibit substantial performance degradation on HPB diagnosis tasks, especially on complex, inpatient clinical cases. Those medical LLMs also show limited generalizability to HPB diseases. Our results reveal the critical limitations of current LLMs in the domain of HPB diseases, underscoring the imperative need for future medical LLMs to handle real, complex clinical diagnostics rather than simple medical exam questions. The benchmark will be released at https://clinbench-hpb.github.io.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma.Clinical and molecular hepatology, 29(3):721, 2023
Yee Hui Yeo, Jamil S Samaan, Wee Han Ng, Peng-Sheng Ting, Hirsh Trivedi, Aarshi Vipani, Walid Ayoub, Ju Dong Yang, Omer Liran, Brennan Spiegel, et al. Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma.Clinical and molecular hepatology, 29(3):721, 2023
2023
-
[2]
Exploring the performance of chatgpt on acute pancreatitis-related questions.Journal of Translational Medicine, 22(1):527, 2024
Ren-Chun Du, Xing Liu, Yong-Kang Lai, Yu-Xin Hu, Hao Deng, Hui-Qiao Zhou, Nong-Hua Lu, Yin Zhu, and Yi Hu. Exploring the performance of chatgpt on acute pancreatitis-related questions.Journal of Translational Medicine, 22(1):527, 2024
2024
-
[3]
Tyrus V ong, Nicholas Rizer, Vedant Jain, Valerie L Thompson, Mark Dredze, Eili Y Klein, Jeremiah S Hinson, Tanjala Purnell, Stephen Kwak, Tinsay Woreta, et al. Automated identification of incidental hepatic steatosis on emergency department imaging using large language models.Hepatology Communications, 9(3):e0638, 2025
work page 2025
-
[4]
Yutaka Endo, Kazunari Sasaki, Zorays Moazzam, Henrique A Lima, Austin Schenk, Ashley Limkemann, Kenneth Washburn, and Timothy M Pawlik. Quality of chatgpt responses to questions related to liver transplantation.Journal of Gastrointestinal Surgery, 27(8):1716–1719, 2023
work page 2023
-
[5]
Simone Kresevic, Mauro Giuffrè, Milos Ajcevic, Agostino Accardo, Lory S Crocè, and Dennis L Shung. Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework.NPJ digital medicine, 7(1):102, 2024
work page 2024
-
[6]
Jin Ge, Steve Sun, Joseph Owens, Victor Galvez, Oksana Gologorskaya, Jennifer C Lai, Mark J Pletcher, and Ki Lai. Development of a liver disease–specific large language model chat interface using retrieval- augmented generation.Hepatology, 80(5):1158–1168, 2024
work page 2024
-
[7]
Nicholas Dietrich and Brett Stubbert. Evaluating adherence to canadian radiology guidelines for incidental hepatobiliary findings using rag-enabled llms.Canadian Association of Radiologists Journal, page 08465371251323124, 2025
work page 2025
-
[8]
Lili Lu, Christina S Mullins, Clemens Schafmayer, Sebastian Zeißig, and Michael Linnebacher. A global assessment of recent trends in gastrointestinal cancer and lifestyle-associated risk factors.Cancer Communications, 41(11):1137–1151, 2021
work page 2021
Show all 82 references
-
[9]
Burden and cost of gastrointestinal, liver, and pancreatic diseases in the united states: update 2024.Gastroenterology, 2025
Anne F Peery, Caitlin C Murphy, Chelsea Anderson, Elizabeth T Jensen, Sasha Deutsch-Link, Matthew D Egberg, Jennifer L Lund, Disha Subramaniam, Evan S Dellon, Ami D Sperber, et al. Burden and cost of gastrointestinal, liver, and pancreatic diseases in the united states: update...
2024
-
[10]
Large language models in gastroenterology and gastrointestinal surgery: A new frontier in patient communication and education.Gastroenterology Research, 18(2):39, 2025
Dushyant Singh Dahiya, Hassam Ali, Vishali Moond, M Danial Ali Shah, Christina Santana, Noor Ali, Abu Baker Sheikh, Muhammad Ahmad Nadeem, Aqsa Munir, Mohammed A Quazi, et al. Large language models in gastroenterology and gastrointestinal surgery: A new frontier in patient com...
2025
-
[11]
Large language models in gastroenterology: Systematic review.Journal of medical Internet research, 26:e66648, 2024
Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee, Jonghyung Park, Eunsil Kim, Subeen Kim, Minjae Kimm, and Seoung-Ho Choi. Large language models in gastroenterology: Systematic review.Journal of medical Internet research, 26:e66648, 2024
2024
-
[12]
Lkan: Llm-based knowledge-aware attention network for clinical staging of liver cancer.IEEE Journal of Biomedical and Health Informatics, 2024
Ya Li, Xuecong Zheng, Jiaping Li, Qingyun Dai, Chang-Dong Wang, and Min Chen. Lkan: Llm-based knowledge-aware attention network for clinical staging of liver cancer.IEEE Journal of Biomedical and Health Informatics, 2024
2024
-
[13]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024
2024 arXiv
-
[14]
The llama 3 herd of models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024
2024 arXiv
-
[15]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 10
2023 arXiv
-
[16]
Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...
2022
-
[17]
Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes.Liver International, 44(9):2114–2124, 2024
Mauro Giuffrè, Simone Kresevic, Nicola Pugliese, Kisung You, and Dennis L Shung. Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes.Liver International, 44(9):2114–2124, 2024
2024
-
[18]
Ruben Geevarghese, Carlie Sigel, John Cadley, Subrata Chatterjee, Pulkit Jain, Alex Hollingsworth, Avijit Chatterjee, Nathaniel Swinburne, Khawaja Hasan Bilal, and Brett Marinelli. Extraction and classification of structured data from unstructured hepatobiliary pathology repor...
2025
-
[19]
Using a customized gpt to provide guideline-based recommendations for management of pancreatic cystic lesions
Yuri Gorelik, Itai Ghersin, Tarek Arraf, Offir Ben-Ishay, Amir Klein, and Iyad Khamaysi. Using a customized gpt to provide guideline-based recommendations for management of pancreatic cystic lesions. Endoscopy International Open, 12(04):E600–E603, 2024
2024
-
[20]
Nicola Pugliese, Vincent Wai-Sun Wong, Jörn M Schattenberg, Manuel Romero-Gomez, Giada Sebastiani, Laurent Castera, Cesare Hassan, Pinelopi Manousou, Luca Miele, Raquel Peck, et al. Accuracy, reliability, and comprehensibility of chatgpt-generated medical responses for patient...
2024
-
[21]
Accuracy of information provided by chatgpt regarding liver cancer surveillance and diagnosis.American Journal of Roentgenology, 221(4):556–559, 2023
Jennie J Cao, Daniel H Kwon, Tara T Ghaziani, Paul Kwo, Gary Tse, Andrew Kesselman, Aya Kamaya, and Justin R Tse. Accuracy of information provided by chatgpt regarding liver cancer surveillance and diagnosis.American Journal of Roentgenology, 221(4):556–559, 2023
2023
-
[22]
Using large language models in the diagnosis of acute cholecystitis: Assessing accuracy and guidelines compliance.The American Surgeon™, page 00031348251323719, 2025
Marta Goglia, Arianna Cicolani, Francesco Maria Carrano, Niccolò Petrucciani, Francesco D’Angelo, Marco Pace, Lucio Chiarini, Gianfranco Silecchia, and Paolo Aurello. Using large language models in the diagnosis of acute cholecystitis: Assessing accuracy and guidelines complia...
2025
-
[23]
Tianyang Mao, Xin Zhao, Kangyi Jiang, Qingyun Xie, Manyu Yang, Ruoxuan Wang, and Fengwei Gao. A comparison of the responses between chatgpt and doctors in the field of cholelithiasis based on clinical practice guidelines: a cross-sectional study.Digital Health, 11:205520762513...
2025
-
[24]
Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge.Physics in Medicine and Biology, 2024
Xuzhou Wu, Guangxin Li, Xing Wang, Zeyu Xu, Yingni Wang, Shuge Lei, Jianming Xian, Xueyu Wang, Yibao Zhang, Gong Li, et al. Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge.Physics in Medicine and Biology, 2024
2024
-
[25]
Icd-10.https://code.nhsa.gov.cn/search.html?sysflag=80
-
[26]
Icd-10 version: 2019.https://icd.who.int/browse10/2019/en
2019
-
[27]
Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications.Advances in Neural Information Processing Systems, 37:138632–138662, 2024
Dingkang Yang, Jinjie Wei, Dongling Xiao, Shunli Wang, Tong Wu, Gang Li, Mingcheng Li, Shuaibing Wang, Jiawei Chen, Yue Jiang, et al. Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications.Advances in Neural Information Processing Systems...
2024
-
[28]
Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023
Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023
2023
-
[29]
Icga-gpt: report generation and question answering for indocyanine green angiography images.British Journal of Ophthalmology, 108(10):1450–1456, 2024
Xiaolan Chen, Weiyi Zhang, Ziwei Zhao, Pusheng Xu, Yingfeng Zheng, Danli Shi, and Mingguang He. Icga-gpt: report generation and question answering for indocyanine green angiography images.British Journal of Ophthalmology, 108(10):1450–1456, 2024
2024
-
[30]
Chatffa: interactive visual question answering on fundus fluorescein angiography image using chatgpt
Xiaolan Chen, Pusheng Xu, Yao Li, Weiyi Zhang, Fan Song, Ying-Feng Zheng, Danli Shi, and Mingguang He. Chatffa: interactive visual question answering on fundus fluorescein angiography image using chatgpt. Available at SSRN 4578568, 2023
2023
-
[31]
Compar- ative experimentation of accuracy metrics in automated medical reporting: the case of otitis consultations
Wouter Faber, Renske Eline Bootsma, Tom Huibers, Sandra van Dulmen, and Sjaak Brinkkemper. Compar- ative experimentation of accuracy metrics in automated medical reporting: the case of otitis consultations. arXiv preprint arXiv:2311.13273, 2023
2023 arXiv
-
[32]
Medredqa for medical consumer question answering: Dataset, tasks, and neural baselines
Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, and Zhenchang Xing. Medredqa for medical consumer question answering: Dataset, tasks, and neural baselines. InProceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the As...
2023
-
[33]
Medic: Towards a comprehensive framework for evaluating llms in clinical applications.arXiv preprint arXiv:2409.07314, 2024
Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. Medic: Towards a comprehensive framework for evaluating llms in clinical applications.arXiv preprint arXiv:2409.0...
2024 arXiv
-
[34]
Pediabench: A comprehensive chinese pediatric dataset for benchmarking large language models
Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, and Yanhao Wang. Pediabench: A comprehensive chinese pediatric dataset for benchmarking large language models. arXiv preprint arXiv:2412.06287, 2024
2024 arXiv
-
[35]
Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency?arXiv preprint arXiv:2404.06503, 2024
Nathan Brake and Thomas Schaaf. Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency?arXiv preprint arXiv:2404.06503, 2024
2024 arXiv
-
[36]
Towards leveraging large language models for automated medical q&a evaluation.arXiv preprint arXiv:2409.01941, 2024
Jack Krolik, Herprit Mahal, Feroz Ahmad, Gaurav Trivedi, and Bahador Saket. Towards leveraging large language models for automated medical q&a evaluation.arXiv preprint arXiv:2409.01941, 2024
2024 arXiv
-
[37]
Large language models for automated synoptic reports and resectability categorization in pancreatic cancer.Radiology, 311(3):e233117, 2024
Rajesh Bhayana, Bipin Nanda, Taher Dehkharghanian, Yangqing Deng, Nishaant Bhambra, Gavin Elias, Daksh Datta, Avinash Kambadakone, Chaya G Shwaartz, Carol-Anne Moulton, et al. Large language models for automated synoptic reports and resectability categorization in pancreatic c...
2024
-
[38]
Ultramedical: Building specialized generalists in biomedicine.Advances in Neural Information Processing Systems, 37:26045–26081, 2024
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. Ultramedical: Building specialized generalists in biomedicine.Advances in Neural Information Processing Systems, 37:26045–26081, 2024
2024
-
[39]
Openbiollms: Advancing open- source large language models for healthcare and life sciences.Hugging Face repository, 2024
Malaikannan Sankarasubbu Ankit Pal and Malaikannan Sankarasubbu. Openbiollms: Advancing open- source large language models for healthcare and life sciences.Hugging Face repository, 2024
2024
-
[40]
O1 replication journey–part 3: Inference-time scaling for medical reasoning.arXiv preprint arXiv:2501.06458, 2025
Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. O1 replication journey–part 3: Inference-time scaling for medical reasoning.arXiv preprint arXiv:2501.06458, 2025
2025 arXiv
-
[41]
m1: Unleash the potential of test-time scaling for medical reasoning with large language models.arXiv preprint arXiv:2504.00869, 2025
Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models.arXiv preprint arXiv:2504.00869, 2025
2025
-
[42]
Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024
2024 arXiv
-
[43]
What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021
2021
-
[44]
Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning, pages 248–260. PMLR, 2022
2022
-
[45]
Cmb: A comprehensive medical benchmark in chinese.arXiv preprint arXiv:2308.08833, 2023
Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. Cmb: A comprehensive medical benchmark in chinese.arXiv preprint arXiv:2308.08833, 2023
2023 arXiv
-
[46]
Benchmarking large language models on answering and explaining challenging medical questions
Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics...
2025
-
[47]
Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019
1909 arXiv
-
[48]
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...
2024
-
[49]
Gpqa: A graduate-level google-proof q&a benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024
2024
-
[50]
It’s time to bench the medical exam benchmark, 2025
Inioluwa Deborah Raji, Roxana Daneshjou, and Emily Alsentzer. It’s time to bench the medical exam benchmark, 2025. 12
2025
-
[51]
Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024
2024 arXiv
-
[52]
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024
2024 arXiv
-
[53]
https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
Gpt-4o-mini. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
-
[54]
Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023
2023 arXiv
-
[55]
Deepseek-v3-0324.https://huggingface.co/deepseek-ai/DeepSeek-V3-0324
-
[56]
Claude 3.7 sonnet.https://www.anthropic.com/claude/sonnet
-
[57]
Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet
-
[58]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025
2025 arXiv
-
[59]
Qwen2.5-max.https://qwenlm.github.io/zh/blog/qwen2.5-max/
-
[60]
Openai-o1.https://openai.com/o1/
-
[61]
Openai o3-mini.https://openai.com/index/openai-o3-mini/
-
[62]
Gemini 2.5 pro.https://deepmind.google/technologies/gemini/pro/
-
[63]
Qwq-32b.https://qwenlm.github.io/blog/qwq-32b/
-
[64]
{disease name}
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, pa...
2023
-
[65]
Anatomical specificity: The affected organ/structure must be clearly specified (e.g., cholecystitis and cholangitis cannot be considered as included)
-
[66]
Pathological mechanism: Distinguish between inflammation/obstruction/tumor/thrombosis and other essential differences (e.g., portal vein tumor thrombus and portal vein thrombosis cannot be considered as matching)
-
[67]
Etiology tracing: Distinguish between causes such as stones/infection/post-surgery/tumor (e.g., obstructive jaundice and gallbladder stones cannot be considered as matching)
-
[68]
Temporal characteristics: Acute/chronic course must be clearly indicated (e.g., acute cholecystitis and chronic cholecystitis cannot be considered as matching)
-
[69]
Test identification: Laboratory/imaging results must be the same to be considered the same disease (e.g., segmental pneumonia and lung nodules cannot be considered as matching)
-
[70]
Includes/Does not include
Treatment strategy: The same intervention methods must be considered the same disease (e.g., cirrhosis with portal hypertension and portal vein tumor thrombus cannot be considered as matching). The output format is: { "Includes/Does not include": "Brief description of the basi...
-
[71]
2.Combine your professional knowledge to infer possible diseases or etiologies
Thoroughly analyze all provided patient information, including chief complaint, medical history, physical examination results, laboratory tests, and imaging studies. 2.Combine your professional knowledge to infer possible diseases or etiologies. 3.Ensure no information is over...
-
[72]
The current patient information is as follows: {clinical case information} Figure 6: Illustration of the prompt templates for the subjective questions
Provide a preliminary diagnosis with a brief explanation of the diagnostic basis. The current patient information is as follows: {clinical case information} Figure 6: Illustration of the prompt templates for the subjective questions. 16 the questions from both CN-QA and EN-QA ...
-
[73]
Gallbladder Adenomyomatosis (with Septation) and Biliary Sludge. Basis for Diagnosis: Imaging Findings: Abdominal ultrasound demonstrates septated gallbladder with sonographic features of adenomyomatosis, along with linear hyperechoic foci (sludge). CT reveals gallbladder fold...
-
[74]
Diagnostic Basis: Laboratory findings: Elevated serum cholesterol (exact values unspecified) documented as abnormal in medical reports
Hypercholesterolemia. Diagnostic Basis: Laboratory findings: Elevated serum cholesterol (exact values unspecified) documented as abnormal in medical reports. A well-established risk factor for gallstones, particularly cholesterol gallstones
-
[75]
Diagnostic Basis: Imaging findings: Cervical spine examination at Center A clearly demonstrates osteophyte formation, indicative of degenerative changes
Cervical Osteophyte Formation. Diagnostic Basis: Imaging findings: Cervical spine examination at Center A clearly demonstrates osteophyte formation, indicative of degenerative changes. Asymptomatic presentation, but radiographic evidence is definitive
-
[76]
adenomyomatosis of the gallbladder (with septation) accompanied by gallstones (sludge-like stones)
Chronic Inflammation in Right Middle Lobe/Upper Lobe Fibrotic Lesions (Bilateral). Diagnostic Basis: Imaging Findings: Chest examination at Center A revealed chronic inflammatory changes and fibrotic foci. Subsequent CT at Hospital C showed no active lesions, suggesting residu...
-
[77]
MRI impression: A space-occupying lesion in the left lateral lobe of the liver is considered to be hepatocellular carcinoma
Multiple small cysts are detected in both kidneys during the scan. MRI impression: A space-occupying lesion in the left lateral lobe of the liver is considered to be hepatocellular carcinoma. Multiple cysts are found in the liver. The volume of the gallbladder is increased, an...
-
[78]
fast in and fast out
Hepatocellular Carcinoma (HCC) Diagnostic bases: • Extremely elevated tumor markers: Alpha-fetoprotein (AFP) is far higher than the normal value (>100,000 μg/L), and Protein Induced by Vitamin K Absence or Antagonist-II (PIVKA-II) is also significantly elevated. The simultaneo...
-
[79]
Cholecystitis Diagnostic bases: • Imaging findings: Both CT and MR show an enlarged gallbladder volume, slightly thickened and enhanced gallbladder wall, which are consistent with the changes of cholecystitis. • Clinical manifestations: Discomfort in the right upper abdomen an...
-
[80]
These are commonly found during physical examinations
Multiple cysts in the liver and both kidneys Diagnostic bases: • Imaging: Multiple round-like low-density (on CT) or long T2 (on MR) non-enhancing lesions are visible in the liver and both kidneys, with clear boundaries, which are in line with the characteristics of cysts. The...
-
[81]
Abnormal lipid metabolism (hypercholesterolemia, low HDL cholesterolemia, etc.) Diagnostic bases: • Laboratory tests: Total cholesterol (TC) is 6.16 mmol/L (↑), high-density lipoprotein (HDL) is only 0.84 mmol/L (↓), and lipoprotein (a) and apolipoprotein B are on the high sid...
-
[82]
Included
Other changes that require attention • Mild abnormalities in liver function and biliary enzyme spectrum: The elevation of indicators such as ALT, AST, GGT, and ALP suggests damage to liver cells and the biliary tract; some of these may be caused by liver cancer itself or concu...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.