Pith. sign in

REVIEW 5 major objections 6 minor 82 references

ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Commercial LLMs score well on HPB exams but miss most concurrent diagnoses in real inpatient cases, and medical LLMs fail to generalize to HPB diseases.

desk verdict A serious HPB benchmark worth refereeing, but the headline degradation numbers are probably overstated by non-inferable reference diagnoses and the lack of a human baseline. read the letter →

arxiv 2506.00095 v3 pith:BMDHWZPO submitted 2025-05-30 cs.CY cs.AIcs.CL

classification cs.CYcs.AIcs.CL
keywords LLMevaluationhepatopancreatobiliarydiseasesclinicaldiagnosisbenchmarkICD-10patient-levelrecallmedicalLLMsLLM-as-a-judgeout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ClinBench-HPB is a benchmark built to test whether LLMs can actually diagnose liver, pancreas, and biliary (HPB) diseases, not just answer exam questions. It pairs 3,535 multiple-choice questions covering all ICD-10 HPB categories with 337 open-ended real clinical cases from journals, case-sharing sites, and a collaborating hospital. Across 26 LLMs, the paper finds that exam accuracy is high, with the best commercial models averaging about 0.84, while clinical case diagnosis is much weaker; on the 50 inpatient hospital cases, patient-level recall, which requires naming every concurrent diagnosis, falls to roughly 0.09–0.22 for commercial models. Medical LLMs fine-tuned on exam data perform worse on these out-of-distribution cases than their general-purpose base models. The paper concludes that current LLM diagnostic skill is overestimated by medical-exam benchmarks and that future medical LLMs should be evaluated on real clinical cases.

What carries the argument

ClinBench-HPB is the central object: 3,535 multiple-choice questions (2,000 Chinese, 1,535 English) drawn from public exam datasets plus synthesized items, and 337 open-ended clinical cases from journals, case-sharing websites, and a collaborating hospital, all mapped to the 33 main and 465 subcategories of ICD-10 HPB diseases. The scoring mechanism is a two-stage LLM-as-a-judge, with DeepSeek-V3-0324 as the strict evaluator deciding whether a prediction includes each reference disease on six dimensions—anatomical specificity, pathological mechanism, etiological origin, temporal characteristics, test identification, and treatment strategy—and Claude-3.7-Sonnet as the check model to catch false negatives from rigid matching. Performance is reported as disease-level recall and patient-level recall, where patient-level recall requires naming every concurrent diagnosis; four distinct prompts and circular option shifts are used to smooth over input sensitivity.

What would settle it

If re-scoring the 50 hospital cases using only each patient's primary diagnosis, dropping incidental comorbidities such as pulmonary bullae or coronary atherosclerosis, erased most of the gap between MCQ accuracy and patient-level recall, the claim that LLMs fail at real HPB diagnosis would be seriously weakened. A second check would be to have experienced clinicians independently list all diagnoses from the same case text and compare their recall to the LLMs' recall.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM performance on HPB medicine is substantially overestimated by closed-ended medical-exam benchmarks. In the benchmark's MCQ portion, leading commercial models reach average accuracies around 0.84, which would suggest strong mastery of HPB knowledge; in the open-ended diagnosis portion, disease-level recall is moderate but patient-level recall collapses, and on the 50 hospital-collected inpatient cases even the best commercial model (DeepSeekV3-1226) captures the full spectrum of concurrent diagnoses in only 22% of patients. The paper further argues that medical LLMs fine-tuned on exam-style data degrade on HPB cases outside their training distribution, with all four HuatuoGPT-o1 models underperforming their base models on website and hospital subsets, while reasoning-enhanced LLMs show the most transfer to diagnosis as they scale. The benchmark's design is the evidence: cases are organized by ICD-10 categories, each reference diagnosis is checked against the model's prediction along six clinical dimensions, and results are averaged over four prompt variants.

Load-bearing premise

The benchmark's central result depends on the reference diagnosis list for each case being complete, correct, and derivable from the text a model sees, so that a model failing to name any listed comorbidity is counted as a diagnostic error.

Editorial extensions

If this is right

  • Medical-exam accuracy should not be read as clinical diagnostic competence, because the same models that reach about 0.84 average accuracy on MCQs drop to low patient-level recall on inpatient HPB cases.
  • Hospital-style HPB cases with multiple comorbidities are the hard setting where current LLMs miss diagnoses that can change management, so deployment in that setting needs additional safeguards.
  • Fine-tuning medical LLMs on exam-style data does not guarantee transfer, and the HuatuoGPT-o1 models underperforming their general-purpose bases on website and hospital cases shows a concrete out-of-distribution risk.
  • Reasoning-enhanced LLMs trained on math and code transfer some capability to diagnosis as they scale, suggesting reasoning training as a plausible path when medical data are scarce.
  • The released benchmark gives a standardized, ICD-10-organized test for future HPB LLMs, covering all 33 main categories and 465 subcategories of HPB disease.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test worth running is whether experienced clinicians, given the same case text, also fail to list every comorbidity; if they do, the patient-level recall gap reflects task ambiguity more than model deficiency.
  • The two-model judge could be audited by physicians on all 337 cases rather than sampled subsets; the reported 97.5–97.7% agreement may not hold uniformly for rare comorbidities.
  • Because the hospital cases are text-only, adding the original imaging could change results in either direction, since the CT and MRI impressions already contain some disease names.
  • The same benchmark construction could be applied to other organ systems to see whether the MCQ-to-clinical gap is a general property of LLMs or specific to HPB complexity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces ClinBench-HPB, a benchmark for hepato-pancreato-biliary (HPB) diseases containing 3,535 multiple-choice questions (CN-QA: 2,000 Chinese; EN-QA: 1,535 English) and 337 open-ended clinical cases (120 from journals, 167 from case-sharing websites, 50 from a collaborating hospital). The authors evaluate 26 LLMs across commercial, open-source general-purpose, medical, and reasoning-enhanced categories. The central claim is that LLMs achieve high MCQ accuracy (up to 0.84) but exhibit substantial performance degradation on clinical case diagnosis, particularly on inpatient cases with an average of 7.86 diagnoses per case (patient-level recall 0.09–0.22). The paper also claims that medical LLMs underperform general-purpose models on HPB diagnosis, indicating limited generalizability.

Significance. If the central claim holds, ClinBench-HPB would be a valuable resource: it provides broad ICD-10 coverage, a large set of clinically sourced cases, a two-stage LLM-as-judge evaluation with physician agreement, and a systematic comparison of 26 models. The construction pipeline is described in detail, and the authors report physician validation of 500 synthetic MCQs (96% quality) and judge agreement of 97.5% and 97.7% on sampled diseases. However, the benchmark's main conclusion rests on the assumption that the reference diagnosis lists are complete, correct, and inferable from the input text. The example in Figure 1 raises a concrete concern, and the lack of a human baseline is a critical omission.

major comments (5)
  1. [Section 3.1, Figure 1, Section 3.2] The reference 'Final Diagnosis' lists comorbidities that are not inferable from the provided patient information. In the Figure 1 example, the reference list includes coronary atherosclerosis and duodenal diverticulum, but the patient information, laboratory tests, and imaging findings shown do not provide evidence for these conditions. Since the disease-level and patient-level recall metrics in Section 3.2 count every reference disease, a model that omits non-inferable items is penalized as if it made a diagnostic error. With an average of 7.86 diagnoses per Hospital case, the reported patient-level recalls of 0.09–0.22 and the contrast with MCQ accuracy near 0.8 conflate true diagnostic misses with impossible-to-derive reference labels. The paper should either restrict reference lists to conditions that are actually supported by the input text, or report a human-physician baseline showing how often clinicians can recover the same list from the same input.
  2. [Section 3.2, Figure 6] The diagnostic prompts instruct the model to 'list all major diseases,' while the reference lists include incidental findings and chronic comorbidities such as pulmonary bullae, coronary atherosclerosis, and duodenal diverticulum (Figure 1). This is a mismatch between the task instruction and the evaluation criterion. A model that deliberately focuses on clinically significant HPB conditions is penalized for not listing incidental imaging findings. The authors should align the reference construction with the prompt (e.g., by scoring only diseases that are clinically relevant to the HPB presentation or that are explicitly inferable from the input) or change the prompt to ask for all documented conditions.
  3. [Section 3.2, Appendix A.1.2] The reported agreement rates (97.5% on 393 Chinese diseases and 97.7% on 132 English diseases) validate the automated judge against physician judgments on the judge's decisions, but they do not validate the completeness or correctness of the reference lists themselves. The sample selection (which cases, how many physicians, inter-rater agreement) is not described. Moreover, DeepSeek-V3-0324 is both the strict evaluation model and one of the evaluated models in Table 3; while the check model (Claude-3.7-Sonnet) and the physician agreement mitigate self-favoritism, the paper should report whether the model's relative ranking changes when an independent judge is used.
  4. [Table 3, Section 4.2] The combined metric Avg = (Avgq + Avgp)/2 gives equal weight to MCQ accuracy and patient-level clinical recall, and the claims that open-source models are 'approaching commercial models' and that reasoning-enhanced LLMs 'show promise' are based on this arbitrary weighting. For instance, Llama3.1-70B (Avg 0.601) is reported as outperforming GPT-4o (Avg 0.595), but GPT-4o is superior on both MCQ subsets (Avgq 0.722 vs 0.809). The paper should present task-specific results and, if a combined metric is used, justify the weighting and report statistical significance.
  5. [Section 3.1] The paper claims that the benchmark encompasses all 33 main categories and 465 subcategories of HPB diseases defined in ICD-10, but no coverage mapping is provided. The construction process (keyword filtering, LLM-based selection, synthetic generation) is described qualitatively, and Table 2 gives only aggregate counts per subset. Without a table or figure mapping the 3,535 MCQs and 337 cases to the ICD-10 codes (or at least to the 33 Level-1 categories), this central claim is unverifiable. The authors should include a coverage analysis as part of the benchmark release.
minor comments (6)
  1. [Throughout] There are numerous typos, including 'eatablish' (abstract), 'evalauting' (abstract), 'diganostic' (Section 1), 'multi-scource' (Section 1), 'improvment' (Section 4.2), 'Hostipal' (Table 3), 'evalaution' (Section 5), and 'inherents' (Section 5). The title and abstract should also use a consistent spelling of the benchmark name ('ClinBench-HPB' vs 'ClinBench-HBP').
  2. [Section 3.1] The number of synthesized questions is reported as 'approximately 1,400' and later as '1,362 questions generated.' These numbers should be reconciled.
  3. [Section 3.1] The EN-QA subset is 1,535 items, described as 173 items from public datasets and 1,362 synthetic items, but no per-source counts are given for the public datasets (MedQA-English, MedMCQA, PubMedQA, MMLU-Pro). Providing a breakdown would improve transparency.
  4. [Table 3, Appendix Tables 4-7] The tables are hard to read because columns are packed without visual separation and the header row is ambiguous. In Table 7, the OpenAI-o1 row shows identical values (Pt=0.180, Dis=0.710) across all four prompts with zero standard deviation; the authors should explain why the prompt variability had no effect for this model.
  5. [Section 4.1, Appendix A.2] The collection of 50 inpatient cases from a collaborating hospital is described, but no ethics approval or data-governance statement is provided; only manual PHI removal is mentioned. A statement on institutional review and consent would be expected for a clinical benchmark.
  6. [Benchmark Release] The benchmark release is described only as a URL (https://clinbench-hpb.github.io). The paper should specify the license, data format, and whether evaluation scripts and full model outputs will be released to enable reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is grounded in external case sources and physician-validated labels, and the reported comparisons are measurements rather than constructions.

full rationale

The paper's central claim is empirical: LLMs score well on MCQ items but show lower recall on open-ended HPB case diagnosis. The clinical-case reference labels come from external sources (NEJM, JAMA, CNKI, case-sharing websites, and a collaborating hospital) and are reviewed by four licensed physicians, so the main diagnostic scores are anchored outside the paper's own construction pipeline. The MCQ subset draws from public examination datasets, with LLM-based filtering and synthesis used for coverage; however, the reported accuracies are not defined by those filters in any equation-level way, and the paper's main finding is not that any specific model must score at a particular value by construction. The use of DeepSeek-V3-0324 as the strict evaluation model while also being one of the evaluated models is a self-evaluation overlap, but it is not a reduction: the judge's decisions are based on a six-dimensional rubric with physician agreement rates of 97.5% and 97.7%, and the same judge gives itself a low Hospital patient-level recall (0.175), so there is no evidence that the scoring rule forces the reported degradation. The only explicit limitations stated by the authors—text-only modality and stochasticity/expense of LLM-based assessment—concern reliability and scope rather than circularity. The concern that some reference comorbidities may not be inferable from the provided text is a label-validity question, not a circularity of derivation: the evaluation would be miscalibrated if true, but it would not be equivalent to its inputs by definition. Overall, the paper is self-contained against external benchmarks and physician review, so no circular step can be exhibited under the required standard.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

All open-ended scores rest on the completeness and inferability of reference diagnoses and on the validity of an LLM-based judge. The MCQ scores rest on the representativeness and contamination status of public exam datasets plus LLM-synthesized items. No new physical or theoretical entities are introduced, and no target quantity is obtained by fitting numeric parameters to data, but several hand-chosen thresholds and inference settings shape the reported numbers.

free parameters (3)
  • synthetic MCQ retention threshold = scores 1-2 excluded, 3 revised, 4-5 retained
    Chosen by the authors to filter LLM-generated questions; affects the composition of the 1,362 synthetic EN-QA items and therefore the benchmark.
  • option permutation count = min(4, n-1) circular shifts
    Chosen to reduce option-order sensitivity; multiplies inference calls and can change accuracy estimates, especially for high-variance models.
  • reasoning-model temperature = 0.6 for reasoning-enhanced models and DeepSeek-R1; greedy for others
    Hand-selected inference setting for stability; reported scores depend on it, and no sensitivity sweep is shown.
assumptions (5)
  • domain assumption ICD-10 HPB disease coding (33 main categories, 465 subcategories) is a complete and valid taxonomy for HPB disease coverage.
    Used to claim comprehensive coverage in Section 3.1. Manual screening excludes ambiguous entries, so coverage is defined by the authors' screening choices.
  • domain assumption Reference diagnoses in clinical cases are complete, correct, and inferable from the text given to the model.
    The paper relies on final diagnoses including all comorbidities (Section 3.1, Figure 1) and on patient-level recall requiring full spectrum capture (Section 3.2). Physician validation checks correctness but not inferability.
  • domain assumption The two-stage LLM judge, DeepSeek-V3-0324 as strict evaluator and Claude-3.7-Sonnet as check model, is a valid proxy for physician assessment of diagnostic coverage.
    Used for all open-ended scores in Section 3.2. Agreement rates are reported on 393 Chinese and 132 English diseases, not on the full evaluation set, and the strict judge is itself an evaluated model.
  • ad hoc to paper LLM-synthesized multiple-choice questions preserve clinical validity and do not introduce leakage or biased content.
    EN-QA includes 1,362 synthetic items; quality is checked on a random 500-sample, not all items, and generation uses an internet-connected LLM (Section 3.1).
  • domain assumption Public exam datasets (MedQA, MedMCQA, CMB, PubMedQA, MMLU-Pro) measure HPB knowledge meaningfully.
    MCQ subsets are derived from these benchmarks; their scope and contamination status affect the exam-versus-clinical comparison (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases." pith.science (2026). https://pith.science/paper/BMDHWZPO

@misc{pith2026250600095,
  author       = {Pith},
  title        = {Pith review of: ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMDHWZPO}},
  note         = {Machine review of arXiv:2506.00095}
}
read the original abstract

Hepato-pancreato-biliary (HPB) disorders represent a global public health challenge due to their high morbidity and mortality. Although large language models (LLMs) have shown promising performance in general medical question-answering tasks, the current evaluation benchmarks are mostly derived from standardized examinations or manually designed questions, lacking HPB coverage and clinical cases. To address these issues, we systematically eatablish an HPB disease evaluation benchmark comprising 3,535 closed-ended multiple-choice questions and 337 open-ended real diagnosis cases, which encompasses all the 33 main categories and 465 subcategories of HPB diseases defined in the International Statistical Classification of Diseases, 10th Revision (ICD-10). The multiple-choice questions are curated from public datasets and synthesized data, and the clinical cases are collected from prestigious medical journals, case-sharing platforms, and collaborating hospitals. By evalauting commercial and open-source general and medical LLMs on our established benchmark, namely ClinBench-HBP, we find that while commercial LLMs perform competently on medical exam questions, they exhibit substantial performance degradation on HPB diagnosis tasks, especially on complex, inpatient clinical cases. Those medical LLMs also show limited generalizability to HPB diseases. Our results reveal the critical limitations of current LLMs in the domain of HPB diseases, underscoring the imperative need for future medical LLMs to handle real, complex clinical diagnostics rather than simple medical exam questions. The benchmark will be released at https://clinbench-hpb.github.io.

Figures

Figures reproduced from arXiv: 2506.00095 by the authors.

Figure 1
Figure 1. Overview of our established ClinBench-HPB. The benchmark comprises 3,535 multiple [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ClinBench-HPB construction process. (a) Three-level organization of HPB [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of response generation for DeepSeekV3-1226 [51] on the Hospital subset. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Illustration of the prompt templates for the objective multiple-choice questions. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the prompt templates for the strict evaluation model (top) and the check [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the prompt templates for the subjective questions. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Clinical case 1 from the Hospital subset. The original Chinese text has been translated into [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Diagnostic results of DeepSeekV3-0324 [55] and its evaluation. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Clinical case 2 from the Hospital subset (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Clinical case 2 from the Hospital subset (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Diagnostic results of OpenAI-o1 [60]. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Evaluation of the prediction from OpenAI-o1 [60]. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 58 canonical work pages

  1. [1]

    Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma.Clinical and molecular hepatology, 29(3):721, 2023

    Yee Hui Yeo, Jamil S Samaan, Wee Han Ng, Peng-Sheng Ting, Hirsh Trivedi, Aarshi Vipani, Walid Ayoub, Ju Dong Yang, Omer Liran, Brennan Spiegel, et al. Assessing the performance of chatgpt in answering questions regarding cirrhosis and hepatocellular carcinoma.Clinical and molecular hepatology, 29(3):721, 2023

  2. [2]

    Exploring the performance of chatgpt on acute pancreatitis-related questions.Journal of Translational Medicine, 22(1):527, 2024

    Ren-Chun Du, Xing Liu, Yong-Kang Lai, Yu-Xin Hu, Hao Deng, Hui-Qiao Zhou, Nong-Hua Lu, Yin Zhu, and Yi Hu. Exploring the performance of chatgpt on acute pancreatitis-related questions.Journal of Translational Medicine, 22(1):527, 2024

  3. [3]

    Automated identification of incidental hepatic steatosis on emergency department imaging using large language models.Hepatology Communications, 9(3):e0638, 2025

    Tyrus V ong, Nicholas Rizer, Vedant Jain, Valerie L Thompson, Mark Dredze, Eili Y Klein, Jeremiah S Hinson, Tanjala Purnell, Stephen Kwak, Tinsay Woreta, et al. Automated identification of incidental hepatic steatosis on emergency department imaging using large language models.Hepatology Communications, 9(3):e0638, 2025

  4. [4]

    Quality of chatgpt responses to questions related to liver transplantation.Journal of Gastrointestinal Surgery, 27(8):1716–1719, 2023

    Yutaka Endo, Kazunari Sasaki, Zorays Moazzam, Henrique A Lima, Austin Schenk, Ashley Limkemann, Kenneth Washburn, and Timothy M Pawlik. Quality of chatgpt responses to questions related to liver transplantation.Journal of Gastrointestinal Surgery, 27(8):1716–1719, 2023

  5. [5]

    Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework.NPJ digital medicine, 7(1):102, 2024

    Simone Kresevic, Mauro Giuffrè, Milos Ajcevic, Agostino Accardo, Lory S Crocè, and Dennis L Shung. Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework.NPJ digital medicine, 7(1):102, 2024

  6. [6]

    Development of a liver disease–specific large language model chat interface using retrieval- augmented generation.Hepatology, 80(5):1158–1168, 2024

    Jin Ge, Steve Sun, Joseph Owens, Victor Galvez, Oksana Gologorskaya, Jennifer C Lai, Mark J Pletcher, and Ki Lai. Development of a liver disease–specific large language model chat interface using retrieval- augmented generation.Hepatology, 80(5):1158–1168, 2024

  7. [7]

    Nicholas Dietrich and Brett Stubbert. Evaluating adherence to canadian radiology guidelines for incidental hepatobiliary findings using rag-enabled llms.Canadian Association of Radiologists Journal, page 08465371251323124, 2025

  8. [8]

    A global assessment of recent trends in gastrointestinal cancer and lifestyle-associated risk factors.Cancer Communications, 41(11):1137–1151, 2021

    Lili Lu, Christina S Mullins, Clemens Schafmayer, Sebastian Zeißig, and Michael Linnebacher. A global assessment of recent trends in gastrointestinal cancer and lifestyle-associated risk factors.Cancer Communications, 41(11):1137–1151, 2021

Show all 82 references
  1. [9]

    Burden and cost of gastrointestinal, liver, and pancreatic diseases in the united states: update 2024.Gastroenterology, 2025

    Anne F Peery, Caitlin C Murphy, Chelsea Anderson, Elizabeth T Jensen, Sasha Deutsch-Link, Matthew D Egberg, Jennifer L Lund, Disha Subramaniam, Evan S Dellon, Ami D Sperber, et al. Burden and cost of gastrointestinal, liver, and pancreatic diseases in the united states: update...

  2. [10]

    Large language models in gastroenterology and gastrointestinal surgery: A new frontier in patient communication and education.Gastroenterology Research, 18(2):39, 2025

    Dushyant Singh Dahiya, Hassam Ali, Vishali Moond, M Danial Ali Shah, Christina Santana, Noor Ali, Abu Baker Sheikh, Muhammad Ahmad Nadeem, Aqsa Munir, Mohammed A Quazi, et al. Large language models in gastroenterology and gastrointestinal surgery: A new frontier in patient com...

  3. [11]

    Large language models in gastroenterology: Systematic review.Journal of medical Internet research, 26:e66648, 2024

    Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee, Jonghyung Park, Eunsil Kim, Subeen Kim, Minjae Kimm, and Seoung-Ho Choi. Large language models in gastroenterology: Systematic review.Journal of medical Internet research, 26:e66648, 2024

  4. [12]

    Lkan: Llm-based knowledge-aware attention network for clinical staging of liver cancer.IEEE Journal of Biomedical and Health Informatics, 2024

    Ya Li, Xuecong Zheng, Jiaping Li, Qingyun Dai, Chang-Dong Wang, and Min Chen. Lkan: Llm-based knowledge-aware attention network for clinical staging of liver cancer.IEEE Journal of Biomedical and Health Informatics, 2024

  5. [13]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

  6. [14]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

  7. [15]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 10

  8. [16]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

  9. [17]

    Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes.Liver International, 44(9):2114–2124, 2024

    Mauro Giuffrè, Simone Kresevic, Nicola Pugliese, Kisung You, and Dennis L Shung. Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes.Liver International, 44(9):2114–2124, 2024

  10. [18]

    Ruben Geevarghese, Carlie Sigel, John Cadley, Subrata Chatterjee, Pulkit Jain, Alex Hollingsworth, Avijit Chatterjee, Nathaniel Swinburne, Khawaja Hasan Bilal, and Brett Marinelli. Extraction and classification of structured data from unstructured hepatobiliary pathology repor...

  11. [19]

    Using a customized gpt to provide guideline-based recommendations for management of pancreatic cystic lesions

    Yuri Gorelik, Itai Ghersin, Tarek Arraf, Offir Ben-Ishay, Amir Klein, and Iyad Khamaysi. Using a customized gpt to provide guideline-based recommendations for management of pancreatic cystic lesions. Endoscopy International Open, 12(04):E600–E603, 2024

  12. [20]

    Nicola Pugliese, Vincent Wai-Sun Wong, Jörn M Schattenberg, Manuel Romero-Gomez, Giada Sebastiani, Laurent Castera, Cesare Hassan, Pinelopi Manousou, Luca Miele, Raquel Peck, et al. Accuracy, reliability, and comprehensibility of chatgpt-generated medical responses for patient...

  13. [21]

    Accuracy of information provided by chatgpt regarding liver cancer surveillance and diagnosis.American Journal of Roentgenology, 221(4):556–559, 2023

    Jennie J Cao, Daniel H Kwon, Tara T Ghaziani, Paul Kwo, Gary Tse, Andrew Kesselman, Aya Kamaya, and Justin R Tse. Accuracy of information provided by chatgpt regarding liver cancer surveillance and diagnosis.American Journal of Roentgenology, 221(4):556–559, 2023

  14. [22]

    Using large language models in the diagnosis of acute cholecystitis: Assessing accuracy and guidelines compliance.The American Surgeon™, page 00031348251323719, 2025

    Marta Goglia, Arianna Cicolani, Francesco Maria Carrano, Niccolò Petrucciani, Francesco D’Angelo, Marco Pace, Lucio Chiarini, Gianfranco Silecchia, and Paolo Aurello. Using large language models in the diagnosis of acute cholecystitis: Assessing accuracy and guidelines complia...

  15. [23]

    Tianyang Mao, Xin Zhao, Kangyi Jiang, Qingyun Xie, Manyu Yang, Ruoxuan Wang, and Fengwei Gao. A comparison of the responses between chatgpt and doctors in the field of cholelithiasis based on clinical practice guidelines: a cross-sectional study.Digital Health, 11:205520762513...

  16. [24]

    Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge.Physics in Medicine and Biology, 2024

    Xuzhou Wu, Guangxin Li, Xing Wang, Zeyu Xu, Yingni Wang, Shuge Lei, Jianming Xian, Xueyu Wang, Yibao Zhang, Gong Li, et al. Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge.Physics in Medicine and Biology, 2024

  17. [25]

    Icd-10.https://code.nhsa.gov.cn/search.html?sysflag=80

  18. [26]

    Icd-10 version: 2019.https://icd.who.int/browse10/2019/en

  19. [27]

    Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications.Advances in Neural Information Processing Systems, 37:138632–138662, 2024

    Dingkang Yang, Jinjie Wei, Dongling Xiao, Shunli Wang, Tong Wu, Gang Li, Mingcheng Li, Shuaibing Wang, Jiawei Chen, Yue Jiang, et al. Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications.Advances in Neural Information Processing Systems...

  20. [28]

    Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023

  21. [29]

    Icga-gpt: report generation and question answering for indocyanine green angiography images.British Journal of Ophthalmology, 108(10):1450–1456, 2024

    Xiaolan Chen, Weiyi Zhang, Ziwei Zhao, Pusheng Xu, Yingfeng Zheng, Danli Shi, and Mingguang He. Icga-gpt: report generation and question answering for indocyanine green angiography images.British Journal of Ophthalmology, 108(10):1450–1456, 2024

  22. [30]

    Chatffa: interactive visual question answering on fundus fluorescein angiography image using chatgpt

    Xiaolan Chen, Pusheng Xu, Yao Li, Weiyi Zhang, Fan Song, Ying-Feng Zheng, Danli Shi, and Mingguang He. Chatffa: interactive visual question answering on fundus fluorescein angiography image using chatgpt. Available at SSRN 4578568, 2023

  23. [31]

    Compar- ative experimentation of accuracy metrics in automated medical reporting: the case of otitis consultations

    Wouter Faber, Renske Eline Bootsma, Tom Huibers, Sandra van Dulmen, and Sjaak Brinkkemper. Compar- ative experimentation of accuracy metrics in automated medical reporting: the case of otitis consultations. arXiv preprint arXiv:2311.13273, 2023

  24. [32]

    Medredqa for medical consumer question answering: Dataset, tasks, and neural baselines

    Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, and Zhenchang Xing. Medredqa for medical consumer question answering: Dataset, tasks, and neural baselines. InProceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the As...

  25. [33]

    Medic: Towards a comprehensive framework for evaluating llms in clinical applications.arXiv preprint arXiv:2409.07314, 2024

    Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. Medic: Towards a comprehensive framework for evaluating llms in clinical applications.arXiv preprint arXiv:2409.0...

  26. [34]

    Pediabench: A comprehensive chinese pediatric dataset for benchmarking large language models

    Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, and Yanhao Wang. Pediabench: A comprehensive chinese pediatric dataset for benchmarking large language models. arXiv preprint arXiv:2412.06287, 2024

  27. [35]

    Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency?arXiv preprint arXiv:2404.06503, 2024

    Nathan Brake and Thomas Schaaf. Comparing two model designs for clinical note generation; is an llm a useful evaluator of consistency?arXiv preprint arXiv:2404.06503, 2024

  28. [36]

    Towards leveraging large language models for automated medical q&a evaluation.arXiv preprint arXiv:2409.01941, 2024

    Jack Krolik, Herprit Mahal, Feroz Ahmad, Gaurav Trivedi, and Bahador Saket. Towards leveraging large language models for automated medical q&a evaluation.arXiv preprint arXiv:2409.01941, 2024

  29. [37]

    Large language models for automated synoptic reports and resectability categorization in pancreatic cancer.Radiology, 311(3):e233117, 2024

    Rajesh Bhayana, Bipin Nanda, Taher Dehkharghanian, Yangqing Deng, Nishaant Bhambra, Gavin Elias, Daksh Datta, Avinash Kambadakone, Chaya G Shwaartz, Carol-Anne Moulton, et al. Large language models for automated synoptic reports and resectability categorization in pancreatic c...

  30. [38]

    Ultramedical: Building specialized generalists in biomedicine.Advances in Neural Information Processing Systems, 37:26045–26081, 2024

    Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. Ultramedical: Building specialized generalists in biomedicine.Advances in Neural Information Processing Systems, 37:26045–26081, 2024

  31. [39]

    Openbiollms: Advancing open- source large language models for healthcare and life sciences.Hugging Face repository, 2024

    Malaikannan Sankarasubbu Ankit Pal and Malaikannan Sankarasubbu. Openbiollms: Advancing open- source large language models for healthcare and life sciences.Hugging Face repository, 2024

  32. [40]

    O1 replication journey–part 3: Inference-time scaling for medical reasoning.arXiv preprint arXiv:2501.06458, 2025

    Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. O1 replication journey–part 3: Inference-time scaling for medical reasoning.arXiv preprint arXiv:2501.06458, 2025

  33. [41]

    m1: Unleash the potential of test-time scaling for medical reasoning with large language models.arXiv preprint arXiv:2504.00869, 2025

    Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models.arXiv preprint arXiv:2504.00869, 2025

  34. [42]

    Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024

  35. [43]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

  36. [44]

    Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning, pages 248–260. PMLR, 2022

  37. [45]

    Cmb: A comprehensive medical benchmark in chinese.arXiv preprint arXiv:2308.08833, 2023

    Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. Cmb: A comprehensive medical benchmark in chinese.arXiv preprint arXiv:2308.08833, 2023

  38. [46]

    Benchmarking large language models on answering and explaining challenging medical questions

    Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics...

  39. [47]

    Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019

  40. [48]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

  41. [49]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

  42. [50]

    It’s time to bench the medical exam benchmark, 2025

    Inioluwa Deborah Raji, Roxana Daneshjou, and Emily Alsentzer. It’s time to bench the medical exam benchmark, 2025. 12

  43. [51]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

  44. [52]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

  45. [53]

    https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    Gpt-4o-mini. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

  46. [54]

    Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023

  47. [55]

    Deepseek-v3-0324.https://huggingface.co/deepseek-ai/DeepSeek-V3-0324

  48. [56]

    Claude 3.7 sonnet.https://www.anthropic.com/claude/sonnet

  49. [57]

    Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet

  50. [58]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

  51. [59]

    Qwen2.5-max.https://qwenlm.github.io/zh/blog/qwen2.5-max/

  52. [60]

    Openai-o1.https://openai.com/o1/

  53. [61]

    Openai o3-mini.https://openai.com/index/openai-o3-mini/

  54. [62]

    Gemini 2.5 pro.https://deepmind.google/technologies/gemini/pro/

  55. [63]

    Qwq-32b.https://qwenlm.github.io/blog/qwq-32b/

  56. [64]

    {disease name}

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, pa...

  57. [65]

    Anatomical specificity: The affected organ/structure must be clearly specified (e.g., cholecystitis and cholangitis cannot be considered as included)

  58. [66]

    Pathological mechanism: Distinguish between inflammation/obstruction/tumor/thrombosis and other essential differences (e.g., portal vein tumor thrombus and portal vein thrombosis cannot be considered as matching)

  59. [67]

    Etiology tracing: Distinguish between causes such as stones/infection/post-surgery/tumor (e.g., obstructive jaundice and gallbladder stones cannot be considered as matching)

  60. [68]

    Temporal characteristics: Acute/chronic course must be clearly indicated (e.g., acute cholecystitis and chronic cholecystitis cannot be considered as matching)

  61. [69]

    Test identification: Laboratory/imaging results must be the same to be considered the same disease (e.g., segmental pneumonia and lung nodules cannot be considered as matching)

  62. [70]

    Includes/Does not include

    Treatment strategy: The same intervention methods must be considered the same disease (e.g., cirrhosis with portal hypertension and portal vein tumor thrombus cannot be considered as matching). The output format is: { "Includes/Does not include": "Brief description of the basi...

  63. [71]

    2.Combine your professional knowledge to infer possible diseases or etiologies

    Thoroughly analyze all provided patient information, including chief complaint, medical history, physical examination results, laboratory tests, and imaging studies. 2.Combine your professional knowledge to infer possible diseases or etiologies. 3.Ensure no information is over...

  64. [72]

    The current patient information is as follows: {clinical case information} Figure 6: Illustration of the prompt templates for the subjective questions

    Provide a preliminary diagnosis with a brief explanation of the diagnostic basis. The current patient information is as follows: {clinical case information} Figure 6: Illustration of the prompt templates for the subjective questions. 16 the questions from both CN-QA and EN-QA ...

  65. [73]

    Gallbladder Adenomyomatosis (with Septation) and Biliary Sludge. Basis for Diagnosis: Imaging Findings: Abdominal ultrasound demonstrates septated gallbladder with sonographic features of adenomyomatosis, along with linear hyperechoic foci (sludge). CT reveals gallbladder fold...

  66. [74]

    Diagnostic Basis: Laboratory findings: Elevated serum cholesterol (exact values unspecified) documented as abnormal in medical reports

    Hypercholesterolemia. Diagnostic Basis: Laboratory findings: Elevated serum cholesterol (exact values unspecified) documented as abnormal in medical reports. A well-established risk factor for gallstones, particularly cholesterol gallstones

  67. [75]

    Diagnostic Basis: Imaging findings: Cervical spine examination at Center A clearly demonstrates osteophyte formation, indicative of degenerative changes

    Cervical Osteophyte Formation. Diagnostic Basis: Imaging findings: Cervical spine examination at Center A clearly demonstrates osteophyte formation, indicative of degenerative changes. Asymptomatic presentation, but radiographic evidence is definitive

  68. [76]

    adenomyomatosis of the gallbladder (with septation) accompanied by gallstones (sludge-like stones)

    Chronic Inflammation in Right Middle Lobe/Upper Lobe Fibrotic Lesions (Bilateral). Diagnostic Basis: Imaging Findings: Chest examination at Center A revealed chronic inflammatory changes and fibrotic foci. Subsequent CT at Hospital C showed no active lesions, suggesting residu...

  69. [77]

    MRI impression: A space-occupying lesion in the left lateral lobe of the liver is considered to be hepatocellular carcinoma

    Multiple small cysts are detected in both kidneys during the scan. MRI impression: A space-occupying lesion in the left lateral lobe of the liver is considered to be hepatocellular carcinoma. Multiple cysts are found in the liver. The volume of the gallbladder is increased, an...

  70. [78]

    fast in and fast out

    Hepatocellular Carcinoma (HCC) Diagnostic bases: • Extremely elevated tumor markers: Alpha-fetoprotein (AFP) is far higher than the normal value (>100,000 μg/L), and Protein Induced by Vitamin K Absence or Antagonist-II (PIVKA-II) is also significantly elevated. The simultaneo...

  71. [79]

    Cholecystitis Diagnostic bases: • Imaging findings: Both CT and MR show an enlarged gallbladder volume, slightly thickened and enhanced gallbladder wall, which are consistent with the changes of cholecystitis. • Clinical manifestations: Discomfort in the right upper abdomen an...

  72. [80]

    These are commonly found during physical examinations

    Multiple cysts in the liver and both kidneys Diagnostic bases: • Imaging: Multiple round-like low-density (on CT) or long T2 (on MR) non-enhancing lesions are visible in the liver and both kidneys, with clear boundaries, which are in line with the characteristics of cysts. The...

  73. [81]

    Abnormal lipid metabolism (hypercholesterolemia, low HDL cholesterolemia, etc.) Diagnostic bases: • Laboratory tests: Total cholesterol (TC) is 6.16 mmol/L (↑), high-density lipoprotein (HDL) is only 0.84 mmol/L (↓), and lipoprotein (a) and apolipoprotein B are on the high sid...

  74. [82]

    Included

    Other changes that require attention • Mild abnormalities in liver function and biliary enzyme spectrum: The elevation of indicators such as ALT, AST, GGT, and ALP suggests damage to liver cells and the biliary tract; some of these may be caused by liver cancer itself or concu...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.