{"id":"ccbc20a2-9614-4e4e-943b-297e7a2d1e1d","arxiv_id":"2506.00095","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark covering all ICD-10 HPB disease categories shows that LLMs, including specialized medical models, perform far worse on real clinical cases than on exam-style questions.","lead":"ClinBench-HPB is a new benchmark with 3,535 HPB multiple-choice questions and 337 real clinical cases spanning all ICD-10 HPB categories. It tests 26 LLMs and finds that commercial models do well on exam questions but miss many diagnoses, especially on complex inpatient cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference diagnosis lists may include comorbidities not inferable from the input text, so low patient-level recall could overstate the diagnostic gap.","rationale":"The reader's weakest-assumption analysis correctly identifies the completeness and inferability of the reference diagnosis lists as the load-bearing premise. The paper's own Figure 1 supplies a concrete example where the final diagnosis contains items not visible in the provided clinical text as excerpted, and Section 3.2's patient-level recall makes full-spectrum recovery the success criterion. I agree that this can inflate the reported MCQ-versus-clinical gap. However, this is an artifact risk rather than an established error: physician validation was performed, and the LLM-as-judge agreement rates are reported, so the benchmark may still be sound once reference-label inferability is checked. The data and code are not yet released, so the check cannot be performed from the paper alone. For that reason, keeping the CONDITIONAL verdict is appropriate; the concern does not warrant REJECT absent evidence that the labels are wrong, nor ACCEPT until the label-inferability audit is run. I found no stronger internal inconsistency: the MCQ filtering, the option-shuffling protocol, and the four-prompt averaging are described in sufficient detail, and the overlap between HuatuoGPT-o1 training data and the benchmark is explicitly acknowledged. The absence of a human baseline is a secondary limitation that the label-inferability test would partially address, since a clinician baseline would also reveal whether the reference lists are recoverable by experts from the same inputs.","tokens_in":29278,"tokens_out":2988,"duration_ms":41638,"concrete_test":"Take a random sample of 50 cases stratified across Journal, Website, and Hospital, and have independent clinicians mark each reference diagnosis as either (a) inferable from the patient information, labs, and imaging text supplied to the models, or (b) requiring external knowledge, full-chart information, or incidental findings not present in the prompt. Recompute disease-level and patient-level recall for a few representative models (e.g., DeepSeekV3-0324, GPT-4o, Qwen2.5-72B) using only the 'inferable' reference diagnoses. If Hospital patient-level recall rises substantially above 0.22, the headline performance gap is partly an artifact of the reference labels; if it remains below roughly 0.3, the central claim survives this critique.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LLMs degrade from MCQ accuracy to substantially weaker HPB clinical diagnosis—depends on the reference diagnosis list being complete, correct, and inferable from the text actually given to each model. In Section 3.1, 'Final Diagnosis' deliberately includes primary conditions and comorbidities, and Figure 1's example lists coronary atherosclerosis, duodenal diverticulum, pulmonary bullae, and segmental pneumonia alongside the HPB primary diagnosis. Yet the shown patient information and imaging impressions do not provide explicit evidence for several of these items; if similar non-inferable comorbidities appear in other cases, then a model cannot be credited for naming them and is penalized for omitting them. Section 3.2 makes this decisive: patient-level recall is satisfied only when the model captures the full spectrum of concurrent diagnoses, and the Hospital subset averages 7.86 diagnoses per case. The reported Hospital patient-level recalls of 0.09–0.22, contrasted with MCQ accuracy near 0.8, therefore conflates true diagnostic misses with impossible-to-derive reference labels. Because disease-level recall also counts every reference disease, both headline metrics are affected, not just the stricter patient-level criterion. The absence of a human physician baseline compounds this: without knowing how often clinicians recover the same reference list from the same input, 'substantial performance degradation' is not calibrated to clinical feasibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ClinBench-HPB, a benchmark for hepato-pancreato-biliary (HPB) diseases containing 3,535 multiple-choice questions (CN-QA: 2,000 Chinese; EN-QA: 1,535 English) and 337 open-ended clinical cases (120 from journals, 167 from case-sharing websites, 50 from a collaborating hospital). The authors evaluate 26 LLMs across commercial, open-source general-purpose, medical, and reasoning-enhanced categories. The central claim is that LLMs achieve high MCQ accuracy (up to 0.84) but exhibit substantial performance degradation on clinical case diagnosis, particularly on inpatient cases with an average of 7.86 diagnoses per case (patient-level recall 0.09–0.22). The paper also claims that medical LLMs underperform general-purpose models on HPB diagnosis, indicating limited generalizability.","tokens_in":29623,"tokens_out":12339,"duration_ms":127707,"significance":"If the central claim holds, ClinBench-HPB would be a valuable resource: it provides broad ICD-10 coverage, a large set of clinically sourced cases, a two-stage LLM-as-judge evaluation with physician agreement, and a systematic comparison of 26 models. The construction pipeline is described in detail, and the authors report physician validation of 500 synthetic MCQs (96% quality) and judge agreement of 97.5% and 97.7% on sampled diseases. However, the benchmark's main conclusion rests on the assumption that the reference diagnosis lists are complete, correct, and inferable from the input text. The example in Figure 1 raises a concrete concern, and the lack of a human baseline is a critical omission.","major_comments":[{"comment":"The reference 'Final Diagnosis' lists comorbidities that are not inferable from the provided patient information. In the Figure 1 example, the reference list includes coronary atherosclerosis and duodenal diverticulum, but the patient information, laboratory tests, and imaging findings shown do not provide evidence for these conditions. Since the disease-level and patient-level recall metrics in Section 3.2 count every reference disease, a model that omits non-inferable items is penalized as if it made a diagnostic error. With an average of 7.86 diagnoses per Hospital case, the reported patient-level recalls of 0.09–0.22 and the contrast with MCQ accuracy near 0.8 conflate true diagnostic misses with impossible-to-derive reference labels. The paper should either restrict reference lists to conditions that are actually supported by the input text, or report a human-physician baseline showing how often clinicians can recover the same list from the same input.","section":"Section 3.1, Figure 1, Section 3.2"},{"comment":"The diagnostic prompts instruct the model to 'list all major diseases,' while the reference lists include incidental findings and chronic comorbidities such as pulmonary bullae, coronary atherosclerosis, and duodenal diverticulum (Figure 1). This is a mismatch between the task instruction and the evaluation criterion. A model that deliberately focuses on clinically significant HPB conditions is penalized for not listing incidental imaging findings. The authors should align the reference construction with the prompt (e.g., by scoring only diseases that are clinically relevant to the HPB presentation or that are explicitly inferable from the input) or change the prompt to ask for all documented conditions.","section":"Section 3.2, Figure 6"},{"comment":"The reported agreement rates (97.5% on 393 Chinese diseases and 97.7% on 132 English diseases) validate the automated judge against physician judgments on the judge's decisions, but they do not validate the completeness or correctness of the reference lists themselves. The sample selection (which cases, how many physicians, inter-rater agreement) is not described. Moreover, DeepSeek-V3-0324 is both the strict evaluation model and one of the evaluated models in Table 3; while the check model (Claude-3.7-Sonnet) and the physician agreement mitigate self-favoritism, the paper should report whether the model's relative ranking changes when an independent judge is used.","section":"Section 3.2, Appendix A.1.2"},{"comment":"The combined metric Avg = (Avgq + Avgp)/2 gives equal weight to MCQ accuracy and patient-level clinical recall, and the claims that open-source models are 'approaching commercial models' and that reasoning-enhanced LLMs 'show promise' are based on this arbitrary weighting. For instance, Llama3.1-70B (Avg 0.601) is reported as outperforming GPT-4o (Avg 0.595), but GPT-4o is superior on both MCQ subsets (Avgq 0.722 vs 0.809). The paper should present task-specific results and, if a combined metric is used, justify the weighting and report statistical significance.","section":"Table 3, Section 4.2"},{"comment":"The paper claims that the benchmark encompasses all 33 main categories and 465 subcategories of HPB diseases defined in ICD-10, but no coverage mapping is provided. The construction process (keyword filtering, LLM-based selection, synthetic generation) is described qualitatively, and Table 2 gives only aggregate counts per subset. Without a table or figure mapping the 3,535 MCQs and 337 cases to the ICD-10 codes (or at least to the 33 Level-1 categories), this central claim is unverifiable. The authors should include a coverage analysis as part of the benchmark release.","section":"Section 3.1"}],"minor_comments":[{"comment":"There are numerous typos, including 'eatablish' (abstract), 'evalauting' (abstract), 'diganostic' (Section 1), 'multi-scource' (Section 1), 'improvment' (Section 4.2), 'Hostipal' (Table 3), 'evalaution' (Section 5), and 'inherents' (Section 5). The title and abstract should also use a consistent spelling of the benchmark name ('ClinBench-HPB' vs 'ClinBench-HBP').","section":"Throughout"},{"comment":"The number of synthesized questions is reported as 'approximately 1,400' and later as '1,362 questions generated.' These numbers should be reconciled.","section":"Section 3.1"},{"comment":"The EN-QA subset is 1,535 items, described as 173 items from public datasets and 1,362 synthetic items, but no per-source counts are given for the public datasets (MedQA-English, MedMCQA, PubMedQA, MMLU-Pro). Providing a breakdown would improve transparency.","section":"Section 3.1"},{"comment":"The tables are hard to read because columns are packed without visual separation and the header row is ambiguous. In Table 7, the OpenAI-o1 row shows identical values (Pt=0.180, Dis=0.710) across all four prompts with zero standard deviation; the authors should explain why the prompt variability had no effect for this model.","section":"Table 3, Appendix Tables 4-7"},{"comment":"The collection of 50 inpatient cases from a collaborating hospital is described, but no ethics approval or data-governance statement is provided; only manual PHI removal is mentioned. A statement on institutional review and consent would be expected for a clinical benchmark.","section":"Section 4.1, Appendix A.2"},{"comment":"The benchmark release is described only as a URL (https://clinbench-hpb.github.io). The paper should specify the license, data format, and whether evaluation scripts and full model outputs will be released to enable reproducibility.","section":"Benchmark Release"}],"recommendation":"major_revision","confidential_remarks":"The core concern is that the reference diagnosis lists may contain non-inferable comorbidities, which would systematically depress patient-level recall. The Figure 1 example is a concrete instance. I recommend asking for a human-physician baseline on a sample of cases, and a sensitivity analysis where recall is recomputed on a restricted reference list (e.g., only conditions explicitly supported by the input text). If the degradation persists under those conditions, the paper's central claim would be much stronger. Also, the paper cites the benchmark website but the dataset is not yet available; the editor should verify that the release plan is real."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious benchmark resource for HPB medical LLM evaluation, worth refereeing, but the headline degradation numbers are probably softer than reported because the reference diagnosis lists contain items that cannot be derived from the input text, and there is no human physician baseline to calibrate what is achievable.\n\nWhat's actually new: ClinBench-HPB covers all 33 main and 465 subcategories of ICD-10 HPB diseases with 3,535 MCQs and 337 real cases, evaluated across 26 LLMs. That is a real step up from single-disease evaluations with at most 493 questions. The pipeline is described in enough detail to be reproducible in principle, and the small physician-validated samples (96% MCQ quality, 97.7% judge agreement) are honest evidence that the construction was checked by humans.\n\nThe soft spots are not fatal but they do matter. The stress-test concern holds up. Section 3.1 defines Final Diagnosis as primary conditions plus comorbidities, and Figure 1's example lists coronary atherosclerosis, duodenal diverticulum, pulmonary bullae, and segmental pneumonia alongside the HPB primary diagnosis, but the provided patient information and imaging impressions do not give explicit evidence for several of those. A model cannot be credited for naming a disease that is not inferable from the text, and it is penalized for omitting it. Since Hospital cases average 7.86 reference diagnoses per patient, low patient-level recall (0.09–0.22) conflates true misses with impossible inferences. This is a real measurement concern, not a manufactured one.\n\nTwo smaller issues compound it. DeepSeek-V3-0324 serves as both strict judge and evaluated model, which is a mild circularity for that model's scores, though the check model is Claude-3.7-Sonnet and the effect is limited to one row in the table. And there is no human physician baseline: without knowing how often clinicians recover the same reference list from the same input, \"substantial performance degradation\" is not calibrated to clinical feasibility.\n\nThat said, the qualitative finding does not collapse. Disease-level recall on the Hospital subset is also well below MCQ accuracy (e.g., 0.598 for GPT-4o), so the exam-vs-clinical gap is not an artifact of the patient-level criterion alone. The degradation claim survives, but the reported magnitudes should be treated with caution. The data and code are only promised, not released, which makes independent verification impossible right now.\n\nWho is this for: anyone building or evaluating medical LLMs, and HPB clinicians interested in LLM capabilities. It deserves a serious referee. If I were the editor, I would send it to review with two requirements: release the benchmark artifacts, and add a human clinician baseline plus a per-case audit of whether each reference diagnosis is inferable from the available text.","headline":"A serious HPB benchmark worth refereeing, but the headline degradation numbers are probably overstated by non-inferable reference diagnoses and the lack of a human baseline.","tokens_in":30097,"tokens_out":2345,"would_cite":true,"duration_ms":26120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Commercial LLMs score well on HPB exams but miss most concurrent diagnoses in real inpatient cases, and medical LLMs fail to generalize to HPB diseases.","keywords":["LLM evaluation","hepatopancreatobiliary diseases","clinical diagnosis benchmark","ICD-10","patient-level recall","medical LLMs","LLM-as-a-judge","out-of-distribution generalization"],"falsifier":"If re-scoring the 50 hospital cases using only each patient's primary diagnosis, dropping incidental comorbidities such as pulmonary bullae or coronary atherosclerosis, erased most of the gap between MCQ accuracy and patient-level recall, the claim that LLMs fail at real HPB diagnosis would be seriously weakened. A second check would be to have experienced clinicians independently list all diagnoses from the same case text and compare their recall to the LLMs' recall.","tokens_in":1675,"feed_emoji":"🩺","tokens_out":2630,"duration_ms":109814,"temperature":0.7,"pith_summary":"ClinBench-HPB is a benchmark built to test whether LLMs can actually diagnose liver, pancreas, and biliary (HPB) diseases, not just answer exam questions. It pairs 3,535 multiple-choice questions covering all ICD-10 HPB categories with 337 open-ended real clinical cases from journals, case-sharing sites, and a collaborating hospital. Across 26 LLMs, the paper finds that exam accuracy is high, with the best commercial models averaging about 0.84, while clinical case diagnosis is much weaker; on the 50 inpatient hospital cases, patient-level recall, which requires naming every concurrent diagnosis, falls to roughly 0.09–0.22 for commercial models. Medical LLMs fine-tuned on exam data perform worse on these out-of-distribution cases than their general-purpose base models. The paper concludes that current LLM diagnostic skill is overestimated by medical-exam benchmarks and that future medical LLMs should be evaluated on real clinical cases.","feed_headline":"LLMs pass HPB exams, flunk real diagnosis","feed_subtitle":"Benchmark of 3,535 questions and 337 clinical cases shows exam scores overstate diagnostic skill.","key_machinery":"ClinBench-HPB is the central object: 3,535 multiple-choice questions (2,000 Chinese, 1,535 English) drawn from public exam datasets plus synthesized items, and 337 open-ended clinical cases from journals, case-sharing websites, and a collaborating hospital, all mapped to the 33 main and 465 subcategories of ICD-10 HPB diseases. The scoring mechanism is a two-stage LLM-as-a-judge, with DeepSeek-V3-0324 as the strict evaluator deciding whether a prediction includes each reference disease on six dimensions—anatomical specificity, pathological mechanism, etiological origin, temporal characteristics, test identification, and treatment strategy—and Claude-3.7-Sonnet as the check model to catch false negatives from rigid matching. Performance is reported as disease-level recall and patient-level recall, where patient-level recall requires naming every concurrent diagnosis; four distinct prompts and circular option shifts are used to smooth over input sensitivity.","core_discovery":"The paper's central claim is that LLM performance on HPB medicine is substantially overestimated by closed-ended medical-exam benchmarks. In the benchmark's MCQ portion, leading commercial models reach average accuracies around 0.84, which would suggest strong mastery of HPB knowledge; in the open-ended diagnosis portion, disease-level recall is moderate but patient-level recall collapses, and on the 50 hospital-collected inpatient cases even the best commercial model (DeepSeekV3-1226) captures the full spectrum of concurrent diagnoses in only 22% of patients. The paper further argues that medical LLMs fine-tuned on exam-style data degrade on HPB cases outside their training distribution, with all four HuatuoGPT-o1 models underperforming their base models on website and hospital subsets, while reasoning-enhanced LLMs show the most transfer to diagnosis as they scale. The benchmark's design is the evidence: cases are organized by ICD-10 categories, each reference diagnosis is checked against the model's prediction along six clinical dimensions, and results are averaged over four prompt variants.","pith_inferences":["A test worth running is whether experienced clinicians, given the same case text, also fail to list every comorbidity; if they do, the patient-level recall gap reflects task ambiguity more than model deficiency.","The two-model judge could be audited by physicians on all 337 cases rather than sampled subsets; the reported 97.5–97.7% agreement may not hold uniformly for rare comorbidities.","Because the hospital cases are text-only, adding the original imaging could change results in either direction, since the CT and MRI impressions already contain some disease names.","The same benchmark construction could be applied to other organ systems to see whether the MCQ-to-clinical gap is a general property of LLMs or specific to HPB complexity."],"forward_implications":["Medical-exam accuracy should not be read as clinical diagnostic competence, because the same models that reach about 0.84 average accuracy on MCQs drop to low patient-level recall on inpatient HPB cases.","Hospital-style HPB cases with multiple comorbidities are the hard setting where current LLMs miss diagnoses that can change management, so deployment in that setting needs additional safeguards.","Fine-tuning medical LLMs on exam-style data does not guarantee transfer, and the HuatuoGPT-o1 models underperforming their general-purpose bases on website and hospital cases shows a concrete out-of-distribution risk.","Reasoning-enhanced LLMs trained on math and code transfer some capability to diagnosis as they scale, suggesting reasoning training as a plausible path when medical data are scarce.","The released benchmark gives a standardized, ICD-10-organized test for future HPB LLMs, covering all 33 main categories and 465 subcategories of HPB disease."],"supporting_citations":[{"why":"Supplies the ICD-10 coding scheme that defines the benchmark's 33 main categories and 465 subcategories.","marker":"[25]"},{"why":"Provides one of the public medical exam datasets filtered for HPB questions and part of the training/exam overlap for medical LLMs.","marker":"[43]"},{"why":"Source of English MCQ items and another benchmark whose test questions overlap with medical LLM training.","marker":"[44]"},{"why":"The medical LLM family whose out-of-distribution degradation on HPB cases is a central finding.","marker":"[42]"},{"why":"Serves as the strict evaluation model in the two-stage LLM-as-a-judge scoring.","marker":"[55]"},{"why":"Serves as the check model that overrides false-negative judgments from the strict evaluator.","marker":"[56]"},{"why":"Provides the reasoning-enhanced LLM family whose transfer to clinical diagnosis is compared against baselines.","marker":"[58]"},{"why":"Provides the Qwen2.5 base models and tokenizer used across model categories and data preprocessing.","marker":"[13]"}],"fun_headline_variants":["HPB exam scores don't predict real diagnostic skill","LLM diagnostic accuracy plunges on complex HPB cases","New HPB benchmark: exam success ≠ clinical competence","On 50 inpatient HPB cases, top LLM fully diagnoses just 22%","Medical LLMs trained on exams fail HPB diagnosis test"],"cache_read_input_tokens":32128,"weakest_assumption_plain":"The benchmark's central result depends on the reference diagnosis list for each case being complete, correct, and derivable from the text a model sees, so that a model failing to name any listed comorbidity is counted as a diagnostic error.","fun_headline_variants_meta":{"raw":{"variants":["HPB exam scores don't predict real diagnostic skill","LLM diagnostic accuracy plunges on complex HPB cases","New HPB benchmark: exam success ≠ clinical competence","On 50 inpatient HPB cases, top LLM fully diagnoses just 22%","Medical LLMs trained on exams fail HPB diagnosis test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1876,"prompt_tokens":1038,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":753}},"tokens_in":654,"tokens_out":838,"duration_ms":10888,"temperature":1.0,"reasoning_tokens":753,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:21:18.624525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If re-scoring the 50 hospital cases using only each patient's primary diagnosis, dropping incidental comorbidities such as pulmonary bullae or coronary atherosclerosis, erased most of the gap between MCQ accuracy and patient-level recall, the claim that LLMs fail at real HPB diagnosis would be seriously weakened. A second check would be to have experienced clinicians independently list all diagnoses from the same case text and compare their recall to the LLMs' recall.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ICD-10 coding scheme that defines the benchmark's 33 main categories and 465 subcategories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the strict evaluation model in the two-stage LLM-as-a-judge scoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the check model that overrides false-negative judgments from the strict evaluator."}],"review_version":1}