Pith. sign in

REVIEW 3 major objections 5 minor 60 references

MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that MedRealMM, a benchmark built from 5,620 real multimodal online consultations, shows frontier LLMs remain below practicing physicians, with the gap driven by unsafe negative behaviors rather than missing correct action

desk verdict A genuinely useful real-world multimodal medical consultation benchmark, but the headline physician-vs-model gap is partly entangled with rubric initialization and needs a blinded audit before I'd fully trust the ranking. read the letter →

arxiv 2607.09142 v3 pith:LCPP7WPW submitted 2026-07-10 cs.AI cs.CLcs.CV

Runhan Shi , Quan Zhou , Yuqian Xu , Shuai Yang , Xin Wu , Zitong Zhou , Hui Liu , Bin Zha
show 9 more authors
Zheming Wang Liya Li Wei Wei Jinru Ding Wenrao Pang Mouxiao Bian Haoyuan Hu Jun Xu Jie Xu
This is my paper
classification cs.AIcs.CLcs.CV
keywords MedRealMMmultimodalmedicalconsultationbenchmarkrubric-basedevaluationLLM-as-a-judgeclinicalsafetyMCCPonline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedRealMM is a benchmark for Chinese online medical consultation built from de-identified, real patient-doctor interactions that include patient-uploaded images. The paper's central claim is that current frontier LLMs, both general-purpose and medical-specialized, do not yet match the treating physician on this benchmark, and that the remaining gap is largely a safety problem: top models satisfy as many or more positive clinical criteria than physicians but trigger more negative, unsafe, unsupported, or contradictory criteria. The paper also claims that image information is critical, since providing patient images raises frontier scores by 14 to 20 points, and that medical-specialized models benefit little from images. A sympathetic reader would care because this is evidence that exam-style or text-only benchmarks overstate deployment readiness, and it points to error avoidance as the bottleneck rather than medical knowledge.

What carries the argument

The central object is the Multimodal Clinical Challenge Point (MCCP): a position in an authentic consultation trajectory, immediately before a physician response, where substantive clinical engagement and dependence on patient-uploaded images coincide. It converts a messy multi-turn dialogue into a standardized next-response generation task. The scoring standard is a physician-refined case-specific rubric: a list of positive and negative criteria with signed importance weights, initialized by an LLM from the case and the original physician response, then iteratively revised by physicians. An LLM grader issues a binary verdict per criterion, and scores are aggregated as weighted sums normaliz

What would settle it

Take a random subset of cases, have physicians write rubrics without ever seeing the original physician response, and re-score both the Online anchor and the best model with those blind rubrics. If the physician advantage shrinks or disappears, the original-response initialization, rather than general clinical appropriateness, is carrying the conclusion.

Watch

Extended reading notes

Core claim

MedRealMM converts real consultations into single-turn next-response generation tasks by identifying Multimodal Clinical Challenge Points (MCCPs)—turns where the physician must engage substantively and patient images materially affect the response. Each case comes with a physician-refined, case-specific rubric with signed weights; an LLM judge scores each generated response criterion-by-criterion. On 5,620 cases across 64 clinical departments, the original online physician response outscores every evaluated model, and models that meet more positive criteria still lose because they trigger more negative criteria. Best multimodal models improve sharply when images are provided, but remain arou

Load-bearing premise

The rubrics, even after physician refinement, are initialized from the original physician response; if they encode that particular doctor's choices and wording, then comparing models against the 'Online' anchor over-rewards the original response and inflates the physician-versus-model gap.

Editorial extensions

If this is right

  • If correct, text-only benchmarks systematically overstate relative model performance; image grounding is required to separate frontier models from weaker ones.
  • Safety-focused training and evaluation, not just medical knowledge, is the key lever: models lose to physicians on negative criteria, not positive coverage.
  • MedRealMM can serve as a deployment-readiness test: weak models cluster near zero, and even the strongest models leave large headroom below the physician anchor.
  • Medical-specialized models are not yet competitive in real multimodal consultation, gaining little from images compared with general-purpose multimodal models.
  • A specific weak spot is history taking: models often fail to ask for missing information, while physicians excel at it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The physician 'Online' anchor is an operational baseline from busy practice, not a human ceiling; the paper's own deliberative-physician results show higher scores, so the model gap is likely conservative.
  • Because rubrics are initialized from the original physician response, part of the documented physician advantage could reflect rubric circularity; a blind-rubric ablation would test this directly.
  • The MCCP extraction idea generalizes: any logged expert-interaction domain (legal advice, technical support) could be converted into benchmark instances by identifying challenge points where engagement and evidence matter jointly.
  • Worst-of-k sampling on this benchmark would likely expose a larger unsafe tail than mean-score evaluation, which the paper notes as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedRealMM, a benchmark for multimodal Chinese online medical consultation built from 5,620 de-identified patient-doctor consultations from JD Health. The construction pipeline filters real consultations, extracts Multimodal Clinical Challenge Points (MCCPs), truncates each trajectory at the challenge point, and pairs the resulting instance with a case-specific rubric that is LLM-initialized and iteratively refined by physicians. Evaluation is performed by an LLM judge that scores each generated response against the rubric criterion by criterion. Nineteen LLMs are compared with the original online physician response. The main reported findings are that image input substantially improves scores, that current frontier models remain below the online physician anchor, and that the physician advantage is driven mainly by avoiding negative (unsafe/unsupported) criteria rather than by satisfying more positive criteria.

Significance. If the rubric protocol is accepted, MedRealMM addresses a genuine gap: it is built from authentic, multimodal, one-on-one consultations rather than simulated patients or curated image-question pairs, and it evaluates open-ended responses with case-specific rubrics rather than lexical overlap. The paper has real strengths: physician acceptance of MCCPs is reported at over 99% on a 200-case sample; inter-judge and judge-physician agreement measured by Gwet's AC1 is above 0.8; the de-identification and privacy protocol is unusually detailed; and the deliberative-physician subset is a sensible attempt to calibrate the anchor. These validations support the benchmark's basic usability. However, the central comparative claim — that current LLMs remain below the online physician and that the gap is safety-driven — rests on rubrics that are initialized from the very physician response later used as the anchor. That dependence is load-bearing and is not yet demonstrated to have been removed by physician refinement. The paper's own limitation appendix also acknowledges single-sample evaluation, which is relevant to the safety conclusion.

major comments (3)
  1. [Section 3.3, Eqs. (8)-(9); Appendix A.5] The rubric-initialization step is the main load-bearing concern. Equation (8) generates R(0) from (x, yorig), and the Online anchor is later scored against R*. Any residual dependence of R* on the original response's wording, content, or omissions will inflate the anchor's score: yorig will tend to satisfy criteria that are paraphrases of its own behavior, while clinically appropriate alternative responses may be under-credited. The statement in §3.3 that criteria 'overly specific to the original response are revised or removed' is an assertion, and the only supporting evidence is a single worked criterion in Figure 9 (Appendix A.5). The paper reports no aggregate counts of criteria added/removed/modified over the refinement iterations, no audit by physicians blind to yorig, and no test that R* rewards a second clinically valid response to the same case. This affects both the headline On
  2. [Section 3.2, Eq. (5); Section 4.3] The claim that 'image information is critical for reliable clinical performance' is partly guaranteed by the benchmark definition. An MCCP is admitted only if M(t*;C)=1, i.e., the images materially influence an appropriate response. Therefore the 14–20 point gains in Figure 5 are a consequence of case selection, not an independent empirical discovery about real consultations. The paper does acknowledge this in §4.3 ('consistent with the benchmark design'), but the abstract and the contribution list present image-criticality as a finding. Please either soften the claim to 'in cases selected for multimodal relevance' or include a control set of image-irrelevant MCCPs to make the comparison informative.
  3. [Section 4.5, Table 2; Eq. (11)] The judge-reliability analysis does not fully establish that the automated judge preserves the physician-vs-model ranking. The physician panel scores about 7 points higher than the LLM judges, and the difference is concentrated in positive criteria; for GPT-5, physician-judge AC1 is 0.768 on positives versus 0.920 on negatives. Since the paper's central contrast is that Online leads by avoiding negatives, an automated judge that systematically under-awards positives could alter both the overall ordering and the positive/negative decomposition. Please report whether the Online-versus-model ordering and the negative-criterion gap hold when the 200-case subset is scored by the physician panel, and include a per-criterion breakdown of LLM-judge false positives/negatives relative to the physician panel. Confidence intervals for the AC1 values would also be useful.
minor comments (5)
  1. [Section 4.1 vs. Section 3.3] Section 4.1 says rubric refinement is performed for three iterations, while Section 3.3 defines K as the first round with no physician feedback. Please clarify whether K=3 was used for all cases or whether convergence was checked; if K=3 is a fixed budget, say so explicitly.
  2. [Eq. (12)] The normalization in Eq. (12) is reasonable, but the clipping means a response triggering many negatives can receive the same floor score as one triggering a single severe negative. This is worth stating explicitly, since the floor spike in Figure 7 is interpreted as a property of weak models.
  3. [Figure 1] Minor typos in the figure: 'Pulmonary Ederna' should be 'Pulmonary Edema', and 'monitor symsptoms' should be 'monitor symptoms'.
  4. [Appendix A.5] The single rubric-iteration example is helpful but does not quantify how often physician feedback changes criteria, weights, or the final score distribution. Reporting aggregate statistics over the full 5,620 cases would substantially strengthen the rubric-construction section.
  5. [Section B.1] The exception for MedGemma-27B's decoding parameters is clearly disclosed, but because this model is in the medical-specialized multimodal group, the reader should be reminded that its score is not directly comparable to the other open-source models under their default settings.

Circularity Check

1 steps flagged · score 6.0 of 10

Online anchor is scored on rubrics initialized from its own response (Eq. 8); the physician-vs-model gap is partially by construction.

  1. fitted input called prediction [Section 3.3, Eq. (8); Section 3.3 refinement defense; Section 4.2 (Online anchor, Fig. 4a); Eq. (13)]
    "Given a benchmark instance x and the corresponding original physician response yorig, an LLM agent Generator ϕR generates an initial rubric R(0)=ϕR(x,yorig)={(cj,wj)}. […] Figure 4(a) summarizes the overall performance of all models, together with the original real-world physician response (Online). The ordering is consistent: […] the physician response sits above all of them."

    R(0)—and after Eq. (9) refinement, R*—is seeded from the exact physician response that is later scored as the Online anchor. Scoring yorig against criteria generated from yorig measures, in part, agreement of a response with itself. The paper's defense—'criteria that are overly specific to the original response are revised or removed'—is asserted, not demonstrated: Appendix A.5 shows a single criterion split, with no aggregate count of removed or rewritten response-specific criteria, no yorig-blinded rubric audit, and no evidence that final rubrics would reward a clinically equivalent but differently worded response. Because the central conclusion is the Online-vs-model gap (Section 4.2, Eq. 13), any residual yorig dependence inflates the physician anchor and makes part of the reported gap

full rationale

MedRealMM's construction is largely transparent and includes a physician-in-the-loop rubric refinement, a separate physician-panel judge-agreement check (Table 2), and a deliberative-physician robustness subset. These features provide partial independent grounding for the rubric content and for the LLM judge, but they do not remove the initialization dependence: the same yorig used to seed R(0) is the Online anchor scored by R*. Relative LLM-vs-LLM comparisons are probably not invalidated by this, since all models are scored with the same rubrics; the specific claim that the physician response sits above all frontier models is where the circularity bites. The paper asserts that refinement removes response-specific criteria, but provides only a single worked example and no aggregate audit, so the magnitude of the residual dependence is unknown. There are no load-bearing self-citations; references to MedBench are related-work only. The LLM-as-judge step is checked against a physician panel, so it is not separately circular. The image-criticality finding is partly design-tautological (MCCPs are selected to require image relevance), but the quantitative image-vs-text gap is empirical. Overall: one central constructional circularity, partial, score 6.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central evaluation rests on rubric weights chosen by LLM/physicians, the assumption that those rubrics are independent of the reference physician response, and the assumption that LLM judges accurately apply them. The MCCP construct is validated but the overall benchmark does not rest on derived physical or mathematical constants.

free parameters (1)
  • Rubric criterion weights w_j
    Each criterion has a signed importance weight in [-20,20]\{0}, generated by an LLM and adjusted by physicians (Eq. 8-9). Every case score and model ranking depends on these hand/LLM-set weights; they are not derived from independent clinical constants.
assumptions (5)
  • ad hoc to paper Rubrics initialized from the original physician response and refined by physicians measure general clinical appropriateness rather than the specific physician's behavior.
    Load-bearing for the Online-vs-model comparison; introduced in Section 3.3, Eq. (8).
  • domain assumption An LLM judge's binary verdict per rubric criterion is sufficiently accurate for ranking models and physicians.
    Used in Eq. (11); supported only partially by Gwet's AC1 >0.8 on a 200-case subset, with an acknowledged ~7-point physician/judge score gap.
  • domain assumption Each selected MCCP is a clinically demanding state in which patient-uploaded images are necessary for the next physician response.
    The A/M criteria in Section 3.2 define the benchmark; physician acceptance of 99% is evidence, but the dataset is restricted to image-relevant moments by design.
  • domain assumption De-identification preserves enough clinical signal for meaningful evaluation.
    Required by Stages 4-5 of Appendix A.1; human review checks this, but the claim is not quantitatively verified.
  • domain assumption Single-turn next-response generation at an MCCP adequately represents consultation competence.
    The benchmark deliberately truncates multi-turn consultations to one prediction target (Section 3.2); this is a modeling choice that may not capture history-dependence fully.
invented entities (1)
  • Multimodal Clinical Challenge Point (MCCP) independent evidence
    purpose: Defines a clinically demanding turn in a consultation trajectory and converts it into a self-contained single-turn evaluation instance.
    The construct has a falsifiable handle: 200 sampled MCCPs were independently judged by physicians with >99% acceptance (Appendix A.3), so it is not entirely LLM-internal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation." pith.science (2026). https://pith.science/paper/LCPP7WPW

@misc{pith2026260709142,
  author       = {Pith},
  title        = {Pith review of: MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCPP7WPW}},
  note         = {Machine review of arXiv:2607.09142}
}
read the original abstract

Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that poorly reflect clinical quality. We introduce \textbf{MedRealMM}, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories and converts each into a standardized next-response generation task while preserving the preceding text-image context. Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviors and penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and multimodal systems. Our results show that image information is critical for reliable clinical performance and that current frontier models remain below the online physician response. Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM offers a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultation. The dataset will be publicly available on Hugging Face at https://huggingface.co/datasets/jdh-algo/MedRealMM.

Figures

Figures reproduced from arXiv: 2607.09142 by the authors.

Figure 1
Figure 1. Three gaps between existing medical LLM benchmarks and real-world online consultation: (a) unrealistic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall MedRealMM construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data statistics of MedRealMM: distributions of (a) clinical departments, (b) patient intents, (c) consultation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Evaluation results on MedRealMM (best available modality per model), including (a) overall performance [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Multimodal (text + image) vs. text-only performance for multimodal models on MedRealMM. Darker and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Results for representative models grouped by (a) major clinical departments, and (b) patient intents and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Per-case score distribution for evaluated models and [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 7
Figure 7. Figure 7: Per-case score distribution for evaluated models and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: An example multimodal consultation dialogue in Chinese. Patient metadata is omitted for brevity. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Physician-guided rubric iteration on a tinea pedis case. The original criterion (C7) conflates two requirements [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Per-criterion grading prompt used by ϕJ . For each benchmark instance, this prompt is instantiated Nx times (once per rubric criterion cj ∈ R∗ ) and the resulting binary verdicts {vj} are aggregated into the case-level score S(x, y, R∗ ). benchmark difficulty is drive…
Figure 11
Figure 11. Figure 11: Results decomposition for representative models on (a) number of images and (b) number of dialogue turns. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Results decomposition for representative models on all departments, ordered by the number of cases in [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 33 linked inside Pith

  1. [1]

    Digital health market statistics 2026: Market size, investment and user adoption, 2026

    Richard Jaimes. Digital health market statistics 2026: Market size, investment and user adoption, 2026. Accessed: 2026-06-03

    2026

  2. [2]

    Telemedicine and AI in healthcare: Top platforms, trends & 2026 outlook

    GAC guidelines. Telemedicine and AI in healthcare: Top platforms, trends & 2026 outlook. https: //gacguidelines.ca, 2026. Accessed: 2026-06-03

    2026

  3. [3]

    Annual results announcement for the year ended december 31, 2025.HKEXnews,

    JD Health International Inc. Annual results announcement for the year ended december 31, 2025.HKEXnews,

    2025

  4. [4]

    Jd health introduces groundbreaking LLM-powered suite for comprehensive online and in-hospital healthcare scenarios

    Vivian Yang. Jd health introduces groundbreaking LLM-powered suite for comprehensive online and in-hospital healthcare scenarios. JD Corporate Blog, 2025

    2025

  5. [5]

    Large language models in medicine.Nature Medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature Medicine, 29(8):1930–1940, 2023

    1930

  6. [6]

    The effect of using a large language model to respond to patient messages.The Lancet Digital Health, 6(6):e379–e381, 2024

    Shan Chen, Marco Guevara, Shalini Moningi, Frank Hoebers, Hesham Elhalawani, Benjamin H Kann, Fallon E Chipidza, Jonathan Leeman, Hugo J W L Aerts, Timothy Miller, Guergana K Savova, Jack Gallifant, Leo A Celi, Raymond H Mak, Maryam Lustberg, Majid Afshar, and Danielle S Bitterman. The effect of using a large language model to respond to patient messages....

    2024

  7. [7]

    Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S. Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R. Webster, Ewa Dominowska, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S. Corrado, Yoss...

    2025

  8. [8]

    Cohen, and Xinghua Lu

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. arXiv:1909.06146, 2019

    arXiv 1909

Show all 60 references
  1. [9]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv:2009.13081, 2020

    2009 arXiv

  2. [10]

    MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. arXiv:2203.14371, 2022

    2022 arXiv

  3. [11]

    Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv:2212.13138, 2023. 14 MedRealMM: A Real-World Multimodal Benchmark ...

    2023 arXiv

  4. [12]

    Pfohl, Heather Cole-Lewis, et al

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, et al. Towards expert-level medical question answering with large language models. arXiv:2305.09617, 2025

    2025 arXiv

  5. [13]

    CBLUE: A chinese biomedical language understanding evaluation benchmark

    Ningyu Zhang et al. CBLUE: A chinese biomedical language understanding evaluation benchmark. arXiv:2106.08087, 2021

    2021 arXiv

  6. [14]

    PromptCBLUE: A chinese prompt tuning benchmark for the medical domain

    Wei Zhu, Xiaoling Wang, Huanran Zheng, Mosha Chen, and Buzhou Tang. PromptCBLUE: A chinese prompt tuning benchmark for the medical domain. arXiv:2310.14151, 2023

    2023 arXiv

  7. [15]

    Benchmarking large language models on CMExam – a comprehensive chinese medical exam dataset

    Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. Benchmarking large language models on CMExam – a comprehensive chinese medical exam dataset. arXiv:2306.03030, 2023

    2023 arXiv

  8. [16]

    CMB: A comprehensive medical benchmark in chinese

    Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. CMB: A comprehensive medical benchmark in chinese. arXiv:2308.08833, 2023

    2023 arXiv

  9. [17]

    MedBench: A comprehensive, standardized, and reliable benchmarking system for evaluating chinese medical large language models

    Mianxin Liu, Jinru Ding, Jie Xu, Weiguo Hu, Xiaoyang Li, Lifeng Zhu, Zhian Bai, Xiaoming Shi, Benyou Wang, Haitao Song, et al. MedBench: A comprehensive, standardized, and reliable benchmarking system for evaluating chinese medical large language models. arXiv:2407.10990, 2024

    2024 arXiv

  10. [18]

    MedBench v4: A robust and scalable benchmark for evaluating chinese medical language models, multimodal models, and intelligent agents

    Jinru Ding, Lu Lu, Chao Ding, Mouxiao Bian, Jiayuan Chen, Renjie Lu, Wenrao Pang, Xiaoqin Wu, Zhiqiang Liu, Luyi Jiang, et al. MedBench v4: A robust and scalable benchmark for evaluating chinese medical language models, multimodal models, and intelligent agents. arXiv:2511.14439, 2025

    2025

  11. [19]

    MedDialog: Two large-scale medical dialogue datasets

    Xuehai He, Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, et al. MedDialog: Two large-scale medical dialogue datasets. arXiv:2004.03329, 2020

    2004 arXiv

  12. [20]

    MedDG: An entity-centric medical consultation dataset for entity-aware medical dialogue generation

    Wenge Liu, Jianheng Tang, Yi Cheng, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. MedDG: An entity-centric medical consultation dataset for entity-aware medical dialogue generation. arXiv:2010.07497, 2020

    2010 arXiv

  13. [21]

    MediTOD: An english dialogue dataset for medical history taking with comprehensive annotations

    Vishal Vivek Saley, Goonjan Saha, Rocktim Jyoti Das, Dinesh Raghu, and Mausam. MediTOD: An english dialogue dataset for medical history taking with comprehensive annotations. arXiv:2410.14204, 2024

    2024 arXiv

  14. [22]

    An automatic evaluation framework for multi-turn medical consultations capabilities of large language models

    Yusheng Liao, Yutong Meng, Hongcheng Liu, Yanfeng Wang, and Yu Wang. An automatic evaluation framework for multi-turn medical consultations capabilities of large language models. arXiv:2309.02077, 2023

    2023 arXiv

  15. [23]

    Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov

    Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. MediQ: Question-asking llms and a benchmark for reliable interactive clinical reasoning. arXiv:2406.00922, 2024

    2024 arXiv

  16. [24]

    AgentClinic: A multimodal agent benchmark to evaluate ai in simulated clinical environments

    Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. AgentClinic: A multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv:2405.07960, 2024

    2024 arXiv

  17. [25]

    The dialogue that heals: A comprehensive evaluation of doctor agents’ inquiry capability

    Linlu Gong, Ante Wang, Yunghwei Lai, Weizhi Ma, and Yang Liu. The dialogue that heals: A comprehensive evaluation of doctor agents’ inquiry capability. arXiv:2509.24958, 2025

    2025

  18. [26]

    MedConsultBench: A full-cycle, fine-grained, process-aware benchmark for medical consultation agents

    Chuhan Qiao, Jianghua Huang, Daxing Zhao, Ziding Liu, Yanjun Shen, Bing Cheng, Wei Lin, and Kai Wu. MedConsultBench: A full-cycle, fine-grained, process-aware benchmark for medical consultation agents. arXiv:2601.12661, 2026

    2026

  19. [27]

    MedDialBench: Benchmarking llm diagnostic robustness under parametric adversarial patient behaviors

    Xiaotian Luo, Xun Jiang, and Jiangcheng Wu. MedDialBench: Benchmarking llm diagnostic robustness under parametric adversarial patient behaviors. arXiv:2604.06846, 2026

    2026 arXiv

  20. [28]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human...

    2025 arXiv

  21. [29]

    MedDialogRubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models

    Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, et al. MedDialogRubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models. arXiv:2601.03023, 2026

    2026

  22. [30]

    Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, et al

    Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K. Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, et al. HealthBench Professional: Evaluating large language models on real clinician chats. arXiv:2604.27470, 2026

    2026 arXiv

  23. [31]

    LiveMedBench: A contamination-free medical benchmark for llms with automated rubric evaluation

    Zhiling Yan, Dingjie Song, Zhe Fang, Yisheng Ji, Xiang Li, Quanzheng Li, and Lichao Sun. LiveMedBench: A contamination-free medical benchmark for llms with automated rubric evaluation. arXiv:2602.10367, 2026

    2026

  24. [32]

    Pathological visual question answering

    Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathological visual question answering. arXiv:2010.12435, 2020

    2010 arXiv

  25. [33]

    SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. arXiv:2102.09542, 2021

    2021 arXiv

  26. [34]

    PMC-VQA: Visual instruction tuning for medical visual question answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: Visual instruction tuning for medical visual question answering. arXiv:2305.10415, 2023

    2023 arXiv

  27. [35]

    OmniMedVQA: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. OmniMedVQA: A new large-scale comprehensive evaluation benchmark for medical lvlm. arXiv:2402.09181, 2024

    2024 arXiv

  28. [36]

    GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical ai

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv:2408.03361, 2024

    2024 arXiv

  29. [37]

    3MDBench: Medical multimodal multi-agent dialogue benchmark

    Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, and Andrey Savchenko. 3MDBench: Medical multimodal multi-agent dialogue benchmark. arXiv:2504.13861, 2025

    2025

  30. [38]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors,Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pa...

  31. [39]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  32. [40]

    Rahmani, and Emine Yilmaz

    Yinzhu Chen, Abdine Maiga, Hossein A. Rahmani, and Emine Yilmaz. Automated rubrics for reliable evaluation of medical dialogue systems. arXiv:2601.15161, 2026

    2026 arXiv

  33. [41]

    Deid-gpt: Zero-shot medical text de-identification by gpt-4

    Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, Fang Zeng, Lichao Sun, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, and Xiang Li. Deid-gpt: Zero-shot medical text de-identification by gpt-4. ar...

    2025

  34. [42]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv:2306.05685, 2023

    2023 arXiv

  35. [43]

    Claude 4 Opus (versions 4.6 and 4.7)

    Anthropic. Claude 4 Opus (versions 4.6 and 4.7). Large Language Models by Anthropic, https://www. anthropic.com, 2026

    2026

  36. [44]

    Gemini: A family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, et al. Gemini: A family of highly capable multimodal models. arXiv:2312.11805, 2025

    2025 arXiv

  37. [45]

    Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicine, 30(9):2613–2622, 2024

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-maki...

    2024

  38. [46]

    Computing inter-rater reliability and its variance in the presence of high agreement.The British Journal of Mathematical and Statistical Psychology, 61(Pt 1):29–48, May 2008

    Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement.The British Journal of Mathematical and Statistical Psychology, 61(Pt 1):29–48, May 2008

    2008

  39. [47]

    Openai gpt-5 system card

    Aaditya Singh, Adam Fry, Adam Perelman, et al. Openai gpt-5 system card. arXiv:2601.03267, 2026

    2026 arXiv

  40. [48]

    Kimi K2.6: Open-weight trillion-parameter MoE agent model

    Moonshot AI. Kimi K2.6: Open-weight trillion-parameter MoE agent model. Hugging Face Model Repository, https://huggingface.co/moonshotai/Kimi-K2.6, April 2026

    2026

  41. [49]

    Kimi k2.5: Visual agentic intelligence

    Kimi Team, Tongtong Bai, Yifan Bai, et al. Kimi k2.5: Visual agentic intelligence. arXiv:2602.02276, 2026

    2026 arXiv

  42. [50]

    Qwen3.6-27B and Qwen3.6-35B-A3B

    Qwen Team. Qwen3.6-27B and Qwen3.6-35B-A3B. Hugging Face Model Repository, https://github.com/ QwenLM/Qwen3.6, April 2026

    2026

  43. [51]

    Glm-5: from vibe coding to agentic engineering

    GLM-5-Team, Aohan Zeng, Xin Lv, et al. Glm-5: from vibe coding to agentic engineering. arXiv:2602.15763, 2026

    2026 arXiv

  44. [52]

    Deepseek-v4: Towards highly efficient million-token context intelligence

    DeepSeek-AI, Anyi Xu, Bangcai Lin, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv:2606.19348, 2026

    2026

  45. [53]

    Medgemma technical report

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, et al. Medgemma technical report. arXiv:2507.05201, 2026

    2026 arXiv

  46. [54]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning

    LASA Team, Weiwen Xu, Hou Pong Chan, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv:2506.07044, 2025

    2025 arXiv

  47. [55]

    HuatuoGPT-3-32B large language model repository

    FreedomIntelligence. HuatuoGPT-3-32B large language model repository. Hugging Face Model Repository, https://huggingface.co/FreedomIntelligence/HuatuoGPT-3-32B, 2026

    2026

  48. [56]

    AntAngelMed medical large language model repository

    MedAIBase. AntAngelMed medical large language model repository. Hugging Face Model Repository, https: //huggingface.co/MedAIBase/AntAngelMed, 2026

    2026

  49. [57]

    Baichuan-m3: Modeling clinical inquiry for reliable medical decision-making

    M3 Team, Chengfeng Dou, Fan Yang, Fei Li, Jiyuan Jia, Qiang Ju, Shuai Wang, Tianpeng Li, Xiangrong Zeng, Yijie Zhou, Hongda Zhang, Jinyang Tai, Linzhuang Sun, Peidong Guo, Yichuan Mo, Xiaochuan Wang, Hengfu Cui, and Zhishou Zhang. Baichuan-m3: Modeling clinical inquiry for rel...

    2026

  50. [58]

    criterion_id

    M2 Team, Chengfeng Dou, Chong Liu, Fan Yang, Fei Li, Jiyuan Jia, Mingyang Chen, Qiang Ju, Shuai Wang, Shunya Dang, Tianpeng Li, Xiangrong Zeng, Yijie Zhou, Chenzheng Zhu, Da Pan, Fei Deng, Guangwei Ai, Guosheng Dong, Hongda Zhang, Jinyang Tai, Jixiang Hong, Kai Lu, Linzhuang S...

    2025 arXiv

  51. [2002]

    Association for Computational Linguistics

  52. [2026]

    Accessed: 2026-06-03

    2026

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.