MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

Bin Zha; Haoyuan Hu; Hui Liu; Jie Xu; Jinru Ding; Jun Xu; Liya Li; Mouxiao Bian; Quan Zhou; Runhan Shi

On real multimodal Chinese online consults, frontier LLMs still trail online physicians because they trigger more unsafe or unsupported criteria.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 05:06 UTC pith:LCPP7WPW

load-bearing objection Real multimodal Chinese online-consult data with a useful safety finding; the Online-vs-LLM gap is real but partly entangled with platform length/style and Online-seeded rubrics. the 3 major comments →

arxiv 2607.09142 v3 pith:LCPP7WPW submitted 2026-07-10 cs.AI cs.CLcs.CV

MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

Runhan Shi , Quan Zhou , Yuqian Xu , Shuai Yang , Xin Wu , Zitong Zhou , Hui Liu , Bin Zha

show 9 more authors

Zheming Wang Liya Li Wei Wei Jinru Ding Wenrao Pang Mouxiao Bian Haoyuan Hu Jun Xu Jie Xu

This is my paper

classification cs.AI cs.CLcs.CV

keywords multimodal medical consultationonline medical consultationLLM evaluationclinical rubricspatient-uploaded imagesChinese medical AIsafety-sensitive evaluationnext-response generation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing medical LLM benchmarks often use synthetic dialogues, leave out patient-uploaded images, or score open-ended answers with multiple-choice and word-overlap metrics that miss clinical quality. MedRealMM is built instead from de-identified real consultations at a nationwide Chinese internet hospital. It extracts Multimodal Clinical Challenge Points—moments where the physician must reason over the accumulated text and images—and turns each into a next-response generation task with a physician-refined, case-specific rubric that rewards good clinical behavior and penalizes unsafe or contradictory replies. Across 5,620 cases in 64 departments, images raise scores sharply, medical-specialized models trail general-purpose ones, and even the best frontier systems sit below the online physician anchor. Some models match or exceed physicians on positive criteria yet accumulate more negative ones, so the practical bottleneck is avoiding harmful errors rather than covering more desirable points.

Core claim

On authentic multimodal Chinese online consultations, current frontier LLMs remain below the online physician response. Some frontier models satisfy as many or more positive clinical criteria than physicians, yet they trigger more negative criteria, so safety-sensitive error avoidance is the central bottleneck separating models from real online care.

What carries the argument

Multimodal Clinical Challenge Point (MCCP) extraction: identify clinically demanding turns in real consultation trajectories (clinical engagement plus multimodal relevance), truncate to a self-contained next-response task with the original text-image context, and score against a physician-refined case-specific rubric of positive and negative criteria.

Load-bearing premise

That physician-refined, case-specific rubrics scored by an LLM judge are a faithful enough measure of clinical quality for open-ended next responses, even though physicians award roughly seven points higher on average and agree less strongly on positive criteria.

What would settle it

On a held-out set of real multimodal cases, show that the best models match or beat deliberative physician responses on both positive and negative criteria under the same rubrics, or show that LLM-judge scores reverse the Online-vs-model ranking relative to independent physician grading.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Image grounding is a first-order requirement for realistic online consultation benchmarks; text-only evaluation compresses model differences and understates capability gaps.
Progress on exam-style medical QA does not automatically translate into safe next-response behavior on real multimodal trajectories.
Avoiding negative clinical criteria (unsafe, unsupported, or contradictory content) is a sharper deployment target than maximizing positive coverage.
Specialty and stage gaps remain, especially history-taking (know-when-to-ask) and traditional Chinese medicine, so aggregate scores alone hide clinically important failures.
Extracting challenge points from authentic logs offers a scalable alternative to full patient simulation for interactive medical evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Training or alignment objectives that explicitly penalize negative-rubric triggers may close more of the Online gap than further positive-coverage gains.
The same MCCP-plus-rubric recipe could be re-applied to other languages and platforms once comparable de-identified multimodal logs exist, testing whether the safety bottleneck is Chinese-specific or general.
Because deliberative physicians still outscore both Online and the best models on the same rubrics, the operational Online baseline understates remaining human headroom and the distance to expert-level care.
Weak agreement on positive criteria relative to negatives suggests future rubrics or judges should separate “must-include” clinical content from more discretionary phrasing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Real multimodal Chinese online-consult data with a useful safety finding; the Online-vs-LLM gap is real but partly entangled with platform length/style and Online-seeded rubrics.

read the letter

The useful thing here is not another medical QA set. They took real JD Health multimodal consultations, kept patient-uploaded images, cut them at Multimodal Clinical Challenge Points into next-response tasks, and scored with physician-refined case-specific rubrics. That combination is genuinely closer to deployed Chinese online care than HealthBench, LiveMedBench, or the usual simulators.

What they do well: scale and honesty of measurement. 5,620 cases, 64 departments, 19 models, clear image ablations (+14–20 for frontier systems), and the positive/negative split in Fig. 4. The headline result—that some models match or beat Online on positives yet lose on negatives—is the right kind of finding for this field. MCCP validation (199/200) and multi-judge AC1 >0.8 with physicians are real checks, not theater. Limitations section is straight about single-platform Chinese scope and single-sample scoring.

Soft spots, in proportion. The stress-test note is partly right and should not be overstated. Rubrics are initialized from the original physician reply and include platform length constraints; Online is an operational baseline under time/length pressure, and deliberative physicians jump to ~60 on the 200-case subset. So “safety bottleneck” is directionally supported by the negative criteria, but absolute Online-vs-LLM gaps partly mix clinical harm with style, completeness, and Online-anchored phrasing. Physician judges also award ~7 points more, mostly on positives. That weakens treating the leaderboard as deployment truth; it does not erase the multimodal gap or the positive/negative decoupling. Mild circularity risk, not a broken design. Dataset is promised on HF rather than shown with hashes in the paper—standard for this venue, still a practical caveat.

Math is simple weighted clip scoring, not load-bearing theory. Citations cover the right priors. Free parameters (weights, filters, three refinement rounds) are disclosed.

This is for people building or evaluating medical dialogue systems, especially multimodal and Chinese digital health. Worth a serious referee. I would engage: use the data when it lands, cite the safety decoupling, and keep the Online comparison caveated.

Referee Report

3 major / 0 minor

Summary. MedRealMM is a real-world multimodal benchmark for Chinese online medical consultation, built from 5,620 de-identified JD Health patient–physician trajectories spanning 64 departments. The authors extract Multimodal Clinical Challenge Points (MCCPs)—turns requiring substantive clinical engagement where patient-uploaded images can materially affect the next response—and convert each into a single-turn next-response task with preserved text–image context. Each case receives a physician-refined, case-specific signed rubric (positive rewards, negative penalties). Evaluating 19 general-purpose and medical LLMs, they report that images raise frontier scores by 14–20 points, that the best models remain below the Online physician anchor, and that some frontier models match or exceed Online on positive criteria while triggering more negatives—interpreted as a safety-sensitive error-avoidance bottleneck.

Significance. If the construction and evaluation hold, MedRealMM is a substantial contribution: authentic multimodal consultation data (not simulators or curated VQA pairs), a practical alternative to full multi-turn patient simulation via MCCP extraction, and physician-guided case-specific rubrics for open-ended clinical responses. Strengths include scale (5,620 cases), MCCP physician validation (199/200 accepted), multi-judge agreement (Gwet’s AC1 > 0.8), multimodal ablations, specialty/intent/stage stratification, and planned public release. The positive/negative decomposition and the Online vs deliberative-physician comparison (Table 3) are useful for the community. The work is timely for deployment-oriented medical LLM evaluation in Chinese digital health.

major comments (3)

Abstract and §4.2 / Fig. 4(b): The headline claim that “safety-sensitive error avoidance remains a central bottleneck” is not yet fully supported by the reported evidence. Frontier models meet as many or more positive criteria than Online but trigger more negatives; the paper equates those negatives with clinical unsafety. §3.3, however, initializes rubrics from the original physician response y_orig and adds case-specific length constraints because JD Health enforces a strict per-turn limit. Models that write more complete answers can therefore fail length-related or Online-style criteria while covering more positives—exactly the Fig. 4(b) pattern. Table 3 already shows deliberative physicians (60.07) beat Online (50.33) on a 200-case subset, confirming Online is an operational, constrained baseline. Please (i) taxonomize negative criteria into safety/harm vs style/length/coverage-of-On
§3.3–3.4 and Table 2: Rubric-grounded LLM-as-a-judge is load-bearing for all ranking claims, yet residual Online-anchoring and judge–physician gap are under-analyzed. Rubrics start from y_orig and are refined until physicians approve; the manuscript asserts that overly specific criteria are revised away, but provides no quantitative check (e.g., fraction of criteria that still paraphrase Online wording, or Online score when scored against rubrics built without seeing y_orig). Table 2 shows physicians award ~7 points higher than LLM judges, with weaker agreement on positive criteria (AC1 0.768 for GPT-5) than negatives (0.920). Please add: (a) a small human-only or y_orig-blind rubric subset to bound Online-specificity, and (b) a sensitivity analysis of model rankings under physician scores vs Gemini-3-Pro-Preview scores on the 200-case panel, so readers can see whether the Online > front
§3.2 (MCCP definition and single-turn formulation): The axiom that a single next response at an MCCP is a valid proxy for real multi-turn consultation competence is central but only lightly defended. History-taking is reported as a large Online–LLM gap (§4.4), yet success there often requires multi-turn information gathering; a single-turn “know-when-to-ask” score may understate or mis-locate interactive competence. Please either (i) discuss failure modes of the single-turn proxy with concrete examples (especially history-taking and diagnosis+treatment stages), or (ii) report a small multi-turn pilot (e.g., fixed follow-up patient turns) showing rank correlation with single-turn MCCP scores. Without this, the claim that MedRealMM evaluates “deployment-like consultation behavior” overstates what single-turn next-response generation measures.

Circularity Check

1 steps flagged

Empirical benchmark paper; only mild Online-seeded-rubric coupling, not a derivation that forces the main claim by construction.

specific steps

self definitional [§3.3 Case-specific rubric generation; §4.2 / Fig. 4 Online anchor]
"Given a benchmark instance x and the corresponding original physician response y_orig, an LLM agent Generator ϕ_R generates an initial rubric R^(0)=ϕ_R(x,y_orig)={(c_j,w_j)}. ... The original physician response is used only to initialize case-specific evaluation criteria rather than to define a reference answer. ... Because JD Health imposes a strict per-turn length limit ... we incorporate case-specific length constraints as rule-based evaluation criteria. ... current frontier models remain below the online physician response."

R* is seeded from the same Online response that is later scored as the human anchor, so Online can inherit partial advantage from criteria and length rules that reflect its own coverage and platform style. This is only mild: physician refinement, non-conditioning of the Grader on y_orig, Claude exceeding Online on positives, and higher deliberative-physician scores show the Online lead is not forced by definition.

full rationale

MedRealMM is a dataset/benchmark paper whose central claims (frontier models below Online physicians; images matter; negatives/safety as bottleneck) are empirical measurements on 5,620 cases, not first-principles derivations. There is no fitted parameter renamed as prediction, no uniqueness theorem imported from the authors, and no load-bearing self-citation chain. The only mild circularity-adjacent design choice is that case-specific rubrics R* are LLM-initialized from the original physician response y_orig while Online is also the leaderboard anchor. The paper itself mitigates this: physicians iteratively revise until F^(k)=∅; criteria overly specific to the original response are revised or removed; the Grader does not condition on y_orig; and Claude-Opus models meet as many or more positive criteria than Online while Online wins mainly by fewer negatives—so Online superiority is not tautological on positives. Table 3 further shows deliberative physicians (60.07) beat Online (50.33), so R* is not locked to the operational Online style. Length constraints tied to the platform’s per-turn limit can bias against verbose models, but that is methodological bias, not a reduction of a claimed derivation to its inputs. Score 2 reflects one minor construction–evaluation coupling that is not load-bearing for a forced result.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The paper's claims rest less on free physical constants than on construction and evaluation choices: what counts as a clinical challenge point, how rubrics are weighted and refined, and whether LLM judges faithfully score open-ended medical replies. Invented entities are methodological constructs (MCCP, signed rubric score) rather than new physical objects. The central comparative claim depends on these operational definitions remaining clinically meaningful outside the authors' pipeline.

free parameters (5)

rubric criterion signed weights w_j in [-20,20]\{0}
Per-case importance weights are LLM-initialized then physician-revised; absolute score magnitudes depend on these hand/LLM-chosen weights and the positive-weight normalization in Eq. 12.
rubric refinement rounds K (reported as 3 iterations)
Stopping after three refinement rounds is a procedural choice that affects which criteria remain in R*.
dialogue length/turn filters ([5,30] turns; max 4000 characters)
Corpus inclusion thresholds are design choices that shape the retained difficulty distribution.
department stratified sampling caps/oversampling
Balancing high-volume specialties versus long-tail departments changes case mix and aggregate scores.
MCCP selection score s_A(t)+s_M(t) argmax
When multiple candidate turns pass filters, the selected challenge point depends on LLM-assigned selection scores.

axioms (5)

ad hoc to paper A position is a valid MCCP iff it requires substantive clinical engagement and patient-uploaded images can materially influence the next physician response.
Definition in §3.2; operationalized by two LLM agents and validated on a 200-case physician sample.
domain assumption Open-ended clinical quality can be measured by binary satisfaction of signed case-specific rubric criteria aggregated with positive-weight normalization and clipping to [0,1].
Scoring framework in §3.4 Eq. 12; inherits HealthBench-style rubric evaluation assumptions.
domain assumption LLM-as-a-judge binary verdicts are reliable enough for model ranking when multi-judge AC1 is high and physician agreement is comparable.
§4.5 Table 2; standard but contested assumption in open-ended medical evaluation.
domain assumption De-identification can remove PII while preserving clinically relevant text/image signal for evaluation.
Appendix A.1–A.2; cases dependent on non-redactable identifiers are dropped.
ad hoc to paper Single next-response generation at an MCCP is a valid proxy for real multi-turn consultation competence without patient simulation.
Core methodological premise of §3.2; avoids simulators but also drops interactive recovery dynamics.

invented entities (3)

Multimodal Clinical Challenge Point (MCCP) independent evidence
purpose: Select clinically demanding, image-relevant turns in real trajectories and convert them into standardized next-response benchmark instances.
Central construction device of the paper; validated by physicians on a sample but defined by the authors' two-agent pipeline.
Physician-approved case-specific signed rubric R* independent evidence
purpose: Provide open-ended clinical scoring that rewards desirable behaviors and penalizes unsafe/unsupported ones without exact-match metrics.
Evaluation target is defined by these rubrics; independent clinical meaning depends on physician refinement quality.
Normalized clipped rubric score S(x,y,R*) no independent evidence
purpose: Aggregate heterogeneous positive/negative criteria into a single [0,1] case score for model ranking.
Eq. 12 is an author-chosen aggregation; different normalizations could reorder close models.

pith-pipeline@v1.1.0-grok45 · 23400 in / 3727 out tokens · 39420 ms · 2026-07-13T05:06:37.669012+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that poorly reflect clinical quality. We introduce \textbf{MedRealMM}, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories and converts each into a standardized next-response generation task while preserving the preceding text-image context. Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviors and penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and multimodal systems. Our results show that image information is critical for reliable clinical performance and that current frontier models remain below the online physician response. Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM offers a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultation. The dataset will be publicly available on Hugging Face at https://huggingface.co/datasets/jdh-algo/MedRealMM.

Figures

Figures reproduced from arXiv: 2607.09142 by Bin Zha, Haoyuan Hu, Hui Liu, Jie Xu, Jinru Ding, Jun Xu, Liya Li, Mouxiao Bian, Quan Zhou, Runhan Shi, Shuai Yang, Wei Wei, Wenrao Pang, Xin Wu, Yuqian Xu, Zheming Wang, Zitong Zhou.