Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Medical-agent diagnosis scores blend the questions a system asks with the final-diagnosis model it uses; this paper holds that model fixed across all agents and shows the swap alone flips 18–36% of pairwise rankings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MedDDC-Eval decouples evaluation of multi-turn consultation agents from their terminal diagnosis generators by scoring policy-elicited histories under one frozen shared diagnostic reader.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A genuinely useful decoupling protocol for evaluating consultation agents, with a clean confound demonstration and an honest limitations section; the main caveat is that the shared reader and semantic judge are unvalidated against human judgment, so the scores are protocol-relative even though the ranking-reversal result holds up. the 3 major comments →

arxiv 2607.18999 v2 pith:XVPWY4TI submitted 2026-07-21 cs.CL cs.AI

MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

Guofeng Zhang , Yizeng Quan , Huaiyi Fang , Jianwei Lv , Jinyao Liu , Xunxu Duan , Lening An , Yu Ouyang
show 1 more author
Junfeng Wang
This is my paper
classification cs.CL cs.AI
keywords multi-turn medical consultationdiagnosis-decoupled evaluationshared diagnostic readerLLM evaluation testbedinformation acquisition coveragesemantic matchingGRPO reward designranking confound
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-turn medical consultation agents are usually evaluated end to end: the same system interviews the simulated patient and produces the terminal diagnosis, so a single diagnosis score mixes the quality of the history it elicited with the quality of its own final-diagnosis generator. This paper argues that the mix is not a uniform offset and can change conclusions: in a fixed-history audit of eight policies, swapping each policy's own generator for one frozen shared diagnostic reader shifts diagnosis F1 by 2.2–19.0 points and reverses 18% (Record) and 36% (Dialogue) of pairwise orderings, so leaderboards built on coupled scores can reward the generator rather than the consultation. Its proposed fix, MedDDC-Eval, treats the elicited history as the evaluation object: a shared reader maps every history to a diagnosis, and auditable one-to-one counts produce separate diagnostic-support, information-coverage, and efficiency scores. The paper shows the same dimensions can drive training — a reinforcement-learning reward (GRPO) with diagnosis and trajectory feedback lifts a Qwen3-32B policy by 9.6 and 4.6 aggregate points on held-out splits — while being explicit that D/T/E are protocol-conditioned research metrics, not clinical endpoints.

Core claim

On its own terms, the paper's claim is that terminal diagnosis generation is a distinct, variable component of a consultation agent, and diagnosis-based comparisons that do not control it compare two things at once. The demonstration is the fixed-history intervention: identical conversations, prompts, and decoding settings, with only the history-to-diagnosis model changed; the resulting 2.2–19.0 point F1 shifts and 18%/36% ordering reversals show that the generator is not a uniform offset but changes which policies look better. The positive proposal is to hold that component fixed: MedDDC-Eval applies a single frozen prompt-based DeepSeek-V3 reader to every elicited history and scores diagno

What carries the argument

The load-bearing mechanism is the shared diagnostic reader: a single frozen prompt-based DeepSeek-V3 model that maps every completed consultation history to a terminal diagnosis, holding the history-to-diagnosis mapping constant across all eight policies. Around it sits an auditable measurement contract — LLM judges propose candidate semantic matches under directional coverage rules, a deterministic maximum-cardinality bipartite matching credits each prediction and reference at most once, and run-level micro F1 for diagnosis (D) and trajectory (T), plus a decay-weighted timing score for efficiency (E), are aggregated as Total = 0.5D + 0.4T + 0.1E. The one-to-one assignment is what converts f

Load-bearing premise

The whole pipeline rests on the LLM judge proposing complete and correct semantic matches and on the frozen shared reader producing valid diagnoses; as the paper's Limitations section and Appendix H state, neither is validated against human agreement or clinical outcomes, the paper explicitly calls D/T/E protocol-conditioned research metrics rather than clinical endpoints, and semantic near-duplicate screening across source text remains a listed follow-up — so any error in ma

What would settle it

Have clinicians independently score a sample of the 170 held-out histories: list every diagnostic statement and information need a physician would credit, and compare that list against the LLM judge's candidate edges and the shared reader's diagnoses. If clinicians recognize a substantial number of matches the judge never proposed, or disagree with the reader's diagnoses on the same histories, the D/T/E scores and the 18%/36% reversal counts are artifacts of an unvalidated measurement pipeline rather than facts about the policies. A secondary audit would apply semantic near-duplicate screening

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Coupled diagnosis scores can misattribute gains: a strong terminal generator can mask a thin elicited history, and swapping in a shared reader alone reverses 18–36% of pairwise orderings on fixed histories.
  • Policies can be ranked by the diagnostic value of the histories they elicit under a common reader, with D, T, and E reported separately so component trade-offs survive the aggregate Total.
  • A training reward organized around the same diagnosis and trajectory dimensions transfers to held-out cases: the GRPO-trained Qwen3-32B gains 9.6 Record and 4.6 Dialogue aggregate points over its initialization, and ablating either signal lowers Total on both splits.
  • Diagnosis and trajectory are positively associated across system means but nearly uncorrelated at the case level (r = 0.147 / 0.101), so the two axes carry non-redundant information for model selection.
  • The reported top ranking holds under all seven tested weight vectors and two alternative shared readers, though the leading margin narrows to 0.44 points in one setting — ordering stability without decisive pairwise separation there.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same confound generalizes: any agent benchmark where the same model both gathers evidence and renders a verdict — legal case building, technical support, financial advice — inherits the coupling this paper isolates, and the shared-reader protocol is a portable fix.
  • The reversal counts imply that existing medical-consultation leaderboards may be partly ranking terminal generators rather than consultation skills; a testable extension would re-run the fixed-history intervention on other public consultation benchmarks to see how large the effect is there.
  • The natural next validation is a human-agreement study: if physician panels accept both the LLM judge's semantic candidate edges and the shared reader's diagnoses, D/T/E become plausible surrogates for clinical consultation quality; if not, the decoupling argument survives but the absolute scores stay protocol-internal.
  • The trajectory denominator already excludes facts the patient volunteers; a further extension would directly penalize redundant or low-information questions, since the current efficiency axis measures timing of coverage rather than question informativeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. MedDDC-Eval proposes a diagnosis-decoupled evaluation protocol for multi-turn medical consultation agents. Each policy-elicited history is scored by a frozen shared DeepSeek-V3 diagnostic reader, and D/T/E (diagnostic support, trajectory coverage, efficiency) are computed through LLM-assisted semantic candidate generation followed by deterministic one-to-one matching. A fixed-history intervention shows that replacing each policy's own terminal diagnosis generator with the shared reader shifts diagnosis F1 by 2.2–19.0 points and reverses 5/28 and 10/28 pairwise orderings on the Record and Dialogue splits. In a GRPO study, a Qwen3-32B policy trained with a separate diagnosis+trajectory reward gains 9.6 and 4.6 Total points on the held-out splits, with both reward components contributing. The paper positions the protocol as a complement to end-to-end evaluation, not a replacement.

Significance. The fixed-history intervention is a clean and convincing demonstration of the attribution confound: terminal diagnosis generation materially changes between-policy comparisons even when histories are identical. The deterministic one-to-one matching and saved candidate correspondences are genuine reproducibility strengths, as is the alternative-reader sensitivity analysis in Table S16. If the measurement pipeline had external validation, this would be a useful methodological contribution for consultation-agent evaluation. However, the central claim—that the shared diagnostic reader measures 'diagnostic support' in a way that supports comparison—rests on an unvalidated LLM reader and an unvalidated semantic judge. The paper is honest about these dependencies (Appendix H, Limitations), but the lack of physician agreement or clinical-outcome anchoring leaves the decoupling claim protocol-relative rather than externally grounded.

major comments (3)
  1. [Shared Diagnostic Reader; Appendix D, Table S5] The decoupling claim is load-bearing but the shared reader is unvalidated. The paper presents no physician agreement, no clinician rating of DS-V3's diagnosis outputs on held-out histories, and no clinical-outcome criterion. Table S16 only swaps DS-V3 for two other unvalidated LLM readers; shared systematic bias would not be detected. Because histories differ across policies, a reader bias can interact with history content and change rankings, not merely add a constant. Please add a human-validation study: have physicians rate diagnostic support on a stratified sample of histories, compare DS-V3 D scores and resulting policy rankings against those ratings, and report agreement/disagreement. This is necessary to support the term 'diagnostic support' rather than 'diagnosis under an arbitrary frozen reader.'
  2. [Appendix B, Eq. (1); Table S3] Deterministic one-to-one matching guarantees reproducibility, but the candidate set C is generated by an LLM judge with no measured coverage. If the judge systematically misses admissible semantic edges, TP/FN counts are biased and the D/T scores and reversal counts inherit that bias. The paper explicitly leaves 'exhaustive coverage of admissible semantic matches' outside scope (Appendix H), but this is not a minor caveat: the scores are only as valid as the candidate set. Please audit candidate-generation completeness on a sample (e.g., physicians or trained annotators manually enumerate admissible matches for 20–30 cases) and report recall of the judge's candidate set, with separate analyses for diagnosis and trajectory dimensions.
  3. [Evaluation-Informed Training; Table S9] The training reward uses DS-3.2 and targets the same diagnosis and trajectory dimensions that the frozen held-out evaluator measures with DS-V3. The reported gains therefore show that a policy can improve on the protocol-conditioned metrics when optimized for them; they do not establish 'downstream utility' in any clinical sense. The abstract's 'downstream utility' phrasing is stronger than the evidence supports. Please reframe the training experiment as an internal-consistency check of the metric (optimizing the metric improves the metric), and state explicitly that external clinical validity has not been demonstrated. This is especially important because the paper's contribution 3 is framed as evaluation-informed policy development.
minor comments (4)
  1. [Table S6] The first column header appears to read 'MQuestion'; this is likely a typo for 'Question' or 'Metric question.'
  2. [Appendix B, Eqs. (3)–(6)] The efficiency timing map parameters (lambda=0.15, breakpoints, penalty 0.3) are prespecified but no sensitivity analysis is reported for them, while sensitivity is reported for the D/T/E weights. A short robustness check for these parameters would strengthen the claim that E is a stable auxiliary axis.
  3. [Appendix A.2] Trajectory targets are described as a 'reviewed consensus reference' rather than independent parallel annotations. Reporting inter-annotator agreement or at least the number of targets changed during adjudication would help readers assess reference quality.
  4. [Limitations] The Limitations and Appendix H already concede that results are conditional on the frozen reader and candidate-generation protocol. This is commendable, but the abstract and contribution list should mirror that caveat more prominently, especially the phrase 'downstream utility.'

Circularity Check

0 steps flagged

No significant circularity; the central claims are empirical, explicitly protocol-conditional, and not reduced to their inputs by construction.

full rationale

The paper does not contain a load-bearing step in which a 'prediction' or 'derived' result is equivalent by definition or by fitted input to its own assumptions. The central decoupling claim is a protocol design: a frozen shared DS-V3 diagnostic reader maps every history to a terminal diagnosis, and D is defined as micro-F1 between that reader's output and reference diagnoses. This is an explicit operational definition, not a hidden reduction, and the paper repeatedly states that results are conditional on the frozen reader and candidate-generation protocol (Appendix H: 'D/T/E should be read as protocol-conditioned research metrics rather than clinical endpoints'; Limitations: 'Results are conditional on the frozen shared diagnostic reader and candidate-generation protocol'). The fixed-history intervention (Table S4) is an empirical demonstration that replacing the terminal generator changes F1 and pairwise rankings; it does not presuppose the conclusion it illustrates. The GRPO training study uses case-disjoint training and held-out evaluation, a separate training-time reward, and a frozen held-out evaluator, so the reported gains are generalization results rather than fitted parameters renamed as predictions. No load-bearing argument rests on self-citation: the cited references (DeepSeek-V3, GRPO, Qwen3, SLIME, etc.) are external technical sources, and no uniqueness or ansatz claim is imported from prior work by the same authors. The lack of external clinical validation of the LLM reader and semantic judge is a validity limitation that the paper explicitly acknowledges, but it does not constitute circularity in the derivation chain. No specific equation-level reduction or fitted-input-called-prediction step can be exhibited, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The evaluation rests on hand-set aggregation and timing weights, unvalidated LLM judge and shared reader, and annotator-built references without reliability statistics. The training study additionally optimizes a reward constructed from the same D/T dimensions that are later measured on the same testbed.

free parameters (4)
  • Total aggregate weights (0.5D, 0.4T, 0.1E) = 0.5/0.4/0.1
    Hand-specified aggregate formula (Eq. 2); headline margins depend on these weights, though rank order is robust to seven tested alternatives (Table S15).
  • Efficiency timing map parameters = λ=0.15; breakpoints 2/3/5/8; lower bound 0.1; inefficiency penalty 0.3
    Hand-set piecewise timing curve (Eqs. 3–6); no data-driven justification is provided.
  • Importance-rating threshold for trajectory targets = highest level only
    Only highest-importance targets enter the frozen Q set; this affects the T denominator and scores.
  • S2/S3 case-pattern thresholds = ΔT≥10, ST≥0.70, ΔD≥−5 (S2); ΔD≥10, SD≥0.70, 3-of-5 runs (S3)
    Defined for the case-level heterogeneity analysis; post-hoc relative to the main result and used to select 62/170 and 7/170 illustrative cases (Table S19).
axioms (5)
  • domain assumption The frozen DS-V3 shared diagnostic reader provides a valid history-to-diagnosis mapping for measuring diagnostic support
    The entire D score and the decoupling argument treat the frozen reader's diagnosis as the standard; no external clinical validation or physician agreement is reported.
  • domain assumption The LLM judge (DS-V3) proposes correct and complete semantic candidate matches under the directional coverage rules
    TP/FP/FN counts for D and T are computed from these candidate edges; no human agreement or accuracy analysis of the judge is reported (Appendix B, Table S3).
  • domain assumption The reference diagnoses G and trajectory targets Q are correct and complete
    Constructed by 15 annotators with staged review, but no inter-annotator agreement or external validation is reported (Appendix A).
  • domain assumption Grounded simulator responses from F are sufficiently realistic to evaluate consultation policies
    Simulated consultations are the only interaction channel; the paper itself flags that real patient diversity is not captured.
  • ad hoc to paper The Total formula (0.5D+0.4T+0.1E) is a meaningful system-level ordering
    Weights are hand-chosen; sensitivity is tested only for rank order, not for the interpretation of component trade-offs.
invented entities (1)
  • Frozen shared diagnostic reader (DS-V3) no independent evidence
    purpose: Holds terminal diagnosis generation fixed across policies so that diagnosis scores attribute to the elicited history alone
    The reader's diagnosis quality is not benchmarked against an external standard; its outputs define 'diagnostic support,' making the evaluation conditional on an unvalidated model choice.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents." pith.science (2026). https://pith.science/paper/XVPWY4TI

@misc{pith2026260718999,
  author       = {Pith},
  title        = {Pith review of: MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVPWY4TI}},
  note         = {Machine review of arXiv:2607.18999}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating multi-turn medical consultation agents requires judging the diagnostic support provided by the histories they elicit through interaction. Yet coupled evaluation lets each policy both elicit the history and generate the terminal diagnosis, so a diagnosis score confounds the elicited history with the policy's own terminal diagnosis generator. We introduce MedDDC-Eval, a diagnosis-decoupled evaluation testbed over held-out cases derived from medical records and online consultations. It applies the same frozen shared diagnostic reader to every policy-elicited history, holding terminal diagnosis generation fixed across policies and enabling comparison under the shared diagnostic reader. It reports diagnostic support, information-acquisition coverage, and efficiency. LLM-assisted semantic matching followed by deterministic one-to-one assignment makes the diagnosis-trajectory-efficiency (D/T/E) scores auditable. In a fixed-history audit across eight policies, replacing each policy's own generator with the shared diagnostic reader shifts diagnosis F1 by 2.2-19.0 points and reverses 18% and 36% of pairwise orderings on the Record and Dialogue splits. To examine downstream utility, we use standard Group Relative Policy Optimization (GRPO) with a separate training-time reward that targets the same diagnosis and trajectory dimensions. Relative to its Qwen3-32B initialization, the trained policy gains 9.6 and 4.6 aggregate-score points on the held-out Record and Dialogue splits, respectively, and ablating either feedback signal reduces the aggregate score on both. Together, MedDDC-Eval supports comparison under a shared diagnostic reader and evaluation-informed policy development, while complementing end-to-end evaluation when terminal diagnosis generation is also part of the target capability.

Figures

Figures reproduced from arXiv: 2607.18999 by Guofeng Zhang, Huaiyi Fang, Jianwei Lv, Jinyao Liu, Junfeng Wang, Lening An, Xunxu Duan, Yizeng Quan, Yu Ouyang.

Figure 1
Figure 1. Figure 1: Fixed-history terminal-diagnosis intervention. (a) The history, prompt, and decoding settings are fixed; replacing each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MedDDC-Eval maps each history h to diagnosis output S with a frozen shared diagnostic reader and ex￾tracts diagnosis predictions and pre-summary questions. LLM judges propose semantic matches to diagnosis references G and trajectory targets Q; deterministic one-to-one matching yields TP/FP/FN for D and T, while E captures when matched targets are covered. D/T/E are reported separately; Total gives only agg… view at source ↗
Figure 3
Figure 3. Figure 3: Case-level trained–base changes under the shared [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    R. K. Arora, J. Wei, R. Soskin Hicks, P. Bowman, J. Qui \ n onero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. doi:10.48550/arXiv.2505.08775. URL https://arxiv.org/abs/2505.08775

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025

  2. [2]

    Baichuan-M2 Team , C. Dou, C. Liu, F. Yang, F. Li, J. Jia, M. Chen, Q. Ju, S. Wang, S. Dang, T. Li, X. Zeng, Y. Zhou, C. Zhu, D. Pan, F. Deng, G. Ai, G. Dong, H. Zhang, J. Tai, J. Hong, K. Lu, L. Sun, P. Guo, Q. Ma, R. Xin, S. Yang, S. Zhang, Y. Mo, Z. Liang, Z. Zhang, H. Cui, Z. Zhu, and X. Wang. Baichuan-m2: Scaling medical capability with large verifie...

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.02208 2025

  3. [3]

    S. Bedi, H. Cui, M. Fuentes, A. Unell, M. Wornow, J. M. Banda, N. Kotecha, T. Keyes, Y. Mai, M. Oez, et al. Medhelm: Holistic evaluation of large language models for medical tasks. arXiv preprint arXiv:2505.23802, 2025. doi:10.48550/arXiv.2505.23802. URL https://arxiv.org/abs/2505.23802

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23802 2025

  4. [4]

    J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang. Huatuogpt-o1, towards medical complex reasoning with LLMs . arXiv preprint arXiv:2412.18925, 2024. doi:10.48550/arXiv.2412.18925. URL https://arxiv.org/abs/2412.18925

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.18925 2024

  5. [5]

    Deepseek-v3 technical report

    DeepSeek-AI . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2025. doi:10.48550/arXiv.2412.19437. URL https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2025

  6. [6]

    J. Ding, L. Lu, C. Ding, M. Bian, J. Chen, W. Pang, R. Chen, X. Peng, R. Lu, S. Ren, G. Zhu, X. Wu, Z. Liu, R. Zhang, L. Jiang, B. Han, Y. Wang, and J. Xu. Medbench v4: A robust and scalable benchmark for evaluating chinese medical language models, multimodal models, and intelligent agents. arXiv preprint arXiv:2511.14439, 2025. doi:10.48550/arXiv.2511.14...

    doi:10.48550/arxiv.2511.14439 2025

  7. [7]

    Djuhera, S

    A. Djuhera, S. R. Kadhe, F. Ahmed, and H. Boche. Tsr: Trajectory-search rollouts for multi-turn rl of llm agents. arXiv preprint arXiv:2602.11767, 2026. doi:10.48550/arXiv.2602.11767. URL https://arxiv.org/abs/2602.11767

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.11767 2026

  8. [8]

    L. Feng, Z. Xue, T. Liu, and B. An. Group-in-group policy optimization for llm agent training. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/420c9f777c0b4f78d515e53cf74d58b2-Abstract-Conference.html

    2025

  9. [9]

    Glm-5: From vibe coding to agentic engineering

    GLM-5 Team . Glm-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. doi:10.48550/arXiv.2602.15763. URL https://arxiv.org/abs/2602.15763

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.15763 2026

  10. [10]

    L. Gong, W. Fang, T. Yang, D. Tao, C. Guo, P. Wei, B. Xie, J. Guan, Z. Chen, F. Shi, J. Gu, and J. Liu. Meddialogrubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models. arXiv preprint arXiv:2601.03023, 2026. doi:10.48550/arXiv.2601.03023. URL https://arxiv.org/abs/2601.03023

    doi:10.48550/arxiv.2601.03023 2026

  11. [11]

    Jiang, K

    Y. Jiang, K. C. Black, G. Geng, D. Park, J. Zou, A. Y. Ng, and J. H. Chen. Medagentbench: A realistic virtual ehr environment to benchmark medical llm agents. arXiv preprint arXiv:2501.14654, 2025. doi:10.48550/arXiv.2501.14654. URL https://arxiv.org/abs/2501.14654

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.14654 2025

  12. [12]

    Y. Lai, K. Liu, Z. Wang, W. Ma, and Y. Liu. Doctor-r1: Mastering clinical inquiry with experiential agentic reinforcement learning. arXiv preprint arXiv:2510.04284, 2025. doi:10.48550/arXiv.2510.04284. URL https://arxiv.org/abs/2510.04284

    doi:10.48550/arxiv.2510.04284 2025

  13. [13]

    G. Lee, E. Bach, E. Yang, T. Pollard, A. Johnson, E. Choi, Y. Jia, and J. H. Lee. Fhir-agentbench: Benchmarking llm agents for realistic interoperable ehr question answering. arXiv preprint arXiv:2509.19319, 2025. doi:10.48550/arXiv.2509.19319. URL https://arxiv.org/abs/2509.19319

    doi:10.48550/arxiv.2509.19319 2025

  14. [14]

    J. Li, P. Zhou, R. Meng, M. P. Vadera, L. Li, and Y. Li. Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms. In Findings of the Association for Computational Linguistics: EACL 2026, pages 6227--6243, Rabat, Morocco, 2026 a . Association for Computational Linguistics. doi:10.18653/v1/2026.findings-eacl.328. URL ht...

    work page doi:10.18653/v1/2026.findings-eacl.328 2026

  15. [15]

    Y. Li, X. Jie, W. Ruan, X. Zhang, H. Zhu, Y. Gao, C. Du, and R. Liu. Beyond idealized patients: Evaluating llms under challenging patient behaviors in medical consultations. arXiv preprint arXiv:2603.29373, 2026 b . doi:10.48550/arXiv.2603.29373. URL https://arxiv.org/abs/2603.29373

    doi:10.48550/arxiv.2603.29373 2026

  16. [16]

    R. Liu, I. Q. Mohiuddin, A. J. Schoeffler, K. Renduchintala, A. Nayak, P. L. Vemu, S. C. Vedak, K. C. Black, J. L. Havlik, I. Ogunmola, et al. Physicianbench: Evaluating llm agents in real-world ehr environments. arXiv preprint arXiv:2605.02240, 2026. doi:10.48550/arXiv.2605.02240. URL https://arxiv.org/abs/2605.02240

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.02240 2026

  17. [17]

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. doi:10.48550/arXiv.2503.20783. URL https://arxiv.org/abs/2503.20783

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.20783 2025

  18. [18]

    Munnangi and S

    M. Munnangi and S. Savage. Threadmed-qa: A multi-turn medical dialogue benchmark from real patient questions. arXiv preprint arXiv:2603.11281, 2026. doi:10.48550/arXiv.2603.11281. URL https://arxiv.org/abs/2603.11281

    work page doi:10.48550/arxiv.2603.11281 2026

  19. [19]

    Gpt-5.4 model documentation

    OpenAI . Gpt-5.4 model documentation. https://platform.openai.com/docs/models/gpt-5.4, 2026. Accessed 2026-07-25

    2026

  20. [20]

    C. Qiao, J. Huang, D. Zhao, Z. Liu, Y. Shen, B. Cheng, W. Lin, and K. Wu. Medconsultbench: A full-cycle, fine-grained, process-aware benchmark for medical consultation agents. arXiv preprint arXiv:2601.12661, 2026 a . doi:10.48550/arXiv.2601.12661. URL https://arxiv.org/abs/2601.12661

    work page doi:10.48550/arxiv.2601.12661 2026

  21. [21]

    Y. Qiao, L. Liu, Y. Shen, J. Wang, J. Gu, Z. Chu, and K. Ren. Ehr-complex: Benchmarking medical agents for complex clinical reasoning. arXiv preprint arXiv:2606.23301, 2026 b . doi:10.48550/arXiv.2606.23301. URL https://arxiv.org/abs/2606.23301

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.23301 2026

  22. [22]

    Sanghvi, N

    A. Sanghvi, N. Akash, R. Imam, A. Sharma, and M. Jain. Medxagent: Multi-agent consultation for interactive medical diagnosis. arXiv preprint arXiv:2606.03416, 2026. doi:10.48550/arXiv.2606.03416. URL https://arxiv.org/abs/2606.03416

    Pith/arXiv arXiv doi:10.48550/arxiv.2606.03416 2026

  23. [23]

    Schmidgall, R

    S. Schmidgall, R. Ziaei, C. Harris, E. Reis, J. Jopling, and M. Moor. Agentclinic: A multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960, 2024. doi:10.48550/arXiv.2405.07960. URL https://arxiv.org/abs/2405.07960

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.07960 2024

  24. [24]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. doi:10.48550/arXiv.2402.03300. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  25. [25]

    Singhal, S

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620: 0 172--180, 2023. doi:10.1038/s41586-023-06291-2. URL https://www.nature.com/articles/s41586-023-06291-2

    doi:10.1038/s41586-023-06291-2 2023

  26. [26]

    A. F. Tchango, R. Goel, Z. Wen, J. Martel, and J. Ghosn. Ddxplus: A new dataset for automatic medical diagnosis. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/cae73a974390c0edd95ae7aeae09139c-Abstract-Datasets_and_Benchmarks.html

    2022

  27. [27]

    T. Tu, M. Schaekermann, A. Palepu, et al. Towards conversational diagnostic artificial intelligence. Nature, 642: 0 442--450, 2025. doi:10.1038/s41586-025-08866-7. URL https://www.nature.com/articles/s41586-025-08866-7

    doi:10.1038/s41586-025-08866-7 2025

  28. [28]

    X. Wang, G. Chen, D. Song, Z. Zhang, Z. Chen, Q. Xiao, J. Chen, F. Jiang, J. Li, X. Wan, B. Wang, and H. Li. Cmb: A comprehensive medical benchmark in chinese. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6184--6205. Associat...

    work page doi:10.18653/v1/2024.naacl-long.343 2024

  29. [29]

    Q. Wei, S. Zeng, C. Li, W. Brown, O. Frunza, W. Deng, A. Schneider, Y. Nevmyvaka, Y. K. Zhao, A. Garcia, and M. Hong. Reinforcing multi-turn reasoning in llm agents via turn-level reward design. arXiv preprint arXiv:2505.11821, 2025. doi:10.48550/arXiv.2505.11821. URL https://arxiv.org/abs/2505.11821

    doi:10.48550/arxiv.2505.11821 2025

  30. [30]

    Yang et al

    A. Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. doi:10.48550/arXiv.2505.09388. URL https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  31. [31]

    Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. doi:10.48550/arXiv.2503.14476. URL https://arxiv.org/abs/2503.14476

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.14476 2025

  32. [32]

    Zhang, X

    H. Zhang, X. Liu, B. Lv, X. Sun, B. Jing, I. L. Iong, Z. Hou, Z. Qi, H. Lai, Y. Xu, R. Lu, H. Wang, J. Tang, and Y. Dong. Agentrl: Scaling agentic reinforcement learning with a multi-turn, multi-task framework. arXiv preprint arXiv:2510.04206, 2025. doi:10.48550/arXiv.2510.04206. URL https://arxiv.org/abs/2510.04206

    doi:10.48550/arxiv.2510.04206 2025

  33. [33]

    Zhang, S

    K. Zhang, S. Zeng, E. Hua, N. Ding, Z.-R. Chen, Z. Ma, H. Li, G. Cui, B. Qi, X. Zhu, X. Lv, J.-F. Hu, Z. Liu, and B. Zhou. Ultramedical: Building specialized generalists in biomedicine. In Advances in Neural Information Processing Systems 37, pages 26045--26081. Neural Information Processing Systems Foundation, Inc., 2024. doi:10.52202/079017-0819. URL ht...

    work page doi:10.52202/079017-0819 2024

  34. [34]

    Z. Zhu, C. Xie, X. Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository

    2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.