Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Frontier medical AI fails not for lack of knowledge but because it stops seeking the data that treatment decisions require.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:58 UTC pith:ZMG3E3JJ

load-bearing objection Solid agentic oncology benchmark showing utilization collapse and novice-like biases; the context-length confound is real but does not erase the result. the 3 major comments →

arxiv 2607.10275 v1 pith:ZMG3E3JJ submitted 2026-07-11 cs.AI cs.CL

Information-seeking failures of large language models in agentic clinical reasoning

Krischan Braitsch , Laura K. Schmalbrock , Theresa Weltermann , Andrew F. Berdel , Isabella Miller , Kai Tran , Michael Heider , Sabrina Kraus
show 6 more authors
Florian Bassermann Jacqueline Lammert Sebastian Ziegelmayer Marcus Makowski Lisa C. Adams Keno K. Bressem
This is my paper
classification cs.AI cs.CL
keywords large language modelsclinical reasoninginformation-seekingagentic evaluationhematologic oncologycognitive biasessearch satisficingdiagnostic anchoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models score highly on medical exams that hand them complete cases, yet real clinical reasoning means deciding what to investigate when information is incomplete and arrives over time. This paper builds an interactive hematologic-oncology benchmark in which 32 frontier models must request one clinical data item per turn across three staged rounds before committing to a diagnosis and treatment plan. The best model reaches only 68 percent overall accuracy. The strongest predictor of success is how much of the available data a model actually requests, yet that utilization rate collapses from about 57 percent in early rounds to 26 percent in the final round, leaving molecular and cytogenetic results unexamined. Reasoning traces look locally coherent on a clinical rubric, but they largely fail to track correct conclusions. The dominant error patterns—search satisficing, anchoring, and premature closure—are the same cognitive biases that characterize novice human clinicians. The authors conclude that the binding constraint is information-seeking under uncertainty, not encoded medical knowledge.

Core claim

Across 32 frontier models and 20 complex hematologic-oncology cases, the primary performance limit is not medical knowledge but a systematic failure of proactive information-seeking under uncertainty: models under-request critical later-round data, and the resulting failure modes match those of novice clinicians under dual-process diagnostic theory.

What carries the argument

OncoRounds: a three-round agentic evaluation in which models must request one clinical information item per turn before solving; performance is scored on diagnosis, differential reasoning, and treatment, with information utilization (fraction of available items requested) as the central behavioral metric.

Load-bearing premise

The claim rests on twenty expert-written complex blood-cancer cases, a one-request-per-turn rule, and an automated judge agreeing substantially with one hematologist being a fair stand-in for real clinical information-seeking rather than an artifact of long context, prompt design, or partial-credit scoring.

What would settle it

Re-run the same models with forced higher utilization of Round-3 molecular and cytogenetic items (or with multi-item requests and shorter contexts) and check whether overall accuracy rises substantially and the search-satisficing / anchoring error rates fall; if utilization rises but accuracy and error modes stay the same, the information-seeking account is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Static complete-vignette medical benchmarks will systematically overestimate clinical readiness of current models.
  • Agentic evaluation that forces active information gathering should become a standard component of medical AI assessment.
  • Training and scaffolding that explicitly practice sequential hypothesis testing under incomplete data are needed; scale alone is unlikely to close the gap.
  • High-quality-looking reasoning traces can coexist with wrong conclusions, so they should not be trusted as automatic signals of reliability for clinicians.
  • The same novice-like biases (search satisficing, anchoring, premature closure) appear in systems that share no developmental path with human trainees.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the utilization collapse is driven mainly by long multi-turn context and positional attention bias, shorter-memory or retrieval-augmented agent designs may fix more of the deficit than further medical pre-training.
  • The knowledge-versus-seeking gap is likely to appear in other sequential high-stakes domains (emergency triage, finance under incomplete markets) that currently rely on complete-case benchmarks.
  • Differential-reasoning scores being the weakest component for every model suggests that reward models and judges may need explicit penalties for premature commitment rather than only accuracy on the final label.
  • A natural next experiment is to train or fine-tune models on trajectories that reward requesting the highest-value unobtained item before solving, then re-measure Round-3 utilization and recovery from early anchoring.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OncoRounds, an agentic benchmark in which 32 frontier LLMs must proactively request clinical data one item per turn across three staged rounds of increasing complexity before committing to diagnosis, differential, and treatment on 20 expert-authored complex hematologic oncology cases. Best overall accuracy is 68.1% (Claude Opus 4.6); information utilization is the strongest predictor of accuracy (R = 0.69) yet collapses from ~57% in R1/R2 to 26% in R3, leaving molecular/cytogenetic data unexamined. R-IDEA reasoning scores are high (91% ≥6) but only moderately correlated with accuracy; error taxonomy (search satisficing, anchoring, premature closure) mirrors novice dual-process biases. The central claim is that the primary limitation is systematic information-seeking failure under uncertainty rather than insufficient medical knowledge.

Significance. If the result holds, this is a high-value contribution: it cleanly separates information-seeking from interpretation in a clinically realistic sequential setting, scales the evaluation to 32 models with a validated LLM-as-judge (quadratic weighted κ = 0.804 vs. a board-certified hematologist), and supplies trajectory, utilization, and error analyses that static knowledge benchmarks cannot. The public release of cases, pipeline, and outputs, plus the explicit dual-process framing, would push the field beyond MedQA-style passive vignettes toward agentic clinical evaluation and training. The observed utilization–accuracy link and R3 collapse are actionable for architecture and data-curation work.

major comments (3)
  1. [Results / Discussion (utilization collapse; context-length paragraph)] Results (utilization rates; Supplementary Figure 1) and Discussion: the sharp R3 utilization drop (57% → 26%) and low request counts are load-bearing for the claim of ‘systematic failure of information-seeking’ and the dual-process analogy. The paper itself notes that by R3 the interaction history routinely exceeds 10 k tokens and cites ‘lost in the middle’ and multi-turn degradation. Without a control that holds effective context length fixed (e.g., summarized history, item-list access without trajectory accumulation, or single-turn equivalents of the same information), it remains ambiguous how much of the collapse is intrinsic search satisficing versus an artifact of the sequential design used to measure it. This directly weakens both the human-novice analogy and the assertion that the barrier is qualitative and unlikely to be resolved by scaling alone. A control experiment or a substa
  2. [Discussion / Error Taxonomy / Methods (Cognitive Error Taxonomy)] Discussion and Error Taxonomy: the claim that models exhibit ‘the same cognitive biases that characterize novice clinicians under dual-process models’ rests on literature analogy and LLM-annotated traces, not a direct human baseline on OncoRounds. Without at least a small cohort of hematologists (or trainees) run under identical single-request-per-turn, three-round constraints, the convergence remains interpretive rather than demonstrated. This is load-bearing for the strongest framing of the paper.
  3. [Methods (Scoring) / Results (Overall Performance) / Limitations] Methods (Automated Evaluation Pipeline) and Limitations: differential-reasoning accuracy is the weakest component for all 32 models (mean gap 23 pp) and is also acknowledged as the most subjective. The majority-vote GPT-5 Mini ensemble (κ = 0.804) is systematically stricter than the single expert, and partial-credit boundaries for differentials and treatment priorities are inherently less standardized than primary diagnosis. While the authors flag this, the ranking of ‘differential reasoning as the primary challenge’ is used to support the information-seeking narrative; a sensitivity analysis with human re-scoring of a larger differential subset, or explicit down-weighting of that component in the composite, would strengthen the claim.
minor comments (5)
  1. [Table 1 / Case Development / Limitations] Table 1 and case development: the 20 cases are all relapsed/refractory/post-transplant; a short note on how performance might differ on de-novo or lower-complexity presentations would help readers gauge generalizability beyond the Limitations paragraph.
  2. [Figure 2 / Supplementary Figure 5] Figure 2 and Supplementary Figure 5: the alluvial and donut visualizations of anchoring are informative, but the distinction between ‘subtype refinement failure’ and true category shift could be made more visually explicit (e.g., color coding of partial vs. full incorrect).
  3. [Methods (Clinical Reasoning Evaluation)] Methods (Clinical Reasoning Evaluation): R-IDEA was scored by Claude Opus 4.6; a brief statement on whether the same model family was used for any of the evaluated candidates, and any decontamination steps, would be useful.
  4. [Supplementary Appendix / Code Availability] Supplementary Appendix: the 49 curated reasoning-trace excerpts are valuable; ensuring the selection script and full set of 1 008 scored entries are released with the code would improve reproducibility.
  5. [Abstract / Results (Overall Performance)] Abstract and Introduction: the phrase ‘the best achieved only 68% overall accuracy’ is accurate but could briefly note the composite definition (balanced accuracy of diagnosis / differential / treatment) so readers do not equate it with pure diagnostic accuracy.

Circularity Check

0 steps flagged

Empirical agentic evaluation with independent reference standards; no derivation that reduces to its own inputs.

full rationale

OncoRounds is an empirical benchmark paper, not a first-principles derivation. Overall accuracy, per-component scores, information utilization (fraction of available items requested), R-IDEA totals, and error-taxonomy rates are measured against expert-authored round-specific reference standards and against independently applied rubrics. The reported correlation (utilization vs accuracy, R = 0.69) is a statistical association between two measured quantities, not a fitted parameter re-labeled as a prediction. The dual-process framing (search satisficing, anchoring, premature closure) is an interpretive mapping of observed failure modes onto Croskerry’s external taxonomy; it does not define the measured quantities in terms of those labels. No equation equates a claimed result to a quantity constructed from the same data used to score it. Self-involvement in case authorship and judge calibration is ordinary for a new benchmark and is not load-bearing circularity of the kinds enumerated. The paper is self-contained against its external reference standards; circularity score is therefore 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

Empirical evaluation paper. Load-bearing elements are domain assumptions about clinical realism and the dual-process analogy, plus the constructed benchmark itself. No free parameters are fitted to produce the central claim; correlations are descriptive. OncoRounds is an invented evaluation entity whose validity rests on expert authorship and one human calibration.

axioms (4)
  • domain assumption Dual-process models of diagnostic reasoning (Croskerry) correctly characterize the information-seeking failures of both novice clinicians and current LLMs.
    Used throughout Discussion and error taxonomy to interpret search satisficing, anchoring and premature closure as the same cognitive biases.
  • domain assumption The 20 de-novo expert-authored complex hematologic cases with staged information release constitute a representative and leakage-free test of real clinical information-seeking.
    Cases developed by one hematologist and reviewed by seven others; no public case reports used. Central to generalizability claims.
  • domain assumption Majority vote of five GPT-5 Mini instances against expert-authored references is a sufficiently accurate and unbiased proxy for clinical correctness (κ = 0.804).
    All accuracy, differential and treatment scores rest on this judge ensemble.
  • ad hoc to paper Single-request-per-turn and three fixed clinical rounds fairly isolate information-seeking skill without confounding by context length or multi-turn degradation.
    Design choice of OncoRounds; authors themselves note context saturation as a possible amplifier of the observed utilization drop.
invented entities (1)
  • OncoRounds agentic evaluation framework no independent evidence
    purpose: Force proactive, sequential information requests across three clinically staged rounds before diagnosis and treatment commitment.
    New benchmark constructed for this study; validity depends on expert case design and judge calibration rather than external independent evidence.

pith-pipeline@v1.1.0-grok45 · 55711 in / 3025 out tokens · 41125 ms · 2026-07-14T12:58:23.829619+00:00 · methodology

0 comments
read the original abstract

Large language models achieve high scores on medical knowledge assessments, yet clinical reasoning requires actively deciding what to investigate under uncertainty. We developed an agentic evaluation framework in hematologic oncology in which models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization, the fraction of available data actually requested, was the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.001), yet utilization collapsed from 57% to 26% in the final round, leaving molecular and cytogenetic data critical for treatment selection unexamined. Reasoning traces scored high on a clinical reasoning rubric (91% above threshold) but decorrelated from accuracy, revealing a gap between locally coherent rationales and globally correct conclusions. Error analysis identified search satisficing, anchoring and premature closure as the dominant failure modes, the same cognitive biases that characterize novice clinicians under dual-process models of diagnostic reasoning. These findings demonstrate that the primary limitation of current models in clinical oncology is not insufficient medical knowledge but a systematic failure of information-seeking under uncertainty.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 6 canonical work pages

  1. [1]

    Application of precision medicine in clinical routine in haematology-challenges and opportunities.J

    Tove Wästerlid, Lucia Cavelier, Claudia Haferlach, Marina Konopleva, Stefan Fröhling, Päivi Östling, Lars Bullinger, Thoas Fioretos, and Karin E Smedby. Application of precision medicine in clinical routine in haematology-challenges and opportunities.J. Intern. Med., 292(2):243–261, August 2022. doi: 10.1111/joim.13508

    work page doi:10.1111/joim.13508 2022

  2. [2]

    Large language models encode clinical knowledge.Nature, 620(7972):172–180, August 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera Y Arcas, Dale Webster, Greg S Corrado, Y...

    doi:10.1038/s41586-023-06291-2 2023

  3. [3]

    Capabilities of GPT-4 on medical challenge problems

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, March 2023. URL https: //arxiv.org/abs/2303.13375

    Pith/arXiv arXiv 2023

  4. [4]

    How does ChatGPT perform on the united states medical licensing examination (USMLE)? theimplicationsoflargelanguagemodelsformedicaleducationandknowledgeassessment.JMIR Med

    Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. How does ChatGPT perform on the united states medical licensing examination (USMLE)? theimplicationsoflargelanguagemodelsformedicaleducationandknowledgeassessment.JMIR Med. Educ., 9:e45312, February 2023. doi: 10.2196/45312

    doi:10.2196/45312 2023

  5. [5]

    GPT-4 for information retrieval and comparison of medical oncology guidelines.NEJM AI, 1(6), May 2024

    Dyke Ferber, Isabella C Wiest, Georg Wölflein, Matthias P Ebert, Gernot Beutel, Jan-Niklas Eckardt, Daniel Truhn, Christoph Springfeld, Dirk Jäger, and Jakob Nikolas Kather. GPT-4 for information retrieval and comparison of medical oncology guidelines.NEJM AI, 1(6), May 2024. doi: 10.1056/AIcs2300235

    work page doi:10.1056/aics2300235 2024

  6. [6]

    Large language models should be used as scientific reasoning engines, not knowledge databases.Nat

    Daniel Truhn, Jorge S Reis-Filho, and Jakob Nikolas Kather. Large language models should be used as scientific reasoning engines, not knowledge databases.Nat. Med., 29(12):2983–2984, December 2023. doi: 10.1038/s41591-023-02594-z

    doi:10.1038/s41591-023-02594-z 2023

  7. [7]

    Large language models in medicine.Nat

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nat. Med., 29(8):1930–1940, August 2023. doi: 10.1038/s41591-023-02448-8

    doi:10.1038/s41591-023-02448-8 1930

  8. [8]

    Multimodal biomedical AI.Nat

    Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical AI.Nat. Med., 28(9):1773–1784, September 2022. doi: 10.1038/s41591-022-01981-2

    doi:10.1038/s41591-022-01981-2 2022

  9. [9]

    Artificialintelligenceformultimodaldataintegrationinoncology.Cancer Cell,40(10):1095–1110, October 2022

    Jana Lipkova, Richard J Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag J Vaidya, Chengkuan Chen, Luoting Zhuang, Drew F K Williamson, Muhammad Shaban, Tiffany Y Chen, and Faisal 10 Mahmood. Artificialintelligenceformultimodaldataintegrationinoncology.Cancer Cell,40(10):1095–1110, October 2022. doi: 10.1016/j.ccell.2022.09.012

    doi:10.1016/j.ccell.2022.09.012 2022

  10. [10]

    Foundation models for generalist medical artificial intelligence.Nature, 616 (7956):259–265, April 2023

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence.Nature, 616 (7956):259–265, April 2023. doi: 10.1038/s41586-023-05881-4

    doi:10.1038/s41586-023-05881-4 2023

  11. [11]

    Towards generalist biomedical AI.NEJM AI, 1(3), February 2024

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, Anil Palepu, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S Sara Mahdavi, Bradley Green, Ewa Dominows...

    doi:10.1056/aioa2300138 2024

  12. [12]

    Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology.Nat

    Dyke Ferber, Omar S M El Nahhas, Georg Wölflein, Isabella C Wiest, Jan Clusmann, Marie-Elisabeth Leßmann, Sebastian Foersch, Jacqueline Lammert, Maximilian Tschochohei, Dirk Jäger, Manuel Salto-Tellez, Nikolaus Schultz, Daniel Truhn, and Jakob Nikolas Kather. Development and validation of an autonomous artificial intelligence agent for clinical decision-m...

  13. [13]

    doi: 10.1038/s43018-025-00991-6

    doi:10.1038/s43018-025-00991-6

  14. [14]

    HowAIagentswillchange cancer research and oncology.Nat

    YongjuLee, DykeFerber, JenniferERood, AvivRegev, andJakobNikolasKather. HowAIagentswillchange cancer research and oncology.Nat. Cancer, 5(12):1765–1767, December 2024. doi: 10.1038/s43018-024- 00861-7

    work page doi:10.1038/s43018-024- 2024

  15. [15]

    Holistic evaluation of large language models for medical tasks with MedHELM.Nat

    Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey Jain, Leonardo Schettini, Mehr Kashyap, Jason Alan Fries, Akshay Swaminathan, Philip Chung, Fateme Nateghi Haredasht, Ivan Lopez, Asad Aali, Gabriel Tse, AshwinNayak,ShivamVedak,SnehaSJain,BirjuPatel,Olusey...

    doi:10.1038/s41591-025-04151-2 2026

  16. [16]

    Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning.Nat

    Mickael Tordjman, Zelong Liu, Murat Yuce, Valentin Fauveau, Yunhao Mei, Jerome Hadjadj, Ian Bolger, Haidara Almansour, Carolyn Horst, Ashwin Singh Parihar, Amine Geahchan, Anis Meribout, Nader Yatim, Nicole Ng, Phillip Robson, Alexander Zhou, Sara Lewis, Mingqian Huang, Timothy Deyer, Bachir Taouli, Hao-Chih Lee, Zahi A Fayad, and Xueyan Mei. Comparative ...

    doi:10.1038/s41591-025-03726-3 2025

  17. [17]

    Toward expert-level medical question answering with large language models.Nat

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, StephenRPfohl,HeatherCole-Lewis,DarleneNeal,QaziMamunurRashid,MikeSchaekermann,AmyWang, Dev Dash, Jonathan H Chen, Nigam H Shah, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, BradleyGreen,EwaDominowska,BlaiseAgüeraYArcas,NenadTomašev,YunLiu,Ren...

    doi:10.1038/s41591-024-03423-7 2025

  18. [18]

    On the planning abilities of large language models : A critical investigation

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models : A critical investigation. arXiv preprint arXiv:2305.15771, May 2023. URL https://arxiv.org/abs/2305.15771

    Pith/arXiv arXiv 2023

  19. [19]

    Large language models for planning: A comprehensive and systematic survey

    Pengfei Cao, Tianyi Men, Wencan Liu, Jingwen Zhang, Xuzhao Li, Xixun Lin, Dianbo Sui, Yanan Cao, Kang Liu, and Jun Zhao. Large language models for planning: A comprehensive and systematic survey. arXiv preprint arXiv:2505.19683, May 2025. URL https://arxiv.org/abs/2505.19683. 11

    Pith/arXiv arXiv 2025

  20. [20]

    A survey on LLM-as-a-Judge

    JiaweiGu,XuhuiJiang,ZhichaoShi,HexiangTan,XuehaoZhai,ChengjinXu,WeiLi,YinghanShen,Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594, October 2025. URL https://arxiv.org/abs/2411.15594

    Pith/arXiv arXiv 2025

  21. [21]

    Development of a clinical reasoning documentation assessment tool for resident and fellow admission notes: A shared mental model for feedback.J

    Verity Schaye, Louis Miller, David Kudlowitz, Jonathan Chun, Jesse Burk-Rafel, Patrick Cocks, Benedict Guzman, Yindalon Aphinyanaphongs, and Marina Marin. Development of a clinical reasoning documentation assessment tool for resident and fellow admission notes: A shared mental model for feedback.J. Gen. Intern. Med., 37(3):507–512, February 2022. doi: 10....

    work page doi:10.1007/s11606-021-06805-6 2022

  22. [22]

    A universal model of diagnostic reasoning.Acad

    Pat Croskerry. A universal model of diagnostic reasoning.Acad. Med., 84(8):1022–1028, August 2009. doi: 10.1097/ACM.0b013e3181ace703

    work page doi:10.1097/acm.0b013e3181ace703 2009

  23. [23]

    Agent- Clinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

    Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agent- Clinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv preprint arXiv:2405.07960, May 2024. URL https://arxiv.org/abs/2405.07960

    Pith/arXiv arXiv 2024

  24. [24]

    MedDialogRubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models

    LechengGong,WeiminFang,TingYang,DongjieTao,ChunxiaoGuo,PengWei,BoXie,JinqunGuan,Zixiao Chen, Fang Shi, Jinjie Gu, and Junwei Liu. MedDialogRubrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models. arXiv preprint arXiv:2601.03023, January 2026. URL https://arxiv.org/abs/2601.03023

    arXiv 2026

  25. [25]

    Self-evolving multi-agent simulations for realistic clinical interactions

    Mohammad Almansoori, Komal Kumar, and Hisham Cholakkal. Self-evolving multi-agent simulations for realistic clinical interactions. arXiv preprint arXiv:2503.22678, October 2025. URL https://arxiv.org/abs/25 03.22678

    arXiv 2025

  26. [27]

    URL https://arxiv.org/abs/2307.03172

    Pith/arXiv arXiv

  27. [28]

    Systematic evaluation of long-context LLMs on financial concepts

    Lavanya Gupta, Saket Sharma, and Yiyun Zhao. Systematic evaluation of long-context LLMs on financial concepts. arXiv preprint arXiv:2412.15386, December 2024. URL https://arxiv.org/abs/2412.15386

    Pith/arXiv arXiv 2024

  28. [29]

    LLMsgetlostinMulti-Turnconversation

    PhilippeLaban,HiroakiHayashi,YingboZhou,andJenniferNeville. LLMsgetlostinMulti-Turnconversation. arXiv preprint arXiv:2505.06120, May 2025. URL https://arxiv.org/abs/2505.06120

    Pith/arXiv arXiv 2025

  29. [30]

    Medical reasoning in LLMs: an in-depth analysis of DeepSeek R1.Front

    Birger Moëll, Fredrik Sand Aronsson, and Sanian Akbar. Medical reasoning in LLMs: an in-depth analysis of DeepSeek R1.Front. Artif. Intell., 8:1616145, June 2025. doi: 10.3389/frai.2025.1616145

    doi:10.3389/frai.2025.1616145 2025

  30. [31]

    Addressing cognitive bias in medical language models

    Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. Addressing cognitive bias in medical language models. arXiv preprint arXiv:2402.08113, February 2024. URL https://arxiv.org/abs/2402.08113

    Pith/arXiv arXiv 2024

  31. [32]

    Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: Simulation study.J

    Yuhe Ke, Rui Yang, Sui An Lie, Taylor Xin Yi Lim, Yilin Ning, Irene Li, Hairil Rizal Abdullah, Daniel Shu Wei Ting, and Nan Liu. Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: Simulation study.J. Med. Internet Res., 26:e59439, November

  32. [33]

    Medical reasoning in the era of LLMs: A systematic review of enhancement techniques and applications

    Wenxuan Wang, Zizhan Ma, Meidan Ding, Shiyi Zheng, Shengyuan Liu, Jie Liu, Jiaming Ji, Wenting Chen, Xiang Li, Linlin Shen, and Yixuan Yuan. Medical reasoning in the era of LLMs: A systematic review of enhancement techniques and applications. arXiv preprint arXiv:2508.00669, August 2025. URL https://arxiv.org/abs/2508.00669

    Pith/arXiv arXiv 2025

  33. [34]

    Automating expert-level medical reasoning evaluation of large language models.NPJ Digit

    Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, Zidu Xu, Yuen-Hei Chung, Yiyun Xing, Meng-Han Tsai, Emma Schaffer, Yucheng Shi, Ninghao Liu, Zirui Liu, and Rui Zhang. Automating expert-level medical reasoning evaluation of large language models.NPJ Digit. Med., 9(1):34, Decemb...

    work page doi:10.1038/s41746-025-02208-7 2025

  34. [35]

    morphologic remission

    Oskar Wysocki, Jessica Katharine Davies, Markel Vigo, Anne Caroline Armstrong, Dónal Landers, Rebecca Lee, and André Freitas. Assessing the communication gap between AI models and healthcare professionals: Explainability, utility and trust in AI-driven clinical decision-making.Artif. Intell., 316:103839, March 2023. doi: 10.1016/j.artint.2022.103839. 12 T...

    doi:10.1016/j.artint.2022.103839 2023

  35. [36]

    Diffuse large B-cell lymphoma NOS (without recognizing the double-hit)

  36. [37]

    Primary mediastinal B-cell lymphoma (given the mediastinal presentation)

  37. [38]

    Solve: The patient is in septic shock secondary to presumed community-acquired pneumonia. Immediate management includes

    Burkitt lymphoma (given MYC rearrangement, but morphology and Ki-67 don’t fit) Treatment for double-hit lymphoma requires intensive regimen: - DA-EPOCH-R is often preferred - OR High-dose methotrexate-basedregimens-CNSprophylaxisisESSENTIAL(highrisk)-Considerconsolidationwithauto-SCT in first remission The airway management is still critical. I need to or...

    2022

  38. [39]

    Pulmonary involvement of KS

  39. [40]

    Opportunistic infections (PCP, TB)

  40. [41]

    inactive post-treatment myeloma lesions

    [...] Case 5| AML | Round 1 | Kimi K2 Thinking Severity: medium Evaluator:Good but inefficient R1 reasoning. The model correctly identifies acute leukemia with 78% blasts and severe cytopenias. The coagulopathy reasoning (INR 1.7, bruising) appropriately raises concern for DIC. However, the model over-anchors on APL without considering therapy-related AML...

  41. [42]

    next treatment options

    [...] 45 Case 17| ALL | Round 1 | MiniMax M2.1 Severity: medium Evaluator:Weak R1 dominated by cardiac concerns that prove to be a secondary finding. The model correctly identifies the post-alloSCT/CAR-T setting and relevant complications (GVHD, relapse, infection, CRS) but then focuses disproportionately on elevated NT-proBNP and troponin rather than dis...

  42. [43]

    Valganciclovir for EBV (priority 1)

  43. [44]

    Stop lenalidomide (therapeutic, priority 2)

  44. [45]

    This might fit

    Transfusions (supportive, priority 2) Zoledronic acid might not be needed if the myeloma is not active. This might fit. Case 9| Multiple Myeloma | Round 3 | Kimi K2 Thinking Severity: low Evaluator:Adequate R3 reasoning that correctly identifies multiple myeloma with extramedullary disease as the primary diagnosis. Treatment recommendations (DRd for trans...

  45. [46]

    Coordinate anticoagulation management for procedural hold

    Imaging: skeletal survey showing lytic lesions, MRI of the mass, PET-CT for staging 5. Biopsy: plasma cell neoplasm on histology, CD138+, CD38+, CD56+, with possible cytogenetic abnormalities (del(17p), t(4;14), etc.) Given the patient’s age (81), comorbidities (AFib on anticoagulation), and likely diagnosis of multiple myeloma with plasmacytoma, treatmen...

    2022

  46. [47]

    PNH Clone: •Granulocytes: 68% FLAER-/CD24- •Monocytes: 72% FLAER-/CD14- •RBCs: 46% CD59-deficient (31% type III, 15% type II) •This is a large PNH clone, confirming PNH diagnosis

  47. [48]

    pneumoniae and VRE E

    HLH Evaluation (from previous request): •Hepatitis B/C negative •HIV negative •CMV/EBV: past infection, no active replication •Blood cultures negative •Sputum cultures negative •Viral PCR panel negative •Wound swab grew KPC K. pneumoniae and VRE E. faecium (colonization, not necessarily infection) •PET-CT: FDG-avid pulmonary nodules (known fungal sequelae...

  48. [49]

    partially_correct

    Clonal evolution with monosomy7 (MDS-associated cytogenetic abnormality) 4. Recent disseminated fusariosis with residual pulmonary involvement Risk Assessment: The presence of monosomy 7 significantly increases risk for: - Progression to AML - Treatment resistance - Poorer overall survival Monosomy 7 is considered a high-risk feature in MDS, warranting ag...

  49. [50]

    Antifungal strategy: •Continue voriconazole (primary for fusariosis, though some species resistant) •Continue terbinafine (synergistic activity) •Need ophthalmology evaluation for possible endophthalmitis •May need adjunctive therapy or consideration of liposomal amphotericin B if progression

  50. [51]

    After reviewing treatment history (5 lines including ASCT x2, CAR-T, multiple PI/IMiD/anti-CD38 combinations), correctly identifies penta-refractory status

    Transplant planning: •Urgent allogeneic HSCT is indicated (high-risk MDS with monosomy 7) •Must balance against active fungal infection (relative contraindication) •Bridge therapy decisions: –Continue pegcetacoplan (controls hemolysis) –Continue ciclosporin A (IST may not be effective now with MDS, but could be continued as bridge) –Discontinue eltrombopa...

  51. [52]

    •Subtype: AML with myelomonocytic differentiation

    Refining the Diagnosis: •Diagnosis: Acute Myeloid Leukemia (AML) with mutated FLT3. •Subtype: AML with myelomonocytic differentiation. •Risk Stratification: Adverse/Intermediate-high (FLT3-ITD high AR, NPM1 WT)

  52. [53]

    incorrect

    Refining the Treatment Plan: • Induction:Cytarabine (7 days) + Daunorubicin (3 days) +Midostaurin(targeted FLT3 inhibitor). (This is the “RATIFY” regimen). •Supportive Care:Tumor lysis prophylaxis (continue/maintain), anti-emetics, blood product support. • Infectious Prophylaxis:Antifungal (Posaconazole/Voriconazole), antibacterial (Lev- ofloxacin/Fluoroq...

  53. [54]

    Solve” Request: •“Final round - provide definitive diagnosis and treatment plan

    Final Check of the “Solve” Request: •“Final round - provide definitive diagnosis and treatment plan.” •I have the FLT3 status. This completes the diagnostic picture. •I do not have the response to Round 1/2 treatments (e.g., did the WBC go down? did the PE resolve?). •However, usually in these cases, the “Day >7” setting implies stabilization. [...] 66