Pith. sign in

REVIEW 4 major objections 5 minor 42 references

MyoCardBench, a 2,263-item cardiology benchmark, shows LLMs are strong on documentation but near chance on ECG reading and ethics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:09 UTC pith:DEGPIYU5

load-bearing objection A genuinely new, well-annotated cardiology LLM benchmark whose headline numbers are undercut by an unnamed holistic scorer and the absence of item-level statistics; worth serious review, but only after the scoring pipeline is disclosed. the 4 major comments →

arxiv 2607.25186 v1 pith:DEGPIYU5 submitted 2026-07-28 cs.CL

MyoCardBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

Xiao Li , Mouxiao Bian , Zhaodi Wu , Sijie Ren , Juechen Chen , Lu Lu , Jingru Ding , Yun Zhong
show 3 more authors
Jie Xu Yixiu Liang Junbo Ge
This is my paper
classification cs.CL
keywords large language modelscardiovascular careclinical benchmarkzero-shot evaluationECG interpretationclinical ethicsclinical documentationmultimodal reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MyoCardBench is a new benchmark for measuring large language models across the whole arc of cardiovascular care rather than through isolated examination questions. The paper argues that existing medical benchmarks miss the longitudinal, multimodal, safety-critical nature of real cardiology work, and it presents 2,263 items in 13 task-specific datasets built from de-identified records and examinations, with reference answers and atomic key points created by cardiologists and cross-reviewed by senior cardiologists. Under standardized zero-shot testing of seven models, structured documentation tasks scored well (auxiliary report integration at 86.38), while ECG image reading (17.25) and five-option clinical ethics (17.34) were near chance. The paper also reports large gaps between holistic clinical quality and explicit key-point coverage, especially in communication, emergency rescue, and treatment planning, which it interprets as fluent but incomplete responses.

Core claim

The paper's central claim, stated in its own conclusions, is that MyoCardBench is to the authors' knowledge the largest real-world, multi-task benchmark developed specifically for evaluating large language models across the cardiovascular care continuum, and offers the broadest coverage of clinically authentic cardiology scenarios reported to date. The benchmark contains 2,263 items in 13 task-specific datasets that reproduce the chronology of care—admission documentation, diagnosis and differential diagnosis, risk scoring, treatment planning, early warning, emergency rescue, ECG and image reading, chronic health and medication management, communication, and ethics. The paper reports that in

What carries the argument

The load-bearing object is the benchmark construction itself: 13 task-specific datasets organized into three clinical dimensions, with prompts designed to preserve chronology, comorbidities, distractors, and evolving findings rather than isolate facts. Two scoring mechanisms carry the evaluation: atomic key-point macro-recall, which checks whether each pre-specified essential element—diagnosis, action, contraindication, monitoring step, escalation threshold—appears in the response, and a holistic clinical-quality score, which grades overall correctness, completeness, consistency, organization, and safety against a task-specific rubric. The interaction between the two metrics is the analytic

Load-bearing premise

The load-bearing premise is that the 'holistic clinical-quality score' is a valid, independent measure of response quality; the paper does not say whether a human or an automated judge computed it, so if an uncalibrated LLM judge produced those scores, the reported quality scores and the large gaps between quality and key-point coverage would reflect grader bias rather than clinical competence.

What would settle it

Take a random sample of open-ended responses, have two independent panels of cardiologists score them blind using the same task-specific rubrics, and compare with the paper's reported scores. If human holistic scores track key-point coverage more closely than the reported automated scores do, the large quality-recall gaps (e.g., about 52 points for communication) are an artifact of the grader rather than a clinical property of the models. Re-keying the CardioEthics options under expert audit would similarly settle whether 17% accuracy is a model deficit or an answer-key problem.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark's findings hold, the safest near-term clinical use of LLMs is drafting and integrating structured documents; all seven models scored above 84 on auxiliary report integration.
  • ECG interpretation and clinical ethics are not ready for autonomous use: ECG reading capped near 20 and ethics accuracy sat below the 20% random-choice level for six of seven models.
  • Safety evaluation should not rely on a single quality score; reporting key-point recall alongside holistic quality is necessary because plausible-sounding answers can omit critical steps.
  • The benchmark's proposed future step of weighting key points by urgency, potential harm, and recoverability could turn the current completeness measures into a safety-weighted score.
  • Because macro-average and item-weighted rankings agreed, differences in task size did not distort overall model ordering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct check of the ethics dataset is within reach: if expert re-audit and re-keying changes any meaningful fraction of the 215 five-option items, the near-chance accuracy may be a measurement problem, not a statement about ethical reasoning.
  • The large holistic-quality/key-point gaps suggest the automated grader may reward fluency and organization; comparing human cardiologist ratings against the reported scores on the same outputs would show whether the gap lives in the models or in the measurement.
  • The same care-continuum design could be adapted to other longitudinal specialties, and the pattern reported here—documentation ahead of multimodal interpretation, ethics far behind—is a plausible default hypothesis to test in those settings.
  • Since ECG scores vary little across models, the bottleneck is likely the multimodal perception and reasoning step rather than cardiology knowledge; coupling a generalist LLM with a dedicated ECG encoder inside this benchmark would test that separation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. MyoCardBench is a real-world benchmark for evaluating LLMs in cardiovascular care, comprising 2,263 items across 13 task-specific datasets, annotated by 16 cardiologists and cross-reviewed by two senior cardiologists. Seven LLMs were evaluated zero-shot, producing 15,841 outputs. Open-ended tasks were scored by key-point macro-recall and a holistic clinical-quality score; CardioEthics was scored by accuracy. GPT-5.4 achieved the highest macro-average (62.55) and ranked first in all dimensions. CardioAuxReport was strongest (86.38), while CardioECGRead (17.25) and CardioEthics (17.34) were weakest. The authors claim this is the largest real-world, multi-task benchmark specifically developed for evaluating LLMs across the cardiovascular care continuum, with the broadest coverage to date.

Significance. If the benchmark resource is as described, it is a substantial contribution: it covers a broad range of clinically authentic cardiology tasks beyond exam-style questions, and its expert-annotation workflow (16 annotators plus senior cross-review) is a strength. The finding that all LLMs are near-chance at ECG interpretation and ethics is clinically relevant and falsifiable. The decision to report both key-point recall and holistic quality separately is also useful. However, the manuscript's scientific claims regarding the holistic clinical-quality scores are under-supported: the scorer is never specified (human, LLM judge, or rubric-based), and the analysis is entirely descriptive, with no item-level confidence intervals or significance testing. These limitations are acknowledged in the paper itself but are not resolved.

major comments (4)
  1. [§2.5 and §2.7] The holistic clinical-quality score, which is load-bearing for the headline results and the large judge-recall gaps in CardioComm, CardioEmergRescue, and CardioTreatPlan, is never operationalized. Section 2.5 states that the score 'assessed the response against the reference and task-specific rubric' but does not state whether the assessor was a cardiologist, a rubric-scored LM, or an LLM judge. Section 2.7 says only 'aggregate LLM-task results' were available, so the scoring pipeline cannot be inspected. Since the Discussion (§4.5, §4.6) explicitly warns about LLM-as-a-judge biases (verbosity, stylistic plausibility), the manuscript must either disclose the scorer and calibration, or refrain from interpreting the holistic scores as independent clinical-quality measurements. Without this, the quality scores and the associated conclusions are not independently verifiable.
  2. [§2.7] No item-level statistics are reported. The text states that 'item-level confidence intervals, paired significance tests, calibration analyses, subgroup analyses, and patient-level clustering were not estimated.' Given that the paper makes quantitative claims ('GPT-5.4 achieved the highest macro-average 62.55', 'Gemini 3.1 Pro ranked second at 59.95', a 2.60-point margin), the absence of any confidence interval or significance test makes it impossible to assess whether the differences are meaningful or noise. This is especially important in tasks with small between-model ranges (e.g., CardioECGRead, where the text itself notes the top difference is 0.06 points and 'not clinically meaningful'). The analysis is acceptable as descriptive, but the overall ranking claims need explicit acknowledgement that they may not reflect statistically reliable differences; this caution is currently unders
  3. [§3.3] The claim that 'The consistently low ECG and ethics scores across the 7 LLMs indicate that these results were not attributable to a single poorly performing system' is partially undermined by the next sentence's admission that the ethics result 'may reflect genuinely difficult scenarios, option ambiguity, keying or option-order problems, output normalization, or systematic reasoning failure' (§4.5). Without item-level ethics analysis or a comparison to chance with confidence intervals, the near-chance ethics scores (15.35–20.47% on five-option questions) cannot be confidently interpreted as poor model ethics reasoning; they could reflect benchmark or keying problems. This is explicitly acknowledged in §4.5, so the abstract's presentation of ethics at 17.34 as a substantive result is over-strong.
  4. [§2.6 and §3.2] The paper states models were evaluated 'exactly as recorded in the updated results workbook' and that only aggregate results were available. This is a transparency problem for a benchmark paper. The raw item-level predictions, the holistic scores per item, and the rubric used for holistic scoring should be released along with the dataset, otherwise the results are not reproducible and the benchmark's utility for future comparisons is limited. The data availability section only mentions the dataset itself, not the scoring outputs.
minor comments (5)
  1. [§2.5] The composite score weights are described only as 'prespecified' with 'greater emphasis on explicit recovery'; the actual weights are not given. Please report the exact formula or weights in the Methods or appendix.
  2. [§3.5] Figure 5C is described in text but not visually included in the provided manuscript; if it is omitted from the main text, please clarify or refer to a supplement.
  3. [§4.4] The phrase 'updated composite scores' implies previous composite scores differed; this is confusing without a description of what changed. Please clarify the 'updated' status of the workbook or remove the word.
  4. [Abstract/Introduction] The claim 'largest real-world, multi-task benchmark specifically developed for evaluating LLMs across the cardiovascular care continuum' and 'broadest coverage' appears in the abstract, introduction, discussion, and conclusion. This is acceptable, but it should be accompanied by a brief specification of the comparator (e.g., number of items vs. existing cardiology LLM benchmarks) to make the 'largest' claim verifiable.
  5. [§6 Data Availability] The data availability statement gives a MedBench URL, but the paper should also state whether the evaluation code and model outputs (item-level scores) are available, and under what license.

Circularity Check

0 steps flagged

No circularity: MyoCardBench is an empirical benchmark with externally grounded key-point and accuracy scoring; the opaque holistic-quality scorer is a transparency limitation, not a demonstrated circular reduction.

full rationale

MyoCardBench is a benchmark-development and model-evaluation study, not a derivation from first principles. The central outputs are measured LLM scores on 2,263 items, and the two quantitatively load-bearing components are key-point macro-recall ('the number of matched atomic key points was divided by the total number of reference key points') and accuracy on five-option CardioEthics items ('scored as the percentage of correctly answered questions'). Both are externally grounded in human-constructed references and key points, so rankings on these measures are not equivalent to their inputs by construction. The 'holistic clinical-quality score' is vaguely specified in §2.5 ('assessed the response against the reference and task-specific rubric'), and §4.6 acknowledges that 'LLM-as-a-Judge can recognize semantically equivalent responses and global coherence but may overvalue verbosity or stylistic plausibility' without disclosing whether such a judge was used here. This is a genuine transparency and reproducibility gap that bears on the validity of the holistic-quality numbers and the judge-recall gaps (e.g., CardioComm 52.71), but the paper does not establish that the holistic score was computed by an LLM, nor does any equation or definition reduce the reported score to a fitted parameter or to a self-citation. The paper explicitly flags its own limitations (§4.5: 'Aggregate results cannot distinguish these explanations' for ethics; §4.6: 'Further validation with independent clinician panels... would provide additional evidence'), and no load-bearing argument rests on a self-citation or a uniqueness theorem imported from the authors' prior work. Therefore the enumerated circularity patterns do not apply; the appropriate verdict is no significant circularity, with the scorer-opacity issue recorded as a correctness/verifiability risk rather than circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The benchmark's claims rest on clinical annotation and evaluation assumptions rather than fitted physical parameters. The main unstated inputs are the composite-score weights, key-point matching rules, and the identity/behavior of the holistic quality scorer.

free parameters (3)
  • Open-ended composite weights = not reported
    The composite score combines key-point recall and holistic quality with 'greater emphasis on explicit recovery' (Section 2.5), but the actual weights are not stated; different weights can change task-level rankings.
  • Key-point matching thresholds = not reported
    Key-point macro-recall requires judging whether model text matches a reference key point, but the matching algorithm, synonym rules, and thresholds are not specified.
  • Task-specific rubrics = not published
    Holistic clinical-quality scoring uses 'task-specific rubric' criteria; the rubric contents are not included, so scoring behavior is not pinned down.
axioms (6)
  • domain assumption The 16-physician reference answers and atomic key points are clinically correct and complete.
    The whole benchmark treats expert-annotated references as ground truth; no inter-annotator agreement or independent adjudication metrics are reported.
  • domain assumption De-identified records from Zhongshan Hospital, Fudan University, are representative of cardiovascular care generally.
    Single-center data; the paper acknowledges institutional variation may limit generalizability.
  • domain assumption Key-point macro-recall and holistic clinical quality measure the intended competencies.
    The paper assumes these two components capture completeness and quality, but matching is mechanical and holistic quality scoring is not fully specified.
  • domain assumption Zero-shot, single-turn, deterministic decoding is a fair probe of clinical capability.
    No retrieval, tools, or conversation memory; real clinical use differs substantially.
  • domain assumption The named guidelines (GRACE, HAS-BLED, CAD-RADS, etc.) are correctly encoded in references.
    Reference answers were aligned to guidelines; transcription or interpretation errors could shift scores.
  • ad hoc to paper Holistic quality scorer exists and behaves consistently.
    Unstated evaluator; if an LLM judge was used, the evaluation loop is self-referential.

pith-pipeline@v1.3.0-alltime-deepseek · 13497 in / 10336 out tokens · 98224 ms · 2026-08-01T03:09:24.972649+00:00 · methodology

0 comments
read the original abstract

Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, multimodal, and safety-critical workflow of cardiovascular care. Objective: To develop MyoCardBench, a real-world benchmark spanning the cardiovascular care continuum, and assess LLM performance across clinical dimensions and specialist tasks. Methods: MyoCardBench includes 2,263 items from 13 task-specific datasets derived from de-identified cardiovascular records and examination data. Sixteen cardiology physicians conducted annotation and reference construction, followed by cross-review from two senior cardiologists. Seven LLMs generated 15,841 outputs under standardized zero-shot settings. Open-ended tasks were evaluated using key-point coverage and holistic clinical quality, while CardioEthics was scored by accuracy. Results: GPT-5.4 achieved the highest macro-average (62.55) and item-weighted mean (62.19), followed by Gemini 3.1 Pro (59.95) and Qwen 3.6 27B (59.72). GPT-5.4 ranked first in all three dimensions. CardioAuxReport performed best (86.38), whereas CardioECGRead (17.25) and CardioEthics (17.34) were lowest. The largest gaps between holistic clinical quality and key-point coverage occurred in CardioComm (52.71), CardioEmergRescue (52.05), and CardioTreatPlan (48.80). Conclusions: To our knowledge, MyoCardBench is the largest real-world, multi-task benchmark for LLM evaluation across the cardiovascular care continuum and offers the broadest coverage of clinically authentic cardiology scenarios reported to date. It provides a rigorous framework for identifying model strengths, clinically important omissions, and priorities for future development.

Figures

Figures reproduced from arXiv: 2607.25186 by Jie Xu, Jingru Ding, Juechen Chen, Junbo Ge, Lu Lu, Mouxiao Bian, Sijie Ren, Xiao Li, Yixiu Liang, Yun Zhong, Zhaodi Wu.

Figure 1
Figure 1. Figure 1: Overview of the MyoCardBench construction and evaluation framework [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dimension-specific and overall model performance [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task-level performance and between-LLM variability across MyoCardBench [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Divergence between holistic clinical quality and key-point coverage [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: LLM-specific strengths and performance stability [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 1 linked inside Pith

  1. [1]

    Global burden of cardiovascular diseases and risks, 1990-2022.Journal of the American College of Cardiology, 82(25):2350–2473, 2023

    George A Mensah, Valentin Fuster, Christopher JL Murray, Gregory A Roth, Yohannes Habtegiorgis Abate, Mohammadreza Abbasian, Foad Abd-Allah, Ashkan Abdollahi, Mohammad Abdollahi, Deldar Morad Abdulah, et al. Global burden of cardiovascular diseases and risks, 1990-2022.Journal of the American College of Cardiology, 82(25):2350–2473, 2023

    1990

  2. [2]

    2025 heart disease and stroke statistics: a report of us and global data from the american heart association.Circulation, 151(8):e41–e660, 2025

    Seth S Martin, Aaron W Aday, Norrina B Allen, Zaid I Almarzooq, Cheryl AM Anderson, Pankaj Arora, Christy L Avery, Carissa M Baker-Smith, Nisha Bansal, Andrea Z Beaton, et al. 2025 heart disease and stroke statistics: a report of us and global data from the american heart association.Circulation, 151(8):e41–e660, 2025

    2025

  3. [3]

    Large language models encode clinical knowledge.arXiv preprint arXiv:2212.13138, 2022

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge.arXiv preprint arXiv:2212.13138, 2022. 13

    Pith/arXiv arXiv 2022

  4. [4]

    Toward expert-level medical question answering with large language models.Nature medicine, 31(3):943–950, 2025

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models.Nature medicine, 31(3):943–950, 2025

    2025

  5. [5]

    Adapted large language models can outperform medical experts in clinical text summarization.Nature medicine, 30(4):1134–1142, 2024

    Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, et al. Adapted large language models can outperform medical experts in clinical text summarization.Nature medicine, 30(4):1134–1142, 2024

    2024

  6. [6]

    Towards accurate differential diagnosis with large language models.Nature, 642(8067):451–457, 2025

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models.Nature, 642(8067):451–457, 2025

    2025

  7. [7]

    Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450, 2025

    Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, et al. Towards conversational diagnostic artificial intelligence.Nature, 642(8067):442–450, 2025

    2025

  8. [8]

    Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature medicine, 30(9):2613–2622, 2024

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature medicine, 30(9):2613–2622, 2024

    2024

  9. [9]

    Testing and evaluation of health care applications of large language models: a systematic review.Jama, 333(4):319–328, 2025

    Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. Testing and evaluation of health care applications of large language models: a systematic review.Jama, 333(4):319–328, 2025

    2025

  10. [10]

    The evaluation illusion of large language models in medicine.npj Digital Medicine, 8(1):600, 2025

    Monica Agrawal, Irene Y Chen, Freya Gulamali, and Shalmali Joshi. The evaluation illusion of large language models in medicine.npj Digital Medicine, 8(1):600, 2025

    2025

  11. [11]

    Benchmark evaluation of deepseek large language models in clinical decision-making.Nature medicine, 31(8):2546–2549, 2025

    Sarah Sandmann, Stefan Hegselmann, Michael Fujarski, Lucas Bickmann, Benjamin Wild, Roland Eils, and Julian Varghese. Benchmark evaluation of deepseek large language models in clinical decision-making.Nature medicine, 31(8):2546–2549, 2025

    2025

  12. [12]

    Grounding large language models in clinical diagnostics.Nature Communications, 2026

    Xi Chen, Hanyu Zhou, Huahui Yi, Mingke You, WeiZhi Liu, Li Wang, Ziyuan Qin, Hairui Li, Xue Zhang, Yingman Guo, et al. Grounding large language models in clinical diagnostics.Nature Communications, 2026

    2026

  13. [13]

    Clinicrealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks.npj Digital Medicine, 2026

    Yinghao Zhu, Junyi Gao, Zixiang Wang, Weibin Liao, Xiaochen Zheng, Lifang Liang, Miguel O Bernabeu, Yasha Wang, Lequan Yu, Chengwei Pan, et al. Clinicrealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks.npj Digital Medicine, 2026

    2026

  14. [14]

    Evaluating clinical competencies of large language models with a general practice benchmark.Nature Communications, 2026

    Zheqing Li, Yiying Yang, Jiping Lang, Wenhao Jiang, Junrong Chen, Yuhang Zhao, Shuang Li, Dingqian Wang, Zhu Lin, Xuanna Li, et al. Evaluating clinical competencies of large language models with a general practice benchmark.Nature Communications, 2026

    2026

  15. [15]

    Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine.NPJ Digital Medicine, 7(1):190, 2024

    Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Marc J Landsman, et al. Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine.NPJ Digital Medicine, 7(1):190, 2024

    2024

  16. [16]

    Multimodal chatgpt-4v for electrocardiogram interpretation: promise and limitations.Journal of Medical Internet Research, 26:e54607, 2024

    Lingxuan Zhu, Weiming Mou, Keren Wu, Yancheng Lai, Anqi Lin, Tao Yang, Jian Zhang, and Peng Luo. Multimodal chatgpt-4v for electrocardiogram interpretation: promise and limitations.Journal of Medical Internet Research, 26:e54607, 2024

    2024

  17. [17]

    Effectiveness of the gpt-4o model in interpreting electrocardiogram images for cardiac diagnostics: Diagnostic accuracy study.Jmir ai, 4:e74426, 2025

    Haya Engelstein, Roni Ramon-Gonen, Avi Sabbag, Eyal Klang, Karin Sudri, Michal Cohen-Shelly, and Israel Barbash. Effectiveness of the gpt-4o model in interpreting electrocardiogram images for cardiac diagnostics: Diagnostic accuracy study.Jmir ai, 4:e74426, 2025

    2025

  18. [18]

    Performance of vision-enabled large language models in image-based electrocar- diogram interpretation: Exploratory evaluation.Journal of Medical Internet Research, 28:e86692, 2026

    Nibras Soubh, Eva Rasenack, Helge Haarmann, Felix Wiedmann, Markus Zabel, Constanze Schmidt, Rayan Suliman, and Leonard Bergau. Performance of vision-enabled large language models in image-based electrocar- diogram interpretation: Exploratory evaluation.Journal of Medical Internet Research, 28:e86692, 2026

    2026

  19. [19]

    Haemin Lee, Sooyoung Yoo, Joonghee Kim, Youngjin Cho, Dongbum Suh, and Keehyuck Lee. Comparative diagnostic performance of a multimodal large language model versus a dedicated electrocardiogram ai in detecting myocardial infarction from electrocardiogram images: Comparative study.JMIR AI, 4:e75910, 2025

    2025

  20. [20]

    Chatgpt takes on the european exam in core cardiology: an artificial intelligence success story?European Heart Journal-Digital Health, 4(3):279–281, 2023

    Ioannis Skalidis, Aurelien Cagnina, Wongsakorn Luangphiphat, Thabo Mahendiran, Olivier Muller, Emmanuel Abbe, and Stephane Fournier. Chatgpt takes on the european exam in core cardiology: an artificial intelligence success story?European Heart Journal-Digital Health, 4(3):279–281, 2023

    2023

  21. [21]

    The opportunities and challenges of large language models in cardiology.JACC: Advances, 2(7):100438, 2023

    Ashish Sarraju, David Ouyang, and Dipti Itchhaporia. The opportunities and challenges of large language models in cardiology.JACC: Advances, 2(7):100438, 2023

    2023

  22. [22]

    Artificial intelligence: revolutionizing cardiology with large language models.European Heart Journal, 45(5):332–345, 2024

    Machteld J Boonstra, Davy Weissenbacher, Jason H Moore, Graciela Gonzalez-Hernandez, and Folkert W Asselbergs. Artificial intelligence: revolutionizing cardiology with large language models.European Heart Journal, 45(5):332–345, 2024. 14

    2024

  23. [23]

    Performance of large language models on the acute coronary syndrome guidelines using retrieval-augmented generation.Cardiovascular Interventions, 18(20):2458–2467, 2025

    Michaella Alexandrou, Sant Kumar, Arun Umesh Mahtani, Dimitrios Strepkos, Pedro EP Carvalho, Deniz Mutlu, Ozgur Selim Ser, Athanasios Rempakos, Olga C Mastrodemos, Bavana V Rangan, et al. Performance of large language models on the acute coronary syndrome guidelines using retrieval-augmented generation.Cardiovascular Interventions, 18(20):2458–2467, 2025

    2025

  24. [24]

    A large language model for complex cardiology care.Nature Medicine, pages 1–8, 2026

    Jack W O’Sullivan, Anil Palepu, Khaled Saab, Wei-Hung Weng, Daniel K Amponsah, Evaline Cheng, Yong Cheng, Emily Chu, Yaanik Desai, Aly Elezaby, et al. A large language model for complex cardiology care.Nature Medicine, pages 1–8, 2026

    2026

  25. [25]

    Benchmarking large language models against clinicians across hospital levels in cardiovascular decision-making: a cross-sectional vignette-based study.Scientific Reports, 2025

    Zixi Zhang, Yingxu Ma, Yichao Xiao, Yongguo Dai, Qiuzhen Lin, Chan Liu, Cancan Wang, Tao Tu, and Qiming Liu. Benchmarking large language models against clinicians across hospital levels in cardiovascular decision-making: a cross-sectional vignette-based study.Scientific Reports, 2025

    2025

  26. [26]

    Evaluating clinical ai summaries with large language models as judges.npj Digital Medicine, 8(1):640, 2025

    Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dligach, et al. Evaluating clinical ai summaries with large language models as judges.npj Digital Medicine, 8(1):640, 2025

    2025

  27. [27]

    Automating expert-level medical reasoning evaluation of large language models

    Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, et al. Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 2025

    2025

  28. [28]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    2023

  29. [29]

    An evaluation framework for clinical use of large language models in patient interaction tasks.Nature medicine, 31(1):77–86, 2025

    Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, et al. An evaluation framework for clinical use of large language models in patient interaction tasks.Nature medicine, 31(1):77–86, 2025

    2025

  30. [30]

    A framework for human evaluation of large language models in healthcare derived from literature review.NPJ digital medicine, 7(1):258, 2024

    Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, et al. A framework for human evaluation of large language models in healthcare derived from literature review.NPJ digital medicine, 7(1):258, 2024

    2024

  31. [31]

    Predictors of hospital mortality in the global registry of acute coronary events.Archives of internal medicine, 163(19):2345–2353, 2003

    Christopher B Granger, Robert J Goldberg, Omar Dabbous, Karen S Pieper, Kim A Eagle, Christopher P Cannon, Frans Van de Werf, Alvaro Avezum, Shaun G Goodman, Marcus D Flather, et al. Predictors of hospital mortality in the global registry of acute coronary events.Archives of internal medicine, 163(19):2345–2353, 2003

    2003

  32. [32]

    A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry.Jama, 291(22):2727–2733, 2004

    Kim A Eagle, Michael J Lim, Omar H Dabbous, Karen S Pieper, Robert J Goldberg, Frans Van de Werf, Shaun G Goodman, Christopher B Granger, P Gabriel Steg, Joel M Gore, et al. A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry.Jama, 291(22):2727–2733, 2004

    2004

  33. [33]

    Gregory YH Lip, Robby Nieuwlaat, Ron Pisters, Deirdre A Lane, and Harry JGM Crijns. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation.Chest, 137(2):263–272, 2010

    2010

  34. [34]

    A novel user-friendly score (has-bled) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the euro heart survey.Chest, 138(5):1093–1100, 2010

    Ron Pisters, Deirdre A Lane, Robby Nieuwlaat, Cees B De V os, Harry JGM Crijns, and Gregory YH Lip. A novel user-friendly score (has-bled) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the euro heart survey.Chest, 138(5):1093–1100, 2010

    2010

  35. [35]

    Sunil V Rao, Michelle L O’Donoghue, Marc Ruel, Tanveer Rab, Jaqueline E Tamis-Holland, John H Alexander, Usman Baber, Heather Baker, Mauricio G Cohen, Mercedes Cruz-Ruiz, et al. 2025 acc/aha/acep/naemsp/scai guideline for the management of patients with acute coronary syndromes: a report of the american college of cardiology/american heart association joi...

    2025

  36. [36]

    Theresa A McDonagh, Marco Metra, Marianna Adamo, Roy S Gardner, Andreas Baumbach, Michael Böhm, Haran Burri, Javed Butler, Jelena ˇCelutkien˙e, Ovidiu Chioncel, et al. 2023 focused update of the 2021 esc guidelines for the diagnosis and treatment of acute and chronic heart failure: developed by the task force for the diagnosis and treatment of acute and c...

    2023

  37. [37]

    Isabelle C Van Gelder, Michiel Rienstra, Karina V Bunting, Ruben Casado-Arroyo, Valeria Caso, Harry JGM Crijns, Tom JR De Potter, Jeremy Dwight, Luigina Guasti, Thorsten Hanke, et al. 2024 esc guidelines for the management of atrial fibrillation developed in collaboration with the european association for cardio-thoracic surgery (eacts) developed by the t...

    2024

  38. [38]

    Ricardo C Cury, Jonathon Leipsic, Suhny Abbara, Stephan Achenbach, Daniel Berman, Marcio Bittencourt, Matthew Budoff, Kavitha Chinnaiyan, Andrew D Choi, Brian Ghoshhajra, et al. Cad-rads™ 2.0–2022 coronary artery disease–reporting and data system an expert consensus document of the society of cardiovascular computed tomography (scct), the american college...

    2022

  39. [39]

    Ivor J Benjamin, C Michael Valentine, William J Oetgen, Katherine A Sheehan, Ralph G Brindis, William H Roach, Robert A Harrington, Glenn N Levine, Rita F Redberg, Bernadette M Broccolo, et al. 2020 american heart association and american college of cardiology consensus conference on professionalism and ethics: a consensus conference report.Circulation, 1...

    2020

  40. [40]

    A novel evaluation benchmark for medical llms illuminating safety and effectiveness in clinical domains.npj Digital Medicine, 2025

    Shirui Wang, Zhihui Tang, Huaxia Yang, Qiuhong Gong, Tiantian Gu, Hongyang Ma, Yongxin Wang, Wubin Sun, Zeliang Lian, Kehang Mao, et al. A novel evaluation benchmark for medical llms illuminating safety and effectiveness in clinical domains.npj Digital Medicine, 2025

    2025

  41. [41]

    A framework to assess clinical safety and hallucination rates of llms for medical text summarisation

    Elham Asgari, Nina Montaña-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung, and Do- minic Pimenta. A framework to assess clinical safety and hallucination rates of llms for medical text summarisation. NPJ digital medicine, 8(1):274, 2025

    2025

  42. [42]

    Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative ai.Npj digital medicine, 7(1):82, 2024

    Mahyar Abbasian, Elahe Khatibi, Iman Azimi, David Oniani, Zahra Shakeri Hossein Abad, Alexander Thieme, Ram Sriram, Zhongqi Yang, Yanshan Wang, Bryant Lin, et al. Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative ai.Npj digital medicine, 7(1):82, 2024. 16

    2024