Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation

Daniel Alexander Alber; Daniel Freedman; Danielle Rigau; Diana Ruan; Eric Karl Oermann; Kang Zhang; Kyunghyun Cho; Motaz Nashawaty; Radhika Dua; Siddhant Dogra

REVIEW 3 major objections 5 minor 33 references

Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that having two AI agents quiz each other on multiple-choice questions about radiology reports yields a metric that aligns with clinician judgment more closely than prior metrics.

desk verdict An interpretable RRG metric with a genuinely useful question-level error analysis, but the central correlation claim is underreported and the closed-loop LLM design needs external answer validation. read the letter →

arxiv 2508.02808 v1 pith:YDFMTRVW submitted 2025-08-04 cs.CL cs.AIcs.LG

Radhika Dua , Young Joon (Fred) Kwon , Siddhant Dogra , Daniel Freedman , Diana Ruan , Motaz Nashawaty , Danielle Rigau , Daniel Alexander Alber

show 3 more authors

Kang Zhang Kyunghyun Cho Eric Karl Oermann

This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords radiologyreportgenerationclinicalevaluationmetricsagent-basedmultiple-choicequestionansweringinterpretableAIchestX-rayomissionvshallucinationLLMansweragreement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ICARE is a proposed evaluation metric for automated radiology report generation. It works by giving the ground-truth report to one language-model agent and the generated report to another, having each agent write multiple-choice questions about the clinical content, and then having both agents answer both question sets using only their own report. Agreement on the ground-truth report's questions is read as preservation of findings (a precision proxy), while agreement on the generated report's questions is read as consistency of added content (a recall proxy). The paper reports that this agreement correlates with radiologist preferences more strongly than existing metrics such as BLEU, BERTScore, and GREEN, and that it exposes a consistent failure pattern across three report-generation models: findings are omitted more often than they are hallucinated. A metric that can be traced back to specific question–answer pairs would give developers a transparent way to see which clinical findings are preserved, dropped, or invented.

What carries the argument

The load-bearing mechanism is a two-agent multiple-choice question-answering protocol. AgentGT and AgentGEN each generate a set of MCQs from their own report, filter out any question the same language model can answer without the report, then answer both filtered sets from their own report only. An external agreement step counts, per question, whether the two agents picked the same answer letter, producing dataset-level and report-level ICARE-GT and ICARE-GEN scores. The filtering step is what pins the questions to report-specific clinical content, and the dual-question-source design is what splits omission-type from hallucination-type errors.

What would settle it

Take report pairs that a panel of radiologists unanimously judge to be clinically very different, run ICARE, and see whether a substantial share of pairs still receive near-identical scores; if so, exact answer agreement is not tracking clinical difference. Also ask clinicians to answer the same multiple-choice questions from the same reports and compare answer-by-answer with the two agents: near-chance clinician–agent agreement would show the metric measures model self-consistency rather than clinical fidelity.

Watch

Extended reading notes

Core claim

The central claim is that clinical similarity between two radiology reports can be measured by exact answer agreement of two report-aware agents on dynamically generated multiple-choice questions, and that this agreement tracks expert judgment better than surface-level or embedding-based metrics. The ground-truth agent and the generated-report agent each produce MCQs, keep only questions the same language model cannot answer without the report, and then answer both filtered question sets from their own report alone. Agreement on questions originating from the ground truth defines ICARE-GT, a proxy for clinical precision; agreement on questions originating from the generated report defines ICARE-GEN, a proxy for clinical recall; their average is ICARE-AVG. The paper reports that ICARE has the highest correlation with clinician preferences among the metrics tested, degrades predictably under word-level perturbation, stays stable across generation seeds, and reveals that all three evaluated report-generation models omit findings more often than they introduce unsupported content.

Load-bearing premise

The entire score rests on treating exact agreement between two runs of the same language model on self-generated, self-filtered multiple-choice questions as a faithful signal that two reports are clinically equivalent; clinicians endorsed the questions' appropriateness but did not verify the correctness of the model's answers.

Editorial extensions

If this is right

Every ICARE score is traceable to specific question–answer pairs, so developers can inspect exactly which clinical findings, anatomical regions, or conditions a model preserves, omits, or alters.
The consistent pattern of ICARE-GT below ICARE-GEN across all three models implies that current report generators fail more by dropping ground-truth findings than by fabricating new ones.
Cluster-level scoring gives a per-condition report card: common findings such as pleural effusion and heart size score high, while rarer findings such as thoracic spine changes and calcified granulomas score low.
The authors state the framework can extend to other imaging modalities and clinical text tasks, and that it can monitor deployed models without ground-truth reports by tracking the answerability of curated questions over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of whether ICARE measures clinical similarity or language-model self-consistency would be to have clinicians answer the same multiple-choice questions from the same reports and compare their answers with the agents' answers; near-chance clinician–agent agreement would suggest the metric tracks the model's own blind spots rather than clinical fidelity.
The filtering step equates 'cannot be answered by this particular language model without the report' with 'clinically report-dependent,' which could discard easy-but-valid questions or retain tricky ones that are actually answerable from medical priors.
The paper's labels invert standard information-retrieval usage: ICARE-GT is called precision even though it measures preservation of reference content, and ICARE-GEN is called recall even though it measures consistency of generated content, so cross-paper comparisons should check definitions rather than names.
Because the clinician study used pairwise preference comparisons rather than absolute quality ratings, the reported alignment shows the metric orders reports like radiologists do, but it does not yet establish a clinically meaningful absolute threshold for safe deployment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Janet, quick take on arXiv:2508.02808. The paper proposes ICARE, an interpretable metric for radiology report generation using two LLM agents that generate and answer multiple-choice questions from ground-truth and generated reports. The core idea is genuinely useful, and the question-level error analysis is a real contribution. They don't fit any parameters, so the metric isn't circular in the derivation sense, and the clinician study (82% endorsement of question appropriateness) is a sensible validation step.

What's actually new: the dual-agent MCQA framework with separate GT/GEN question sources and the filtering rule that keeps only questions the LLM can answer with a report but not without it. The cluster-level analysis, which separates omission-dominated from hallucination-dominated errors, is the most valuable part and could give model developers actionable feedback.

The soft spots are real but mostly addressable. The biggest is the headline claim: 'aligns significantly more with expert judgment' is supported only by a bar chart (Fig 2c) with no correlation coefficients, confidence intervals, or significance tests. For a metric paper, that's the load-bearing number, and it's missing. Second, the closed-loop design means the same LLM writes the questions, designates the correct answers, filters them, and answers them on both sides. The clinicians validated that questions are answerable from a report, but not that the designated answers are correct. So the score could partly measure the LLM's self-consistency rather than clinical accuracy. This needs external validation, e.g., a clinician-checked subset of answers or a different model for answering. Third, the precision/recall labels look swapped: agreement on ground-truth questions is recall (does the generated report cover findings?), and agreement on generated-report questions is precision (is generated content supported?). They have it reversed, and the omission/hallucination conclusions rely on that distinction. Fourth, report-level scores are based on 8-13 questions per report, so they're noisy; the stability numbers show a chunk of reports with standard deviation up to 15 points. Finally, no code or data is released, and the paper doesn't cite the QA-based evaluation line from summarization (e.g., QAGS). Minor, but worth a mention.

Overall, the framework deserves serious referee time. The authors need to report the correlation statistics and validate answer correctness before the 'significantly more aligned' claim holds up. If they do that, this could become a standard tool in the RRG evaluation subfield. Send it to review, but expect revisions.

Referee Report

3 major / 5 minor

Summary. The paper introduces ICARE, an interpretable evaluation metric for radiology report generation that uses two LLM-based agents: one receives the ground-truth report and the other the generated report. Each agent generates multiple-choice questions from its own report, the questions are filtered so that they are answerable with the report but not without it, and then each agent answers both question sets. Agreement on ground-truth-derived questions (ICARE-GT) and generated-report-derived questions (ICARE-GEN) are presented as proxies for clinical precision and recall, with an overall score ICARE-AVG. The authors validate question quality with a clinician endorsement study, report a human-preference correlation study comparing ICARE to prior metrics, provide perturbation and stability analyses, and apply the metric to three report generation models to draw conclusions about omission versus hallucination patterns.

Significance. If substantiated, ICARE would be a valuable contribution to radiology report evaluation: it is automated, provides score-level interpretability by linking agreement to specific question-answer pairs, and its fine-grained GT/GEN decomposition could reveal clinically meaningful error patterns. The paper's strengths include the careful clinician endorsement study for question appropriateness, the explicit filtering for report-specificity, and the systematic perturbation and stability analyses. However, the central claim of superior alignment with expert judgment currently rests on a figure without reported correlation coefficients or significance tests, and the closed-loop use of a single LLM for question generation, filtering, and answering leaves the clinical correctness of answers unvalidated. These issues are addressable but are load-bearing for the headline conclusions.

major comments (3)

[§1.3, Fig. 2(c)] The abstract and Section 1.3 claim that ICARE 'aligns significantly more with expert judgment than prior metrics,' but the paper reports no correlation coefficients, p-values, or confidence intervals for the comparisons in Fig. 2(c). The bar chart alone does not support the word 'significantly.' Please report exact Pearson (or ordinal) correlations for every metric, along with uncertainty estimates (e.g., bootstrap CIs) and a formal test comparing ICARE's correlation with each prior metric. In addition, the conversion of clinician 'Can't say' responses and metric ties to 0-labels should be justified, with a sensitivity analysis that excludes ties or treats them separately.
[§3.2–§3.4, Fig. 2(a)] The filtering criterion and the answer-agreement score both depend on Llama 3.1 70B's own outputs: the same model generates questions, designates correct answers, filters them using its own with/without-report accuracy, and then answers every question. The clinician study validated that questions are appropriate and answerable from chest X-ray reports, but it did not validate the clinical correctness of the designated answers or of the LLM's answers from either report. Without external validation, ICARE-GT and ICARE-GEN may partly measure the model's self-consistency rather than clinical precision and recall. Please provide an external check: for example, have clinicians answer a representative sample of filtered questions from both reports and measure agreement with the LLM's answers, or reproduce the answer-generation step with an independent LLM and show that ICARE scores and the reported model rankings are stable across answer models.
[§1.4, §3.4] The conclusion that 'models are more likely to omit relevant clinical findings than to introduce unsupported content' rests on comparing ICARE-GT and ICARE-GEN, but these scores are computed on different question sets (questions generated from ground-truth reports versus from generated reports) that may differ systematically in difficulty, filtering outcomes, or topical composition. Lower agreement on GT-derived questions could reflect that such questions are harder for the shared LLM, not that the generated reports omit more findings. To support the omission versus hallucination claim, the two question sets should be shown to be comparable (e.g., matched on clinical cluster and on per-question answerability statistics), or a paired analysis should be performed on a common set of questions that both agents answer.

minor comments (5)

[Abstract] The phrase 'transparent, and interpretable' contains an unnecessary comma before 'and'; it should read 'transparent and interpretable.'
[Extended Data Table A3 caption] The caption references 'Section 2.5,' but the cluster analysis appears in Section 1.5; the cross-reference should be corrected.
[Fig. 2(c)] The figure lacks axis labels and a clear legend. Please label the y-axis (e.g., 'Correlation with clinician preference') and identify which metric each bar corresponds to.
[§3.2] The definitions of Pwith(Q_k, R) and Pwithout(Q_k) are informal; please clarify whether these are probabilities over repeated stochastic sampling, how many samples per question are used, and how 'accuracy' is computed for a single question.
[§1.4 and Extended Data Fig. A3] The stability reporting mixes units: 'standard deviations below 1% at the dataset level' is later followed by report-level standard deviations of 'at most 15' and 'greater than 20.' Please state whether these are percentage-point standard deviations or absolute agreement units, and keep the reporting consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ICARE is a defined agreement metric whose central alignment claim is tested against independent clinician judgments, not derived from its own inputs.

full rationale

ICARE is not a fitted model or a first-principles derivation; it is an explicitly defined evaluation procedure. The pipeline is: each agent generates MCQs from its assigned report (Section 3.2), the same LLM filters questions by answering with and without the report, each agent answers both question sets, and the score is the indicator-function agreement between the two answer sets (Section 3.4). No parameter is fit to clinician labels, and the precision/recall interpretation is an interpretive label attached to the two agreement scores, not a quantity recovered from those labels. The central claim, that ICARE aligns with expert judgment, is supported by an external human study in which six clinicians compared report pairs and the metric's preferences were correlated with theirs (Section 1.3, Fig. 2c). That is independent evidence rather than a circular restatement of the metric's definition. The perturbation and stability analyses are not predictions derived from the metric; they are empirical characterizations of its behavior. The paper does not rely on a self-citation chain or an imported uniqueness theorem. The main validity concern is that Llama 3.1 70B both creates the questions, designates the correct answers, filters them using its own with/without-report accuracy, and then answers every question from both reports, so the agreement score partly measures the model's self-consistency; moreover, the clinician study in Section 1.2 endorsed question appropriateness and answerability (82 percent) but did not verify the correctness of the LLM's designated answers. This is a substantive limitation for clinical validity, but it is not a derivation-level circularity: the metric is not equivalent to its inputs by construction, and the headline correlation claim is tested against independent clinician preferences. Under the stated rubric, the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The metric does not introduce new physical entities or fitted constants, but it relies on several domain assumptions about the gold standard and the validity of LLM-based QA agreement as a clinical proxy. The hand-chosen filtering thresholds and K-means K are free parameters.

free parameters (3)

Number of MCQs generated per report (n)
The paper says 'n MCQs' are generated per report but does not state the value; this affects the number of agreement measurements per report.
Number of semantic clusters (K) = 20
K-means clustering used K=20 for question categorization; this is a hand-chosen parameter.
Filtering thresholds = Pwith=1, Pwithout=0
Questions are retained only if the LLM answers correctly with the report and incorrectly without; this binary threshold is hand-chosen and not justified.

assumptions (4)

domain assumption Ground-truth reports are the gold standard and contain all clinically relevant information for the chest X-ray.
The entire evaluation treats the clinician-written report as the reference; if the ground truth is incomplete, the metric may penalize correct extra findings.
domain assumption Agreement on MCQA answers between two LLM agents is a valid proxy for clinical similarity.
The central metric assumes that if two reports yield the same answers to self-generated questions, they are clinically equivalent; this is not independently proven.
ad hoc to paper Questions filtered by Pwith=1, Pwithout=0 are clinically meaningful and report-specific.
This criterion relies on the LLM's own knowledge to define what is report-specific, which may be biased.
domain assumption The LLM (Llama 3.1 70B) answers questions accurately based on reports.
No clinician validation of answer accuracy is provided; endorsement is only for question appropriateness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation." pith.science (2026). https://pith.science/paper/YDFMTRVW

@misc{pith2026250802808,
  author       = {Pith},
  title        = {Pith review of: Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDFMTRVW}},
  note         = {Machine review of arXiv:2508.02808}
}

read the original abstract

Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 24 canonical work pages

[1]

A1 : Clinical relevance of generated questions across semantic clusters

Tanno, R., Barrett, D.G.T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Singhal, K., Azizi, S., Tu, T., Schaekermann, M., May, R., Lee, R., Man, S., Ahmed, Z., Mahdavi, S.S., Belgrave, D., Natarajan, V., Shetty, S., Kohli, P., Huang, P.-S., Karthikesalingam, A., Ktena, I.: Consensus, dissensus and 20 0 20 40 60 80 100 Percentage of Res...

arXiv 2023
[2]

PLoS ONE 17 (2022)

Sirshar, M., Paracha, M.F.K., Akram, M.U., Alghamdi, N.S., Zaidi, S.Z.Y., Fatima, T.: Attention based automated radiology report generation using cnn and lstm. PLoS ONE 17 (2022)

work page 2022
[3]

In: Annual Meeting of the Association for Computational Linguistics (2023)

Kale, K., Bhattacharyya, P., Jadhav, K.S.: Replace and report: Nlp assisted radi- ology report generation. In: Annual Meeting of the Association for Computational Linguistics (2023). https://api.semanticscholar.org/CorpusID:259309063

work page 2023
[4]

ArXiv abs/2311.13668 (2023) 21 CheXpertPlus MIMIC CheXpertPlus CheX MIMIC MAIRA-2 Dataset-level Report-level Extended Data Fig

Hyland, S.L., Bannur, S., Bouzid, K., Castro, D.C., Ranjit, M.P., Schwaighofer, A., P´ erez-Garc´ ıa, F., Salvatelli, V., Srivastav, S., Thieme, A., Codella, N., Lun- gren, M.P., Wetscherek, M.T., Oktay, O., Alvarez-Valle, J.: Maira-1: A specialised large multimodal model for radiology report generation. ArXiv abs/2311.13668 (2023) 21 CheXpertPlus MIMIC C...

arXiv 2023
[5]

ArXiv abs/2406.04449 (2024)

Bannur, S., Bouzid, K., Castro, D.C., Schwaighofer, A., Bond-Taylor, S., Ilse, M., P’erez-Garc’ia, F., Salvatelli, V., Sharma, H., Meissen, F., Ranjit, M.P., Srivastav, S., Gong, J., Falck, F., Oktay, O., Thieme, A., Lungren, M.P., Wetscherek, M.T., Alvarez-Valle, J., Hyland, S.L.: Maira-2: Grounded radiology report generation. ArXiv abs/2406.04449 (2024)

arXiv 2024
[6]

Nature communications 16 1 , 3108 (2024) 22 ICARE-GT ICARE-GEN Extended Data Fig

Chaves, J.M.Z., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H.H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M.-H., Naumann, T., Chen, M., Lungren, M.P., Yeung-Levy, S., Langlotz, C.P., Wang, S., Poon, H.: A clinically accessible small multimodal radiology model and eva...

work page 2024
[7]

ArXiv abs/2311.18681 (2023)

Pellegrini, C., ¨Ozsoy, E., Busam, B., Navab, N., Keicher, M.: Radialog: A large vision-language model for radiology report generation and conversational assistance. ArXiv abs/2311.18681 (2023)

arXiv 2023
[8]

Nature 620, 172–180 (2022)

Singhal, K., Azizi, S., Tu, T., Mahdavi, S., Wei, J., Chung, H.W., Scales, N., Tanwani, A.K., Cole-Lewis, H.J., Pfohl, S.J., Payne, P.A., Seneviratne, M.G., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P.A., Arcas, B.A., Webster, D.R., Corrado, G.S., Matias, Y., Chou, K.H.-L., Gottweis, J., Tomaˇ sev, N., Liu, Y., Rajkomar, A., Barral, J....

work page 2022

Show all 33 references

[9]

Zhou, H.-Y., Acosta, J.N., Adithan, S., Datta, S., Topol, E.J., Rajpurkar, P.: Medversa: A generalist foundation model for medical image interpretation. (2024). https://api.semanticscholar.org/CorpusID:269756808 23 ID Questions from Ground-T ruth Report Questions from Generate...

2024
[10]

In: Annual Meeting of the Association for Com- putational Linguistics (2002)

Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Annual Meeting of the Association for Com- putational Linguistics (2002). https://api.semanticscholar.org/CorpusID:11080756

2002
[11]

In: Annual Meeting of the Association for Computational Linguistics (2004)

Lin, C.-Y.: Rouge: A package for automatic evaluation of summaries. In: Annual Meeting of the Association for Computational Linguistics (2004). https://api.semanticscholar.org/CorpusID:964287

2004
[12]

ArXiv abs/1904.09675 (2019)

Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluat- ing text generation with bert. ArXiv abs/1904.09675 (2019)

2019 arXiv
[13]

In: AAAI Conference on Artificial Intelligence (2019)

Irvin, J.A., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R.L., Shpanskaya, K.S., Seekins, J., Mong, D.A., Halabi, S.S., Sandberg, J.K., Jones, R., Larson, D.B., Langlotz, C., Patel, B.N., Lungren, M.P., Ng, A.: Chexpert: A large...

2019
[14]

"Please use the following format exactly as your life depends on sticking to these formats.:\n\n

Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., Lungren, M.P.: Chexbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. In: Conference on Empirical Methods in Natural Language Processing (2020). https://api.semanti...

2020
[15]

Patterns 4 (2022)

Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E.P., Fonseca, E.K.U.N., Lee, H.H., Abad, Z.H., Ng, A.Y., Langlotz, C., Venugopal, V.K., Rajpurkar, P.: Evaluating progress in automatic chest x-ray radiology report generation. Patterns 4 (2022)

2022
[16]

ArXiv abs/2405.03595 (2024)

Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., Michalson, A.E., Moseley, M.E., Langlotz, C.P., Chaudhari, A.S., Delbrouck, J.-B.: Green: Generative radiology report evaluation and error notation. ArXiv abs/2405.03595 (2024)

2024 arXiv
[17]

ArXiv abs/2405.20613 (2024)

Huang, A., Banerjee, O., Wu, K., Reis, E.P., Rajpurkar, P.: Fineradscore: A radi- ology report line-by-line evaluation technique generating corrections with severity scores. ArXiv abs/2405.20613 (2024)

2024 arXiv
[18]

In: Conference on Empirical Methods in Natural Language Processing (2024)

Zhao, W., Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Ratescore: A metric for radiology report generation. In: Conference on Empirical Methods in Natural Language Processing (2024). https://api.semanticscholar.org/CorpusID:270699665

2024
[19]

ArXiv abs/2403.08002 (2024)

Chaves, J.M.Z., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H.H., Gong, J., Hu, H., Yang, J., Li, C.-y., Gao, J., Gu, Y., Wong, C., Wei, M.-H., Naumann, T., Chen, M., Lungren, M.P., Yeung-Levy, S., Langlotz, C.P., W...

2024 arXiv
[20]

Each question corresponds to one of the 20 semantic clusters identified in Section 2.5

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A.S., Yang, A., Mitra, 25 Questions from MCQ Dataset Generation Module Cluster Name Are the lungs clear of any conditions according to th...

2024 arXiv
[21]

Journal of the American Medical Informatics Association : JAMIA 23 2 , 304–10 (2015)

Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L.M., Antani, S.K., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association : JAMIA 23 2 , ...

2015
[22]

https://api.semanticscholar.org/CorpusID:270123479

Chexpert plus: Hundreds of thousands of aligned radiology texts, images and patients. https://api.semanticscholar.org/CorpusID:270123479
[23]

Bioinformatics 39 11 (2023) 28 You are an expert in radiology and medical natural language processing

Jin, Q., Kim, W., Chen, Q., Comeau, D.C., Yeganova, L., Wilbur, J., Lu, Z.: Biocpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics 39 11 (2023) 28 You are an expert in radiology and medica...

2023
[24]

Are the lungs clear of any conditions according to the report?
[25]

What does the report imply about the overall condition of the lungs?
[26]

What is indicated about the condition of both lungs in the report?
[27]

What does the report indicate about the condition of both lungs?
[28]

What is indicated about the condition of the lungs in the report? Questions from cluster 19:
[29]

Is there any mention of abnormalities in the heart according to the report?
[30]

Is there any mention of heart abnormality in the report?
[31]

Is there any mention of the heart being abnormal in the report?
[32]

Is the heart relevant to the diagnosis mentioned in the report?
[33]

A5 : Prompts given to the language model for the Question Categorization and Analysis

Are there any abnormalities in the heart according to the report? Prompt for Question Categorization and Analysis … Extended Data Fig. A5 : Prompts given to the language model for the Question Categorization and Analysis. 29

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

A1 : Clinical relevance of generated questions across semantic clusters

Tanno, R., Barrett, D.G.T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Singhal, K., Azizi, S., Tu, T., Schaekermann, M., May, R., Lee, R., Man, S., Ahmed, Z., Mahdavi, S.S., Belgrave, D., Natarajan, V., Shetty, S., Kohli, P., Huang, P.-S., Karthikesalingam, A., Ktena, I.: Consensus, dissensus and 20 0 20 40 60 80 100 Percentage of Res...

arXiv 2023

[2] [2]

PLoS ONE 17 (2022)

Sirshar, M., Paracha, M.F.K., Akram, M.U., Alghamdi, N.S., Zaidi, S.Z.Y., Fatima, T.: Attention based automated radiology report generation using cnn and lstm. PLoS ONE 17 (2022)

work page 2022

[3] [3]

In: Annual Meeting of the Association for Computational Linguistics (2023)

Kale, K., Bhattacharyya, P., Jadhav, K.S.: Replace and report: Nlp assisted radi- ology report generation. In: Annual Meeting of the Association for Computational Linguistics (2023). https://api.semanticscholar.org/CorpusID:259309063

work page 2023

[4] [4]

ArXiv abs/2311.13668 (2023) 21 CheXpertPlus MIMIC CheXpertPlus CheX MIMIC MAIRA-2 Dataset-level Report-level Extended Data Fig

Hyland, S.L., Bannur, S., Bouzid, K., Castro, D.C., Ranjit, M.P., Schwaighofer, A., P´ erez-Garc´ ıa, F., Salvatelli, V., Srivastav, S., Thieme, A., Codella, N., Lun- gren, M.P., Wetscherek, M.T., Oktay, O., Alvarez-Valle, J.: Maira-1: A specialised large multimodal model for radiology report generation. ArXiv abs/2311.13668 (2023) 21 CheXpertPlus MIMIC C...

arXiv 2023

[5] [5]

ArXiv abs/2406.04449 (2024)

Bannur, S., Bouzid, K., Castro, D.C., Schwaighofer, A., Bond-Taylor, S., Ilse, M., P’erez-Garc’ia, F., Salvatelli, V., Sharma, H., Meissen, F., Ranjit, M.P., Srivastav, S., Gong, J., Falck, F., Oktay, O., Thieme, A., Lungren, M.P., Wetscherek, M.T., Alvarez-Valle, J., Hyland, S.L.: Maira-2: Grounded radiology report generation. ArXiv abs/2406.04449 (2024)

arXiv 2024

[6] [6]

Nature communications 16 1 , 3108 (2024) 22 ICARE-GT ICARE-GEN Extended Data Fig

Chaves, J.M.Z., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., Awadalla, H.H., Gong, J., Hu, H., Yang, J., Li, C., Gao, J., Gu, Y., Wong, C., Wei, M.-H., Naumann, T., Chen, M., Lungren, M.P., Yeung-Levy, S., Langlotz, C.P., Wang, S., Poon, H.: A clinically accessible small multimodal radiology model and eva...

work page 2024

[7] [7]

ArXiv abs/2311.18681 (2023)

Pellegrini, C., ¨Ozsoy, E., Busam, B., Navab, N., Keicher, M.: Radialog: A large vision-language model for radiology report generation and conversational assistance. ArXiv abs/2311.18681 (2023)

arXiv 2023

[8] [8]

Nature 620, 172–180 (2022)

Singhal, K., Azizi, S., Tu, T., Mahdavi, S., Wei, J., Chung, H.W., Scales, N., Tanwani, A.K., Cole-Lewis, H.J., Pfohl, S.J., Payne, P.A., Seneviratne, M.G., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P.A., Arcas, B.A., Webster, D.R., Corrado, G.S., Matias, Y., Chou, K.H.-L., Gottweis, J., Tomaˇ sev, N., Liu, Y., Rajkomar, A., Barral, J....

work page 2022