{"id":"2890c9cc-d487-46da-b6e6-345f29200e58","arxiv_id":"2508.02808","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ICARE uses two LLM agents that generate and answer multiple-choice questions from ground-truth and generated radiology reports, and their answer agreement serves as an interpretable metric for clinical fidelity.","lead":"This paper introduces ICARE, an automated metric for radiology report generation that uses two AI agents to generate and answer clinical multiple-choice questions, with answer agreement as the score. A clinician study suggests it matches expert preferences more closely than existing metrics, and it provides interpretable question-by-question scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed-loop LLM answers are unvalidated for clinical correctness; ICARE may measure self-consistency rather than clinical accuracy.","rationale":"The reader's conditional verdict and weakest-assumption analysis already point to the closed LLM loop, so I agree. I sharpen the concern: the missing validation is not question appropriateness but answer correctness. The framework's filtering criterion Pwith=1 and Pwithout=0 guarantees that the source agent's answer equals the designated answer by construction, leaving the other agent's answer as the only free variable; if that variable is driven by correlated LLM biases rather than clinical facts, the agreement scores lose their claimed meaning. This is the most load-bearing issue because the abstract's 'aligns significantly more with expert judgment' and the omission/hallucination claims both descend from these agreement scores. A clinician answer-annotation study, or a paraphrase-invariance test, would settle it. I keep the reader's conditional verdict: the paper is promising but should not be accepted until answer correctness is demonstrated or the claims are softened. The precision/recall terminology also appears swapped (ICARE-GT measures preservation of GT findings, i.e., recall), which should be corrected, but the primary validity risk is the unvalidated LLM answer signal.","tokens_in":19793,"tokens_out":12129,"duration_ms":130143,"concrete_test":"Select about 100 filtered questions (roughly 5 per clinical cluster, per question source and model) and have two or three board-certified radiologists answer each question using only the originating report, blinded to Llama's designated answer. Compute clinician-LLM agreement on the designated correct answer and on the answers each agent produced. If clinician-LLM agreement is below about 95% or inter-clinician disagreement exceeds about 10%, the answer signal is not clinically validated, and the Section 1.3 correlation analysis should be redone with clinician-verified answers. A complementary check: compute ICARE between a report and a synonym-paraphrased version; a large score drop would indicate surface-form sensitivity rather than semantic understanding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that ICARE 'aligns significantly more with expert judgment than prior metrics' and that its dual scores expose clinically meaningful omission/hallucination patterns. The load-bearing assumption is that the LLM-generated MCQA answers are clinically correct. The pipeline is closed-loop: Llama 3.1 70B creates the questions, designates the correct answers, filters them using its own with/without-report answering accuracy, and then answers every question from both reports; the score is the agreement between two runs of the same model. The Section 1.2 clinician study validated that questions are appropriate and answerable from a chest X-ray report (82% endorsement), but it did not validate the correctness of the designated answers or of Llama's answers from either report. Because the same model and prompt produce all answer sets, systematic errors (e.g., insensitivity to synonymous clinical phrasing, option-position bias, shared training priors) are correlated across agents and can inflate or deflate agreement independently of clinical content. The perturbation and stability analyses in Section 1.4 show sensitivity to word deletion and run-to-run reproducibility, not semantic correctness. Unless the LLM answers are externally validated, ICARE-GT/GEN may be measuring self-consistency of a single LLM rather than clinical precision/recall, which would undermine the headline correlation and the omission/hallucination conclusions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ICARE, an interpretable evaluation metric for radiology report generation that uses two LLM-based agents: one receives the ground-truth report and the other the generated report. Each agent generates multiple-choice questions from its own report, the questions are filtered so that they are answerable with the report but not without it, and then each agent answers both question sets. Agreement on ground-truth-derived questions (ICARE-GT) and generated-report-derived questions (ICARE-GEN) are presented as proxies for clinical precision and recall, with an overall score ICARE-AVG. The authors validate question quality with a clinician endorsement study, report a human-preference correlation study comparing ICARE to prior metrics, provide perturbation and stability analyses, and apply the metric to three report generation models to draw conclusions about omission versus hallucination patterns.","tokens_in":20014,"tokens_out":3601,"duration_ms":39677,"significance":"If substantiated, ICARE would be a valuable contribution to radiology report evaluation: it is automated, provides score-level interpretability by linking agreement to specific question-answer pairs, and its fine-grained GT/GEN decomposition could reveal clinically meaningful error patterns. The paper's strengths include the careful clinician endorsement study for question appropriateness, the explicit filtering for report-specificity, and the systematic perturbation and stability analyses. However, the central claim of superior alignment with expert judgment currently rests on a figure without reported correlation coefficients or significance tests, and the closed-loop use of a single LLM for question generation, filtering, and answering leaves the clinical correctness of answers unvalidated. These issues are addressable but are load-bearing for the headline conclusions.","major_comments":[{"comment":"The abstract and Section 1.3 claim that ICARE 'aligns significantly more with expert judgment than prior metrics,' but the paper reports no correlation coefficients, p-values, or confidence intervals for the comparisons in Fig. 2(c). The bar chart alone does not support the word 'significantly.' Please report exact Pearson (or ordinal) correlations for every metric, along with uncertainty estimates (e.g., bootstrap CIs) and a formal test comparing ICARE's correlation with each prior metric. In addition, the conversion of clinician 'Can't say' responses and metric ties to 0-labels should be justified, with a sensitivity analysis that excludes ties or treats them separately.","section":"§1.3, Fig. 2(c)"},{"comment":"The filtering criterion and the answer-agreement score both depend on Llama 3.1 70B's own outputs: the same model generates questions, designates correct answers, filters them using its own with/without-report accuracy, and then answers every question. The clinician study validated that questions are appropriate and answerable from chest X-ray reports, but it did not validate the clinical correctness of the designated answers or of the LLM's answers from either report. Without external validation, ICARE-GT and ICARE-GEN may partly measure the model's self-consistency rather than clinical precision and recall. Please provide an external check: for example, have clinicians answer a representative sample of filtered questions from both reports and measure agreement with the LLM's answers, or reproduce the answer-generation step with an independent LLM and show that ICARE scores and the reported model rankings are stable across answer models.","section":"§3.2–§3.4, Fig. 2(a)"},{"comment":"The conclusion that 'models are more likely to omit relevant clinical findings than to introduce unsupported content' rests on comparing ICARE-GT and ICARE-GEN, but these scores are computed on different question sets (questions generated from ground-truth reports versus from generated reports) that may differ systematically in difficulty, filtering outcomes, or topical composition. Lower agreement on GT-derived questions could reflect that such questions are harder for the shared LLM, not that the generated reports omit more findings. To support the omission versus hallucination claim, the two question sets should be shown to be comparable (e.g., matched on clinical cluster and on per-question answerability statistics), or a paired analysis should be performed on a common set of questions that both agents answer.","section":"§1.4, §3.4"}],"minor_comments":[{"comment":"The phrase 'transparent, and interpretable' contains an unnecessary comma before 'and'; it should read 'transparent and interpretable.'","section":"Abstract"},{"comment":"The caption references 'Section 2.5,' but the cluster analysis appears in Section 1.5; the cross-reference should be corrected.","section":"Extended Data Table A3 caption"},{"comment":"The figure lacks axis labels and a clear legend. Please label the y-axis (e.g., 'Correlation with clinician preference') and identify which metric each bar corresponds to.","section":"Fig. 2(c)"},{"comment":"The definitions of Pwith(Q_k, R) and Pwithout(Q_k) are informal; please clarify whether these are probabilities over repeated stochastic sampling, how many samples per question are used, and how 'accuracy' is computed for a single question.","section":"§3.2"},{"comment":"The stability reporting mixes units: 'standard deviations below 1% at the dataset level' is later followed by report-level standard deviations of 'at most 15' and 'greater than 20.' Please state whether these are percentage-point standard deviations or absolute agreement units, and keep the reporting consistent.","section":"§1.4 and Extended Data Fig. A3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and the framework is well-motivated. The main barrier to acceptance is the absence of statistical support for the headline correlation claim and the lack of external validation of LLM answer correctness. Both concerns are addressable with additional analyses; they do not appear to require a change in the core methodology. I would also encourage the authors to soften the abstract wording ('aligns significantly more') until the supporting statistics are actually reported, since the current phrasing overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Janet, quick take on arXiv:2508.02808. The paper proposes ICARE, an interpretable metric for radiology report generation using two LLM agents that generate and answer multiple-choice questions from ground-truth and generated reports. The core idea is genuinely useful, and the question-level error analysis is a real contribution. They don't fit any parameters, so the metric isn't circular in the derivation sense, and the clinician study (82% endorsement of question appropriateness) is a sensible validation step.\n\nWhat's actually new: the dual-agent MCQA framework with separate GT/GEN question sources and the filtering rule that keeps only questions the LLM can answer with a report but not without it. The cluster-level analysis, which separates omission-dominated from hallucination-dominated errors, is the most valuable part and could give model developers actionable feedback.\n\nThe soft spots are real but mostly addressable. The biggest is the headline claim: 'aligns significantly more with expert judgment' is supported only by a bar chart (Fig 2c) with no correlation coefficients, confidence intervals, or significance tests. For a metric paper, that's the load-bearing number, and it's missing. Second, the closed-loop design means the same LLM writes the questions, designates the correct answers, filters them, and answers them on both sides. The clinicians validated that questions are answerable from a report, but not that the designated answers are correct. So the score could partly measure the LLM's self-consistency rather than clinical accuracy. This needs external validation, e.g., a clinician-checked subset of answers or a different model for answering. Third, the precision/recall labels look swapped: agreement on ground-truth questions is recall (does the generated report cover findings?), and agreement on generated-report questions is precision (is generated content supported?). They have it reversed, and the omission/hallucination conclusions rely on that distinction. Fourth, report-level scores are based on 8-13 questions per report, so they're noisy; the stability numbers show a chunk of reports with standard deviation up to 15 points. Finally, no code or data is released, and the paper doesn't cite the QA-based evaluation line from summarization (e.g., QAGS). Minor, but worth a mention.\n\nOverall, the framework deserves serious referee time. The authors need to report the correlation statistics and validate answer correctness before the 'significantly more aligned' claim holds up. If they do that, this could become a standard tool in the RRG evaluation subfield. Send it to review, but expect revisions.","headline":"An interpretable RRG metric with a genuinely useful question-level error analysis, but the central correlation claim is underreported and the closed-loop LLM design needs external answer validation.","tokens_in":20591,"tokens_out":5189,"would_cite":true,"duration_ms":54838,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that having two AI agents quiz each other on multiple-choice questions about radiology reports yields a metric that aligns with clinician judgment more closely than prior metrics.","keywords":["radiology report generation","clinical evaluation metrics","agent-based evaluation","multiple-choice question answering","interpretable AI","chest X-ray","omission vs hallucination","LLM answer agreement"],"falsifier":"Take report pairs that a panel of radiologists unanimously judge to be clinically very different, run ICARE, and see whether a substantial share of pairs still receive near-identical scores; if so, exact answer agreement is not tracking clinical difference. Also ask clinicians to answer the same multiple-choice questions from the same reports and compare answer-by-answer with the two agents: near-chance clinician–agent agreement would show the metric measures model self-consistency rather than clinical fidelity.","tokens_in":19576,"feed_emoji":"🩻","tokens_out":6764,"duration_ms":67813,"temperature":0.7,"pith_summary":"ICARE is a proposed evaluation metric for automated radiology report generation. It works by giving the ground-truth report to one language-model agent and the generated report to another, having each agent write multiple-choice questions about the clinical content, and then having both agents answer both question sets using only their own report. Agreement on the ground-truth report's questions is read as preservation of findings (a precision proxy), while agreement on the generated report's questions is read as consistency of added content (a recall proxy). The paper reports that this agreement correlates with radiologist preferences more strongly than existing metrics such as BLEU, BERTScore, and GREEN, and that it exposes a consistent failure pattern across three report-generation models: findings are omitted more often than they are hallucinated. A metric that can be traced back to specific question–answer pairs would give developers a transparent way to see which clinical findings are preserved, dropped, or invented.","feed_headline":"AI agents quiz each other to judge radiology reports","feed_subtitle":"ICARE's question-answer agreement tracks expert preference better than BLEU, BERTScore, or other AI metrics.","key_machinery":"The load-bearing mechanism is a two-agent multiple-choice question-answering protocol. AgentGT and AgentGEN each generate a set of MCQs from their own report, filter out any question the same language model can answer without the report, then answer both filtered sets from their own report only. An external agreement step counts, per question, whether the two agents picked the same answer letter, producing dataset-level and report-level ICARE-GT and ICARE-GEN scores. The filtering step is what pins the questions to report-specific clinical content, and the dual-question-source design is what splits omission-type from hallucination-type errors.","core_discovery":"The central claim is that clinical similarity between two radiology reports can be measured by exact answer agreement of two report-aware agents on dynamically generated multiple-choice questions, and that this agreement tracks expert judgment better than surface-level or embedding-based metrics. The ground-truth agent and the generated-report agent each produce MCQs, keep only questions the same language model cannot answer without the report, and then answer both filtered question sets from their own report alone. Agreement on questions originating from the ground truth defines ICARE-GT, a proxy for clinical precision; agreement on questions originating from the generated report defines ICARE-GEN, a proxy for clinical recall; their average is ICARE-AVG. The paper reports that ICARE has the highest correlation with clinician preferences among the metrics tested, degrades predictably under word-level perturbation, stays stable across generation seeds, and reveals that all three evaluated report-generation models omit findings more often than they introduce unsupported content.","pith_inferences":["A direct test of whether ICARE measures clinical similarity or language-model self-consistency would be to have clinicians answer the same multiple-choice questions from the same reports and compare their answers with the agents' answers; near-chance clinician–agent agreement would suggest the metric tracks the model's own blind spots rather than clinical fidelity.","The filtering step equates 'cannot be answered by this particular language model without the report' with 'clinically report-dependent,' which could discard easy-but-valid questions or retain tricky ones that are actually answerable from medical priors.","The paper's labels invert standard information-retrieval usage: ICARE-GT is called precision even though it measures preservation of reference content, and ICARE-GEN is called recall even though it measures consistency of generated content, so cross-paper comparisons should check definitions rather than names.","Because the clinician study used pairwise preference comparisons rather than absolute quality ratings, the reported alignment shows the metric orders reports like radiologists do, but it does not yet establish a clinically meaningful absolute threshold for safe deployment."],"forward_implications":["Every ICARE score is traceable to specific question–answer pairs, so developers can inspect exactly which clinical findings, anatomical regions, or conditions a model preserves, omits, or alters.","The consistent pattern of ICARE-GT below ICARE-GEN across all three models implies that current report generators fail more by dropping ground-truth findings than by fabricating new ones.","Cluster-level scoring gives a per-condition report card: common findings such as pleural effusion and heart size score high, while rarer findings such as thoracic spine changes and calcified granulomas score low.","The authors state the framework can extend to other imaging modalities and clinical text tasks, and that it can monitor deployed models without ground-truth reports by tracking the answerability of curated questions over time."],"supporting_citations":[{"why":"Supplies the Llama 3.1 70B language model that generates, filters, and answers the multiple-choice questions, so the entire protocol runs on it.","marker":"[20]"},{"why":"Provides the IU X-ray dataset of chest radiographs and ground-truth reports on which all report-generation evaluations are performed.","marker":"[21]"},{"why":"Defines MAIRA-2, one of the three report-generation models evaluated, whose high ICARE scores anchor the model comparison.","marker":"[5]"},{"why":"Introduces RadCliQ and the evaluation benchmark that ICARE is compared against in the clinician-alignment study.","marker":"[15]"},{"why":"Provides GREEN, a GPT-based similarity metric recomputed as a prior-metric baseline in the correlation analysis.","marker":"[16]"},{"why":"Supplies BERTScore as a semantic embedding-based baseline metric in the clinician-alignment comparison.","marker":"[12]"},{"why":"Supplies the medical-domain embedding model used to cluster generated questions into the twenty semantic categories for cluster-level analysis.","marker":"[23]"},{"why":"Provides structured CheXpert labels used by the F1-CheXpert baseline in the metric comparison.","marker":"[13]"}],"fun_headline_variants":["Agent quiz duel yields interpretable radiology metric","AI agents' question agreement mirrors clinical judgment","ICARE: agents quiz each other to rate radiology reports","Radiology reports scored by AI agents' answer alignment","New metric: AI agents' Q&A consistency predicts expert preference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire score rests on treating exact agreement between two runs of the same language model on self-generated, self-filtered multiple-choice questions as a faithful signal that two reports are clinically equivalent; clinicians endorsed the questions' appropriateness but did not verify the correctness of the model's answers.","fun_headline_variants_meta":{"raw":{"variants":["Agent quiz duel yields interpretable radiology metric","AI agents' question agreement mirrors clinical judgment","ICARE: agents quiz each other to rate radiology reports","Radiology reports scored by AI agents' answer alignment","New metric: AI agents' Q&A consistency predicts expert preference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1519,"prompt_tokens":905,"completion_tokens":614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":521,"tokens_out":614,"duration_ms":6888,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:36:27.074817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take report pairs that a panel of radiologists unanimously judge to be clinically very different, run ICARE, and see whether a substantial share of pairs still receive near-identical scores; if so, exact answer agreement is not tracking clinical difference. Also ask clinicians to answer the same multiple-choice questions from the same reports and compare answer-by-answer with the two agents: near-chance clinician–agent agreement would show the metric measures model self-consistency rather than clinical fidelity.","supporting_citations":[{"cited_title":"Journal of the American Medical Informatics Association : JAMIA 23 2 , 304–10 (2015)","cited_arxiv_id":null,"evidence_quote":"Provides the IU X-ray dataset of chest radiographs and ground-truth reports on which all report-generation evaluations are performed."},{"cited_title":"Patterns 4 (2022)","cited_arxiv_id":null,"evidence_quote":"Introduces RadCliQ and the evaluation benchmark that ICARE is compared against in the clinician-alignment study."},{"cited_title":"Bioinformatics 39 11 (2023) 28 You are an expert in radiology and medical natural language processing","cited_arxiv_id":null,"evidence_quote":"Supplies the medical-domain embedding model used to cluster generated questions into the twenty semantic categories for cluster-level analysis."},{"cited_title":"In: AAAI Conference on Artificial Intelligence (2019)","cited_arxiv_id":null,"evidence_quote":"Provides structured CheXpert labels used by the F1-CheXpert baseline in the metric comparison."}],"review_version":2}