{"id":"251ee6ea-6de8-4ec1-b509-3990fb0d0895","arxiv_id":"2505.17167","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CRG Score is a prevalence-weighted clinical accuracy metric that ignores true negatives and assigns a score of one third to trivial outputs, designed for evaluating and training radiology report generators.","lead":"The paper proposes CRG Score, a metric for radiology report generation that weights missed and over-reported findings according to how rare they are. A smart generalist might care because it offers a way to grade AI-generated CT and X-ray reports by clinical relevance rather than word overlap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The clinical-alignment claim is asserted, not demonstrated: §2.1 fixes error costs by equating trivial extremes and imposing wTP=wFN, and no radiologist-preference or reward-learning validation is provided.","rationale":"The reader identified wTP=wFN in §2.1 as the weakest assumption. That is a reasonable place to look, and it is indeed arbitrary. However, deriving the simplified form CRG = 1/(3 - 2r + 2f) shows that varying wTP versus wFN does not change model ranking on a fixed test set; it only shifts the baseline score for trivial outputs. The real load-bearing condition for the paper's central claim is the untested equivalence between high CRG (equivalently high Youden's J) and clinically preferable reports. The paper calculates CRG on one binary-label dataset and does not validate against any external signal of clinical utility. Consequently, the verdict should remain CONDITIONAL as the reader decided, pending the proposed radiologist-preference study. I mark agreement as partial because the reader's specific assumption is symptomatic of the broader missing validation, but it is not itself the decisive weakness.","tokens_in":3743,"tokens_out":17218,"duration_ms":153947,"concrete_test":"Run a radiologist preference study on a held-out sample of CT-RATE validation reports: generate reports from several models, have radiologists independently rate clinical acceptability (especially missed critical findings and harmful false positives), then compute the Spearman correlation between per-model and per-report CRG scores and mean radiologist ratings. If CRG does not rank reports and models in line with the radiologists, the clinical-alignment claim is falsified. As a sensitivity check, repeat with a severity-weighted cost matrix; if rankings flip, the equal-weight-per-label assumption is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, CRG Score must rank reports and models the way clinicians would. The paper never tests this. The weighting in §2.1 is a normalization choice, not a measured clinical cost: equating empty and exhaustive reports sets one degree of freedom, and wTP=wFN closes it arbitrarily. After substitution the score collapses to CRG = 1/(3 - 2r + 2f), where r = TP/A and f = FP/N. This is a monotone transform of Youden's J (r + specificity - 1), so on a fixed test set the chosen wTP=wFN ratio does not even change model ranking; it only sets the trivial-score baseline. What actually carries the clinical-alignment claim is the implicit equation of clinical utility with r - f on the 18 CT-RATE labels, with every abnormality (a stable granuloma versus an acute pneumothorax) weighted equally. The paper offers no radiologist ratings, no severity weighting, no comparison to clinician-preferred rankings, and no RL experiment showing that optimizing CRG improves clinically meaningful report content. Section 2.3 only flags the absence of structured-label evaluation, not the missing clinical validation. Until such evidence exists, 'fairer' and 'clinically aligned' are unsupported assertions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the CRG Score, a metric for radiology report generation that is intended to be distribution-aware and clinically aligned by weighting true positives, false negatives, and false positives according to label prevalence. The authors derive the weights from an axiom equating empty and exhaustive reports plus an assumption that the reward for a true positive equals the penalty for a false negative, then normalize the raw score so that trivial solutions receive 1/3. They evaluate the metric on CT-RATE with four models and report CRG scores alongside conventional clinical accuracy metrics.","tokens_in":3984,"tokens_out":8767,"duration_ms":67375,"significance":"If the claims were substantiated, a distribution-aware, clinically aligned metric for 3D radiology report generation would address a real need, since existing NLG and LLM-based metrics are poorly suited to clinical correctness. The paper provides a clear mathematical derivation, releases code, and evaluates on a relevant dataset, which are commendable. However, as shown in the major comments, the metric reduces to a monotone transformation of Youden's J, so the claimed distribution-aware weighting has no effect on model ranking. Additionally, the clinical-alignment claim is entirely unvalidated. The paper's practical significance therefore hinges on whether the normalization and framing add value beyond existing metrics, which is not demonstrated.","major_comments":[{"comment":"Substituting the derived weights wTP = wFN = (T−A)/(2A) and wFP = 1 into the raw score and the normalization in Eq. (3) yields CRG = 1/(3 − 2r + 2f), where r = TP/A and f = FP/(T−A). This is a monotone increasing function of Youden's J = r + specificity − 1. Consequently, the prevalence-dependent weights cancel out completely, and CRG induces exactly the same model ranking as Youden's J on any fixed test set. The claim that CRG 'balances penalties based on label distribution' is therefore not supported by the metric as defined. The authors should either modify the metric so that the weighting affects the ranking, or explicitly acknowledge and discuss the equivalence to Youden's J and reframe the contribution accordingly.","section":"§2.1–2.2, Eqs. (1)–(3)"},{"comment":"The abstract and Section 2.3 state that CRG is 'clinically aligned' and 'serves as a clinically aligned reward function', but no clinical validation is provided. The paper reports no radiologist preference ratings, no severity weighting (e.g., a stable granuloma versus an acute pneumothorax), no comparison with clinician-preferred rankings, and no reinforcement-learning experiment showing that optimizing CRG improves clinically meaningful report content. The limitation paragraph in Section 2.3 only mentions the absence of structured-label evaluation; it does not address the missing clinical validation. The authors should either add clinical validation or substantially temper the claims.","section":"Abstract and §2.3"},{"comment":"The reported CRG scores have no error bars or statistical significance tests. The differences among CT2Rep, CT-CHAT, and Merlin (CRG 0.359, 0.368, and 0.352, respectively) may be within noise. The paper should include bootstrap confidence intervals or a significance test, and ideally evaluate on additional test sets to support the claim that CRG enables 'fairer, more robust evaluation'.","section":"Table 2"},{"comment":"The paper does not compare CRG against other imbalance-aware metrics such as Youden's J, balanced accuracy, Matthews correlation coefficient, or prevalence-weighted F-beta. Given the equivalence to Youden's J established in the first major comment, this comparison is essential to support the claim of 'fairer' evaluation. Without it, the reader cannot determine whether CRG offers any advantage over existing metrics beyond a different scaling.","section":"Sections 2.2–2.3"}],"minor_comments":[{"comment":"The text says 'CRG = 1/3 = 0.3'; since 1/3 is approximately 0.333, the paper should write '0.333' or 'approximately 0.3' for precision.","section":"§2.2"},{"comment":"The total number of labels T and the number of positive labels A are not reported in the text; adding these values would make the degree of class imbalance in the CT-RATE validation set explicit and help the reader interpret the metric.","section":"Table 2"},{"comment":"The sentence 'CRG can serve as a reward function...' is speculative; the paper should clarify that this is a proposal not yet tested in an RL training scenario.","section":"§2.3"},{"comment":"The abbreviation 'CA' is used for 'clinical accuracy' without being defined at first use in the Introduction; please define it.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short MIDL submission. The main issue is that the metric is mathematically equivalent to Youden's J, which is not acknowledged and which undercuts the central claim of being distribution-aware. If the authors can reframe the contribution as a normalized, interpretable version of Youden's J and add the necessary comparisons and clinical context, the paper might be acceptable as a short paper. As written, however, the claims of 'fairer' and 'clinically aligned' are unsupported. The editor may wish to consider whether the novelty threshold for acceptance is met."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this is a short MIDL paper proposing CRG Score, a class-imbalance-aware metric for radiology report generation. The formula is new and simple, and the implementation ships. But the headline claims — fairer, clinically aligned, reward-ready — are asserted, not demonstrated.\n\nThe good parts: CRG ignores true negatives, reweights penalties by label prevalence, and supports structured labels via a simple averaging scheme. The normalization trick that forces trivial reports to score 1/3 is neat and makes the metric easy to interpret. The code is public. As a definition, the math is correct and it is genuinely not a restatement of Green or standard accuracy.\n\nThe soft spots are real. The derivation in Section 2.1 equates empty and exhaustive reports, then assumes wTP = wFN with the justification that missing a finding and reporting one are equally costly. That is a value judgment, not a measured clinical cost. Worse, on a fixed test set this choice does not affect model ordering at all: after substitution CRG = 1/(3 - 2r + 2f), a monotone transform of r - f, i.e., Youden's J. So the actual ranking is equivalent to a decades-old imbalance-aware measure. The paper never compares against Youden's J, balanced accuracy, or Matthews correlation. The 'clinically aligned' claim rests on weighting all 18 CT-RATE abnormalities equally — a stable granuloma and an acute pneumothorax count the same — and on the unstated assumption that clinical utility is proportional to r - f. No radiologist preference data, no severity weighting, no RL experiment. Section 2.3 only flags the absence of structured-label evaluation, not the missing clinical validation. Minor: no error bars, though that is typical for a short venue.\n\nThis is not a takedown. The metric is clean, reimplementable, and honest about its scope. But as a claim of clinical alignment, the evidence is thin. The paper deserves peer review because the question it raises — what should a report-generation metric reward, and how should class imbalance be handled — matters, and the paper is a useful starting point. I would ask for a revision that adds comparisons to Youden's J and MCC, and either a small human-preference study or an explicit caveat that clinical alignment is a hypothesis, not a measured property.\n\nSend it to review, with the expectation of heavy revision.","headline":"A clean, transparent metric whose claimed clinical alignment is asserted rather than shown — and whose ranking reduces to Youden's J.","tokens_in":4506,"tokens_out":2493,"would_cite":false,"duration_ms":21353,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRG Score grades radiology reports by clinical findings, not word overlap, and can serve as a training reward for AI report generators.","keywords":["radiology report generation","clinical accuracy metric","class imbalance","distribution-aware evaluation","reinforcement learning reward","CT report evaluation","LLM-based feature extraction","report generation evaluation"],"falsifier":"Take a held-out set of CT reports with radiologist-assigned clinical-quality ratings, compute CRG for two systems—one that reports all major findings with many false positives, and one that omits a rare critical finding—and check whether CRG ranks them in the same order as the radiologists. If the metric prefers the system that misses the critical finding in a low-prevalence setting, the weight balance is wrong.","tokens_in":3552,"feed_emoji":"🩻","tokens_out":5005,"duration_ms":37212,"temperature":0.7,"pith_summary":"The paper proposes CRG Score, a metric for evaluating radiology report generation that focuses only on clinically relevant abnormalities named in the reference report. It argues that standard natural-language metrics miss clinical correctness and that existing clinical accuracy metrics are distorted by class imbalance—high accuracy often just reflects many true negatives, and recall can reward verbose over-reporting. CRG Score balances penalties according to how rare abnormalities are in the dataset, so missing a finding in a low-prevalence setting costs more than a false positive, and vice versa. The authors show the score ranks four CT report generators differently from standard metrics and yields a fixed value (1/3) for trivial always-normal or always-abnormal outputs. They also propose the score as a reward function for reinforcement learning, so report generators can be trained to maximize clinical correctness rather than token-level likelihood.","feed_headline":"CRG Score ranks AI radiology reports by clinical value, not word overlap","feed_subtitle":"The distribution-aware metric ignores trivial normal findings and can train generators to catch real abnormalities.","key_machinery":"The central object is the CRG score itself, a distribution-aware clinical accuracy metric. Its construction has three steps: it restricts evaluation to clinically meaningful outcomes (ignoring true negatives), it equates the scores of the two extreme outputs—an empty report and an exhaustive report—to derive the weight ratio $(w_{TP} + w_{FN})/w_{FP} = (T-A)/A$, and it closes the remaining degree of freedom by assuming $w_{TP} = w_{FN}$. The resulting normalized score, CRG $= S_{\\max}/(2S_{\\max} - s)$, assigns 1/3 to trivial predictions. What makes it work is that the penalty for a false negative grows as the abnormality becomes rarer in the dataset, which matches the clinical intuition that missing an uncommon finding is more dangerous than an occasional false alarm.","core_discovery":"The central claim is that CRG Score gives a fairer and more clinically truthful evaluation of generated radiology reports than existing metrics, and that the same score can serve as a training reward. The metric ignores true negatives, counts only abnormalities explicitly described in the reference report, and rebalances the weights of true positives, false negatives, and false positives by the label prevalence. It is defined so that an empty report and an exhaustive report receive the same raw score, which pins down the weight ratio; with the added assumption that the reward for a true positive equals the penalty for a false negative, the weights become $w_{TP} = w_{FN} = (T-A)/(2A)$ and $w_{FP} = 1$. The normalized score maps trivial solutions to 1/3 and higher values to better clinical performance. The paper demonstrates the metric on CT report generation with an 18-class abnormality labeler and reports that models with high lexical scores can be ranked differently once clinical balance is taken into account.","pith_inferences":["If CRG is adopted as an evaluation standard, model development will shift from mimicking reference style to optimizing detection of clinically relevant findings, which may change which architectures and training objectives win on leaderboards.","The metric's weight-balancing rule—making false negatives costlier when abnormalities are rare—is a sensible default but not the only clinically defensible one; a radiologist-grounded cost matrix could replace the $w_{TP} = w_{FN}$ assumption per use-case without changing the metric's structure.","The same distribution-aware balancing principle could transfer to other long-context generation tasks with skewed label distributions, such as pathology reports or discharge summaries, wherever true negatives are uninformative.","A direct test of clinical alignment would be comparing CRG rankings with radiologist preference judgments; if they diverge on a specific dataset, the weight ratio could be recalibrated rather than abandoning the metric."],"forward_implications":["CRG can replace or complement NLG metrics for ranking radiology report generators, since it evaluates clinically relevant content rather than lexical overlap.","In low-prevalence settings, CRG penalizes missed abnormalities more heavily than false positives; in high-prevalence settings it does the reverse, so the same metric adapts to different data distributions.","CRG can serve as a reward function in reinforcement learning, allowing report generators to be optimized for clinical correctness, possibly combined with BLEU or cross-entropy for fluency.","CRG supports structured labels, so it can be extended beyond binary abnormality presence to include type, location, laterality, severity, or count.","The metric fixes the trivial-solution problem: always-normal and always-abnormal reports both receive CRG = 1/3, so a generator cannot inflate its score by predicting the majority class."],"supporting_citations":[{"why":"Supplies the CT-RATE dataset and its 18-class abnormality labeler used for the evaluation and the example implementation.","marker":"Hamamci et al., 2024a"},{"why":"Provides Green, the fine-tuned LLM-based metric the paper argues generalizes poorly from X-ray to CT, motivating a model-agnostic alternative.","marker":"Ostmeier et al., 2024"},{"why":"RadFM is one of the CT report generation models evaluated in the comparison tables.","marker":"Wu et al., 2023"},{"why":"CT2Rep is one of the CT report generation models evaluated in the comparison tables.","marker":"Hamamci et al., 2024b"},{"why":"Merlin is one of the CT report generation models evaluated in the comparison tables.","marker":"Blankemeier et al., 2024"},{"why":"Describes the token-level cross-entropy training objective for biomedical LLM assistants that CRG is proposed to augment or replace as a reward.","marker":"Li et al., 2023"}],"fun_headline_variants":["CRG Score: Fair radiology AI that ignores trivial findings and catches real ones","Distribution-aware CRG Score rewards clinically relevant abnormalities only","CRG Score rebalances radiology metrics to rank real clinical value","New metric CRG Score punishes missed findings, not normal anatomy","CRG Score: a fairer reward for training AI to spot real abnormalities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's balance between missing a finding and correctly reporting one is fixed by assuming the reward for a true positive equals the penalty for a false negative ($w_{TP} = w_{FN}$); if real clinical costs differ, the score's ranking of models could be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["CRG Score: Fair radiology AI that ignores trivial findings and catches real ones","Distribution-aware CRG Score rewards clinically relevant abnormalities only","CRG Score rebalances radiology metrics to rank real clinical value","New metric CRG Score punishes missed findings, not normal anatomy","CRG Score: a fairer reward for training AI to spot real abnormalities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1571,"prompt_tokens":842,"completion_tokens":729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":458,"tokens_out":729,"duration_ms":5273,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:19.711985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of CT reports with radiologist-assigned clinical-quality ratings, compute CRG for two systems—one that reports all major findings with many false positives, and one that omits a rare critical finding—and check whether CRG ranks them in the same order as the radiologists. If the metric prefers the system that misses the critical finding in a low-prevalence setting, the weight balance is wrong.","supporting_citations":[],"review_version":1}