REVIEW 4 major objections 6 minor 6 cited by
ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read ReXrank is a public leaderboard for AI-powered chest X-ray report generation that ranks sixteen models on a new 10,000-study private test set and three public datasets, using eight evaluation metrics to compare findings-only and…
desk verdict Useful leaderboard with an unvalidated private test set; ship it to review but require reference validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the leaderboard itself, but the argument runs through ReXGradient and the primary metric. ReXGradient is a private test set of 10,000 studies from 67 U.S. sites, chosen so models cannot train on it; the primary metric, RadCliQ-v1, is a composite of BLEU, BERTScore, SembScore, and RadGraph-F1, reported as its reciprocal so higher is better. The framework also scores each report with BLEU-2, BERTScore, SembScore, RadGraph-F1, RaTEScore, GREEN, and FineRadScore, and it evaluates findings-only and findings-plus-impression generation in separate tracks. Confidence intervals are computed under a normal-distribution assumption using the standard error of the mean.
What would settle it
Take a random sample of 200 ReXGradient studies, have two radiologists score each reference report for completeness and clinical correctness against the images, and correlate those scores with the models' RadCliQ-v1 rankings on the same studies; if the correlation is near zero, the leaderboard is measuring textual similarity rather than clinical quality.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a single standardized evaluation can produce a defensible ordering of current chest X-ray report generators and, at the same time, characterize the test distributions themselves. It reports that MedVersa consistently outperforms the other models, including GPT-4V, across most datasets and metrics, while models trained on multiple datasets tend to generalize better than single-dataset models. It also finds that IU-Xray is too easy to separate models, CheXpert Plus is the hardest and noisiest distribution, and ReXGradient shows strikingly small confidence intervals, which the paper reads as a sign of high data quality and reliability as a benchmark.
Load-bearing premise
The ranking stands or falls on ReXGradient's reference reports being accurate and complete and on RadCliQ-v1 (and the other seven metrics) capturing clinically meaningful report quality, because the paper provides no radiologist audit or human-metric correlation for the new test set.
Editorial extensions
If this is right
- Any model can be submitted to ReXrank and compared against all existing entries on the same private and public test sets, making reported gains in radiology report generation checkable rather than self-claimed.
- MedVersa's top ranking establishes a concrete baseline that future chest X-ray report generators need to beat.
- The separate findings-only and full-report tracks let developers see whether apparent gains come from findings content or from impression text, and reveal architectural choices such as using dedicated models per section.
- The large spread and low scores on CheXpert Plus indicate that distribution shift remains a major barrier, so progress measured on MIMIC-CXR alone is not enough.
- The same dataset-plus-metrics infrastructure is designed to extend to other imaging modalities beyond chest X-rays.
Reading between the lines
- A radiologist audit of a random sample of ReXGradient reference reports would tell whether the small confidence intervals reflect genuinely consistent data quality or simply metric saturation; the paper itself reports no such audit.
- RadCliQ-v1 is a composite of lexical and entity-overlap metrics, so a model tuned to maximize it could rise in the ranking without improving clinical accuracy; holding out a hidden subset of ReXGradient would make such gaming harder.
- If ReXrank is extended to other modalities, the same logic would require a comparably large private multi-site test set with expert-verified reference reports, since public training sets would otherwise contaminate the benchmark.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ReXrank, a public leaderboard for chest X-ray radiology report generation. The authors combine a new private dataset, ReXGradient (10,000 studies, 7,004 patients, 67 U.S. sites), with three public datasets (MIMIC-CXR, IU-Xray, CheXpert Plus) and eight automated metrics (BLEU-2, BERTScore, SembScore, RadGraph-F1, RadCliQ-v1, RaTEScore, GREEN, FineRadScore), using RadCliQ-v1 as the primary metric. Sixteen models from ten institutions are evaluated, and the paper reports that MedVersa is the top-scoring model on the findings-only task on ReXGradient and MIMIC-CXR. The authors argue that the framework provides standardized, meaningful comparisons and offers robustness insights across clinical settings.
Significance. If the framework is reliable, ReXrank would be a useful community asset for benchmarking report generation models, and the inclusion of 16 models with multiple established metrics is a strength. The publicly accessible leaderboard and the use of official implementations for several metrics are also positive aspects. However, the paper's central claim depends on the trustworthiness of the private ReXGradient references, which are not validated, and on the transparency of the evaluation pipeline, which is incomplete. These gaps currently prevent the rankings from being interpreted as clinically meaningful, so the manuscript needs substantial revision before its main assertions can be accepted.
major comments (4)
- [Section 3.1 / Section 2] ReXGradient, the private 10,000-study dataset that underpins the leaderboard rankings in Tables 1-4, is described only by size and provenance (Section 3.1). The manuscript does not state how the reference reports were produced, whether they are original clinical reports, how studies were selected, or whether any radiologist verified their accuracy and completeness. Section 2's inference that ReXGradient has 'high data quality' from 'remarkably low-performance variance across models' is not logically valid: low variance across models could equally reflect metric insensitivity, case homogeneity, or a narrow pathology distribution. Because ReXGradient is private and no human audit or human-metric correlation is provided, the RadCliQ-v1 scores and the resulting model rankings are not yet interpretable. I request at least a radiologist audit on a random subset, inter-rater reliability statistics, or a correlation of automated metrics with expert judgments on ReXGradient.
- [Section 1 / Disclosures] The paper calls ReXrank a 'public leaderboard' but does not describe how to submit models or how the leaderboard is updated, and the Disclosures mention a '4x downsampled version' of ReXGradient without a URL or access procedure. The leaderboard URL in the abstract is a useful start, but without submission instructions, dataset access details, and a description of the evaluation pipeline used to produce Tables 1-4, the reported results cannot be reproduced or independently verified.
- [Section 3.4 / Table 4] The CheXpert Plus evaluation uses only 200 validation studies, and Table 4 shows wide, heavily overlapping confidence intervals on the primary metric: for example, in the findings-only section the top three models are 0.81±0.12, 0.79±0.10, and 0.79±0.10, and in the findings+impression section the top two models tie at 0.51±0.07. These intervals mean that the CheXpert Plus rankings, and any conclusions about robustness drawn from them, are not statistically supported. The text should either restrict claims to datasets with adequate power or explicitly mark CheXpert Plus results as exploratory only.
- [Section 3.3 / Tables 1-4] The LLM-based metrics GREEN and FineRadScore are computed with GPT-4o according to official implementations, but the manuscript does not report the exact model version, decoding parameters (temperature, top-p, max tokens), the number of runs, or how stochasticity was handled. Since these metrics contribute to the reported values in Tables 1-4 (e.g., GREEN and 1/FineRadScore with ±0.01 to ±0.02 intervals), the absence of these details makes the numbers non-reproducible and the reported confidence intervals uninterpretable.
minor comments (6)
- [Section 2 / Section 4 / Table 1] MedVersa's ReXGradient findings+impression 1/RadCliQ-v1 is reported as 0.98±0.05 in Sections 2 and 4, but Table 1 lists it as 0.98±0.01; please reconcile the discrepancy.
- [Table 4] The BERTScore values of -0.04±0.02 and -0.07±0.02 for RadFM and GPT4V in the findings+impression rows are surprisingly negative for BERTScore; please check the computation or clarify whether a different reference normalization was used.
- [Table 3] The metric column is labeled 'BLEU' in Table 3 but 'BLEU-2' in the other tables; the naming should be consistent.
- [Section 1] The phrase 'including ... etc.' after enumerating eight metrics is unnecessary and ambiguous, since all eight are already listed; please remove the 'etc.'
- [Figures 2 and 3] Figure 2's y-axis has no visible label (the caption says it is the average 1/RadCliQ-v1), and the model names in Figure 2 are too small to read; improving axis labels and font sizes would aid comprehension.
- [Section 3.4] The confidence intervals assume a normal distribution of the data, but several metrics are bounded or skewed (e.g., BERTScore and the reciprocal transforms); a brief justification or a bootstrap alternative would strengthen the statistical reporting.
Circularity Check
No circularity: the leaderboard rankings are computed evaluations, not derived predictions; self-cited metrics and models are not load-bearing.
full rationale
ReXrank makes no first-principles prediction or fitted derivation. Its central outputs are empirical scores: each model checkpoint (publicly released, from 10 institutions) is run on fixed reference reports (ReXGradient, MIMIC-CXR, IU-Xray, CheXpert Plus) and scored with eight pre-existing metrics using their official implementations. The primary ranking metric RadCliQ-v1 is taken from Yu et al. (2023), not defined here, and the paper does not fit any parameter to the leaderboard data; therefore the claim that MedVersa is top is a measurement, not a construction. Self-citations appear (FineRadScore, RaTEScore, RadFM, MedVersa, and RadFM's authors overlap with the present authors), but none is load-bearing: no uniqueness theorem, ansatz, or fitted input is imported from these works, and the top ranking would be unchanged by removing any one self-cited metric or model. The statement that low variance on ReXGradient 'underscores its high data quality' (Section 2) is an interpretive inference, and I flag the absence of radiologist validation of ReXGradient reference reports (Section 3.1 describes only size and provenance) and the disclosed Gradient Health affiliation as trustworthiness concerns rather than circularity. Because no prediction reduces by definition to an input, the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The eight automated metrics, especially RadCliQ-v1, are valid proxies for radiology report quality.
- domain assumption ReXGradient's 10,000 studies are representative of U.S. clinical practice and its reference reports are accurate and complete.
- domain assumption The official test splits and the CheXpert Plus validation set are fair test distributions.
- standard math Normal-distribution confidence intervals describe the uncertainty of the reported means.
Cite this review
Pith. "Pith review of ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation." pith.science (2026). https://pith.science/paper/X32JKEIW
@misc{pith2026241115122,
author = {Pith},
title = {Pith review of: ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/X32JKEIW}},
note = {Machine review of arXiv:2411.15122}
}
read the original abstract
AI-driven models have demonstrated significant potential in automating radiology report generation for chest X-rays. However, there is no standardized benchmark for objectively evaluating their performance. To address this, we present ReXrank, https://rexrank.ai, a public leaderboard and challenge for assessing AI-powered radiology report generation. Our framework incorporates ReXGradient, the largest test dataset consisting of 10,000 studies, and three public datasets (MIMIC-CXR, IU-Xray, CheXpert Plus) for report generation assessment. ReXrank employs 8 evaluation metrics and separately assesses models capable of generating only findings sections and those providing both findings and impressions sections. By providing this standardized evaluation framework, ReXrank enables meaningful comparisons of model performance and offers crucial insights into their robustness across diverse clinical settings. Beyond its current focus on chest X-rays, ReXrank's framework sets the stage for comprehensive evaluation of automated reporting across the full spectrum of medical imaging.
Forward citations
Cited by 6 Pith papers
-
CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
A chest X-ray VLM co-trained with classification and grounding heads, tuned with DAPO reinforcement learning, and augmented with deterministic measurement tools outperforms prior radiology VLMs on report generation, V...
-
Scaling medical imaging report generation with multimodal reinforcement learning
UniRG-CXR, a Qwen3-VL-8B model trained with SFT plus GRPO reinforcement learning that directly optimizes the ReXrank metric components, reports state-of-the-art 1/RadCliQ-v1 results on all four ReXrank chest X-ray dat...
-
Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Domain-adapted LLM encoders trained with masked token prediction and supervised contrastive learning improve chest X-ray image-text retrieval and external generalization, reaching GREEN scores of 0.308 on MIMIC-CXR an...
-
CorBenchX: Large-Scale Chest X-Ray Error Dataset and Vision-Language Model Benchmark for Report Error Correction
CorBenchX provides a large-scale synthetic error dataset and benchmark for chest X-ray report error detection and correction, plus a multi-step RL method that improves model performance.
-
ReXGradient-160K: A Large-Scale Publicly Available Dataset of Chest Radiographs with Free-text Reports
ReXGradient-160K provides 160,000 chest X-ray studies with free-text reports from 109,487 patients across 79 medical sites, with public and private test splits.
-
From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
A PRISMA-ScR scoping review of 144 studies finds the field shifting from text-only LLMs to multimodal AI in medicine, with evaluation and data diversity still the main bottlenecks.
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.