{"id":"fa73d17e-84d8-44c8-a0f8-9ac15b18949b","arxiv_id":"2411.15122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ReXrank is a standardized public leaderboard for AI chest X-ray report generation, built on a 10,000-study private test set and 8 metrics, and it ranks 16 models with MedVersa as the current top performer.","lead":"This paper presents ReXrank, a public leaderboard that evaluates 16 AI models that generate radiology reports from chest X-rays, using a new private test set of 10,000 studies and 8 automated metrics. It offers the field a shared yardstick for comparing such models, with the MedVersa system at the top of most current rankings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReXGradient reference reports have no documented radiologist validation; the leaderboard's rankings rest on an untested assumption that these private references are accurate and complete.","rationale":"The reader's weakest assumption is that ReXGradient contains accurate, complete reference reports and that the automated metrics, above all RadCliQ-v1, reflect clinically meaningful quality. My read converges on the same point: the paper's unique contribution is the private ReXGradient set, and the entire leaderboard ranking is only as trustworthy as those references and the metric's validity on them. The manuscript offers no radiologist audit, inter-rater reliability, or human-metric correlation for this set. The inference from low performance variance to high data quality is a logical leap, since variance across models is not a direct measurement of reference quality. This gap does not invalidate the benchmark outright; the framework is well structured, uses established metrics, and the model evaluation appears internally consistent. However, it is a condition that must be met before ReXrank can be treated as a community standard. The reader already issued a CONDITIONAL verdict, and my concern is the same load-bearing assumption, so no verdict adjustment is needed. I recommend requiring the proposed validation study, or a clear data-access and audit pathway, as a condition of acceptance.","tokens_in":16085,"tokens_out":4145,"duration_ms":40984,"concrete_test":"Select a random sample of 300 ReXGradient studies. Have two board-certified radiologists independently audit and, where needed, re-annotate the reference reports for completeness, accuracy, and section structure. Compute inter-rater agreement and then recompute RadCliQ-v1 for all 16 models on this sample using the original references versus the re-annotated references. Additionally, have radiologists rate 100 paired model outputs for clinical quality and measure the correlation between RadCliQ-v1 and expert quality scores. If the top-3 model ranking changes under re-annotated references, or if RadCliQ-v1 correlates with expert ratings below a pre-specified threshold (e.g., 0.7), the leaderboard's primary ranking is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ReXrank enables meaningful comparisons and robustness insights depends on ReXGradient (10,000 studies, 67 sites) being a trustworthy ground truth. Section 3.1 describes only size and provenance; it does not state how reference reports were produced, whether they are original clinical reports, how studies were selected, or whether any radiologist verified completeness and accuracy. Section 2 then infers 'high data quality' from 'remarkably low-performance variance across models.' Low variance across models is a property of the metric and model distribution, not direct evidence of reference quality; it could equally reflect case homogeneity, limited pathology diversity, or metric insensitivity. Because ReXGradient is private and only a 4x downsampled version is mentioned with no URL or access procedure, external verification is currently impossible. If the references are noisy, incomplete, or systematically biased, the RadCliQ-v1 scores and the resulting MedVersa-first ranking become uninterpretable. RadCliQ-v1 was validated on other datasets, but no evidence shows it behaves well on this new private set. This is a load-bearing, empirically testable gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ReXrank, a public leaderboard for chest X-ray radiology report generation. The authors combine a new private dataset, ReXGradient (10,000 studies, 7,004 patients, 67 U.S. sites), with three public datasets (MIMIC-CXR, IU-Xray, CheXpert Plus) and eight automated metrics (BLEU-2, BERTScore, SembScore, RadGraph-F1, RadCliQ-v1, RaTEScore, GREEN, FineRadScore), using RadCliQ-v1 as the primary metric. Sixteen models from ten institutions are evaluated, and the paper reports that MedVersa is the top-scoring model on the findings-only task on ReXGradient and MIMIC-CXR. The authors argue that the framework provides standardized, meaningful comparisons and offers robustness insights across clinical settings.","tokens_in":16203,"tokens_out":8214,"duration_ms":67873,"significance":"If the framework is reliable, ReXrank would be a useful community asset for benchmarking report generation models, and the inclusion of 16 models with multiple established metrics is a strength. The publicly accessible leaderboard and the use of official implementations for several metrics are also positive aspects. However, the paper's central claim depends on the trustworthiness of the private ReXGradient references, which are not validated, and on the transparency of the evaluation pipeline, which is incomplete. These gaps currently prevent the rankings from being interpreted as clinically meaningful, so the manuscript needs substantial revision before its main assertions can be accepted.","major_comments":[{"comment":"ReXGradient, the private 10,000-study dataset that underpins the leaderboard rankings in Tables 1-4, is described only by size and provenance (Section 3.1). The manuscript does not state how the reference reports were produced, whether they are original clinical reports, how studies were selected, or whether any radiologist verified their accuracy and completeness. Section 2's inference that ReXGradient has 'high data quality' from 'remarkably low-performance variance across models' is not logically valid: low variance across models could equally reflect metric insensitivity, case homogeneity, or a narrow pathology distribution. Because ReXGradient is private and no human audit or human-metric correlation is provided, the RadCliQ-v1 scores and the resulting model rankings are not yet interpretable. I request at least a radiologist audit on a random subset, inter-rater reliability statistics, or a correlation of automated metrics with expert judgments on ReXGradient.","section":"Section 3.1 / Section 2"},{"comment":"The paper calls ReXrank a 'public leaderboard' but does not describe how to submit models or how the leaderboard is updated, and the Disclosures mention a '4x downsampled version' of ReXGradient without a URL or access procedure. The leaderboard URL in the abstract is a useful start, but without submission instructions, dataset access details, and a description of the evaluation pipeline used to produce Tables 1-4, the reported results cannot be reproduced or independently verified.","section":"Section 1 / Disclosures"},{"comment":"The CheXpert Plus evaluation uses only 200 validation studies, and Table 4 shows wide, heavily overlapping confidence intervals on the primary metric: for example, in the findings-only section the top three models are 0.81±0.12, 0.79±0.10, and 0.79±0.10, and in the findings+impression section the top two models tie at 0.51±0.07. These intervals mean that the CheXpert Plus rankings, and any conclusions about robustness drawn from them, are not statistically supported. The text should either restrict claims to datasets with adequate power or explicitly mark CheXpert Plus results as exploratory only.","section":"Section 3.4 / Table 4"},{"comment":"The LLM-based metrics GREEN and FineRadScore are computed with GPT-4o according to official implementations, but the manuscript does not report the exact model version, decoding parameters (temperature, top-p, max tokens), the number of runs, or how stochasticity was handled. Since these metrics contribute to the reported values in Tables 1-4 (e.g., GREEN and 1/FineRadScore with ±0.01 to ±0.02 intervals), the absence of these details makes the numbers non-reproducible and the reported confidence intervals uninterpretable.","section":"Section 3.3 / Tables 1-4"}],"minor_comments":[{"comment":"MedVersa's ReXGradient findings+impression 1/RadCliQ-v1 is reported as 0.98±0.05 in Sections 2 and 4, but Table 1 lists it as 0.98±0.01; please reconcile the discrepancy.","section":"Section 2 / Section 4 / Table 1"},{"comment":"The BERTScore values of -0.04±0.02 and -0.07±0.02 for RadFM and GPT4V in the findings+impression rows are surprisingly negative for BERTScore; please check the computation or clarify whether a different reference normalization was used.","section":"Table 4"},{"comment":"The metric column is labeled 'BLEU' in Table 3 but 'BLEU-2' in the other tables; the naming should be consistent.","section":"Table 3"},{"comment":"The phrase 'including ... etc.' after enumerating eight metrics is unnecessary and ambiguous, since all eight are already listed; please remove the 'etc.'","section":"Section 1"},{"comment":"Figure 2's y-axis has no visible label (the caption says it is the average 1/RadCliQ-v1), and the model names in Figure 2 are too small to read; improving axis labels and font sizes would aid comprehension.","section":"Figures 2 and 3"},{"comment":"The confidence intervals assume a normal distribution of the data, but several metrics are bounded or skewed (e.g., BERTScore and the reciprocal transforms); a brief justification or a bootstrap alternative would strengthen the statistical reporting.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"This is a benchmark/resource paper rather than a methodological advance. The conflict of interest regarding Gradient Health is disclosed, and the authors include public leaderboard access, which is commendable. The main risk is that the private ReXGradient dataset is unvalidated and inaccessible, so external verification of the central claims is currently impossible. I would urge the editor to treat the validation gap as a condition for acceptance rather than a mere presentation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReXrank fills a real gap in radiology report generation: there was no common leaderboard with consistent splits and metrics, and the field was comparing models trained on different splits. The paper ships a usable protocol, runs 16 public checkpoints through 8 metrics, and is transparent about what it does. ReXGradient is a genuinely new artifact: 10,000 studies from 67 sites, private, provided by Gradient Health. The disclosure that two authors hold equity in Gradient Health is good to see.\n\nThe public part is reproducible: MIMIC-CXR, IU-Xray, and CheXpert Plus splits are standard, checkpoints are linked, and metrics use official implementations. The numbers look internally consistent across tables and figures.\n\nThe soft spot is the private test set. Section 3.1 says only that ReXGradient has 10,000 studies from 7,004 patients across 67 sites. It never says how the reference reports were produced, whether they are original clinical reports, or whether any radiologist checked completeness or accuracy. The claim in Section 2 that low performance variance 'underscores' ReXGradient's 'high data quality' is a logical leap: low variance across models can reflect metric insensitivity or case homogeneity as easily as good references. Since the set is private and no access procedure is given beyond a mention of a 4x downsampled version with no URL, external verification is currently impossible. If the references are noisy or biased, the RadCliQ-v1 rankings, including MedVersa's top position, are uninterpretable.\n\nThe other issues are minor. CheXpert Plus validation set is only 200 studies, and confidence intervals are wide; the paper says this. The automated metrics are established elsewhere but not re-validated on ReXGradient; that is a limitation worth stating, not a fatal flaw. The circularity burden is low: the metrics come from prior work, the paper does not fit parameters, and self-citation of FineRadScore is not a problem since the implementation is public.\n\nWho is this for? Anyone building or evaluating chest X-ray report generation models. It is a benchmark resource, not a scientific derivation, so it should be judged on infrastructure and honesty. The infrastructure is solid; the honesty is mostly there, but the missing validation of ReXGradient's references is load-bearing.\n\nRecommendation: send it to peer review, but require the authors to document how ReXGradient references were created, report inter-rater reliability or a radiologist audit, and provide a concrete access procedure. With those, it could become a community standard.","headline":"Useful leaderboard with an unvalidated private test set; ship it to review but require reference validation.","tokens_in":16811,"tokens_out":1929,"would_cite":true,"duration_ms":17292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReXrank is a public leaderboard for AI-powered chest X-ray report generation that ranks sixteen models on a new 10,000-study private test set and three public datasets, using eight evaluation metrics to compare findings-only and…","keywords":["ReXrank","radiology report generation","chest X-ray","leaderboard","ReXGradient","RadCliQ-v1","MedVersa","evaluation metrics"],"falsifier":"Take a random sample of 200 ReXGradient studies, have two radiologists score each reference report for completeness and clinical correctness against the images, and correlate those scores with the models' RadCliQ-v1 rankings on the same studies; if the correlation is near zero, the leaderboard is measuring textual similarity rather than clinical quality.","tokens_in":15833,"feed_emoji":"🩻","tokens_out":6210,"duration_ms":54912,"temperature":0.7,"pith_summary":"The paper introduces ReXrank, a public leaderboard and challenge for AI-generated radiology reports from chest X-rays, built around a new private test set, ReXGradient, of 10,000 studies from 67 U.S. medical sites. ReXrank also evaluates models on the official test splits of MIMIC-CXR and IU-Xray and on CheXpert Plus's validation set, scoring each report with eight metrics and ranking findings-only and findings-plus-impression tracks separately. The paper's central claim is that this standardized setup makes model performances meaningfully comparable and reveals how robust each model is across different clinical data distributions. On this benchmark, MedVersa is the top model, reaching best findings-only $1/\\mathrm{RadCliQ\\text{-}v1}$ scores of $1.01 \\pm 0.01$ on ReXGradient and $1.10 \\pm 0.02$ on MIMIC-CXR. If the benchmark holds up, the field gains a shared yardstick for measuring progress in automated radiology reporting.","feed_headline":"New leaderboard crowns MedVersa top AI chest X-ray reporter","feed_subtitle":"ReXrank tests 16 models on 10,000 private studies plus MIMIC-CXR, IU-Xray, and CheXpert Plus with eight metrics.","key_machinery":"The load-bearing object is the leaderboard itself, but the argument runs through ReXGradient and the primary metric. ReXGradient is a private test set of 10,000 studies from 67 U.S. sites, chosen so models cannot train on it; the primary metric, RadCliQ-v1, is a composite of BLEU, BERTScore, SembScore, and RadGraph-F1, reported as its reciprocal so higher is better. The framework also scores each report with BLEU-2, BERTScore, SembScore, RadGraph-F1, RaTEScore, GREEN, and FineRadScore, and it evaluates findings-only and findings-plus-impression generation in separate tracks. Confidence intervals are computed under a normal-distribution assumption using the standard error of the mean.","core_discovery":"On its own terms, the paper's discovery is that a single standardized evaluation can produce a defensible ordering of current chest X-ray report generators and, at the same time, characterize the test distributions themselves. It reports that MedVersa consistently outperforms the other models, including GPT-4V, across most datasets and metrics, while models trained on multiple datasets tend to generalize better than single-dataset models. It also finds that IU-Xray is too easy to separate models, CheXpert Plus is the hardest and noisiest distribution, and ReXGradient shows strikingly small confidence intervals, which the paper reads as a sign of high data quality and reliability as a benchmark.","pith_inferences":["A radiologist audit of a random sample of ReXGradient reference reports would tell whether the small confidence intervals reflect genuinely consistent data quality or simply metric saturation; the paper itself reports no such audit.","RadCliQ-v1 is a composite of lexical and entity-overlap metrics, so a model tuned to maximize it could rise in the ranking without improving clinical accuracy; holding out a hidden subset of ReXGradient would make such gaming harder.","If ReXrank is extended to other modalities, the same logic would require a comparably large private multi-site test set with expert-verified reference reports, since public training sets would otherwise contaminate the benchmark."],"forward_implications":["Any model can be submitted to ReXrank and compared against all existing entries on the same private and public test sets, making reported gains in radiology report generation checkable rather than self-claimed.","MedVersa's top ranking establishes a concrete baseline that future chest X-ray report generators need to beat.","The separate findings-only and full-report tracks let developers see whether apparent gains come from findings content or from impression text, and reveal architectural choices such as using dedicated models per section.","The large spread and low scores on CheXpert Plus indicate that distribution shift remains a major barrier, so progress measured on MIMIC-CXR alone is not enough.","The same dataset-plus-metrics infrastructure is designed to extend to other imaging modalities beyond chest X-rays."],"supporting_citations":[{"why":"Supplies the MIMIC-CXR official test split of 2,347 studies used as one of the four benchmark datasets.","marker":"[12]"},{"why":"Supplies the IU-Xray public test set of 590 studies used as the low-difficulty distribution.","marker":"[8]"},{"why":"Supplies the CheXpert Plus validation set of 200 studies and the CheXpertPlus baseline model checkpoints.","marker":"[4]"},{"why":"Provides the R2Gen split of IU-Xray that the benchmark adopts.","marker":"[6]"},{"why":"Provides the RadCliQ composite metric and RadGraph-F1, the primary ranking tools.","marker":"[26]"},{"why":"Provides SembScore, the CheXbert-based domain metric included in the evaluation suite.","marker":"[20]"},{"why":"Supplies the MedVersa model and checkpoint that becomes the top-ranked system.","marker":"[30]"},{"why":"Supplies GREEN, an LLM-based clinically oriented metric in the eight-metric suite.","marker":"[17]"},{"why":"Supplies FineRadScore, an LLM-based correction-cost metric used in the suite.","marker":"[10]"}],"fun_headline_variants":["New leaderboard ranks AI chest X-ray reporters on 10k studies","ReXrank: AI radiology report writers face off on public bench","MedVersa wins first ReXrank crown for chest X-ray reporting","10k X-rays reveal best AI reporter for radiology reports"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking stands or falls on ReXGradient's reference reports being accurate and complete and on RadCliQ-v1 (and the other seven metrics) capturing clinically meaningful report quality, because the paper provides no radiologist audit or human-metric correlation for the new test set.","fun_headline_variants_meta":{"raw":{"variants":["New leaderboard ranks AI chest X-ray reporters on 10k studies","ReXrank: AI radiology report writers face off on public bench","MedVersa wins first ReXrank crown for chest X-ray reporting","10k X-rays reveal best AI reporter for radiology reports"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1895,"prompt_tokens":856,"completion_tokens":1039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":962}},"tokens_in":472,"tokens_out":1039,"duration_ms":9631,"temperature":1.0,"reasoning_tokens":962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:28:07.937316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 ReXGradient studies, have two radiologists score each reference report for completeness and clinical correctness against the images, and correlate those scores with the models' RadCliQ-v1 rankings on the same studies; if the correlation is near zero, the leaderboard is measuring textual similarity rather than clinical quality.","supporting_citations":[{"cited_title":"Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Association, 23(2):304–310, 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the IU-Xray public test set of 590 studies used as the low-difficulty distribution."},{"cited_title":"Evaluating progress in automatic chest x-ray radiology report generation.Patterns, 4(9), 2023","cited_arxiv_id":null,"evidence_quote":"Provides the RadCliQ composite metric and RadGraph-F1, the primary ranking tools."}],"review_version":1}