{"id":"86b96b3d-9d46-4a71-9fa0-f98e834cd9db","arxiv_id":"2506.18819","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RWESummary, a benchmark of 13 proprietary RWE studies graded by an LLM jury, ranked Gemini 2.5 Flash and Pro as the best summarizers among nine tested models.","lead":"This paper introduces RWESummary, a benchmark for testing how well large language models summarize structured real-world evidence (RWE) study results. It compares nine LLMs and finds Google's Gemini 2.5 models scored highest on its rubric, though the result depends on the chosen weights.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Gemini-2.5 'best overall' ranking depends on an LLM jury that includes Gemini 2.5 Pro; a jury swap or human validation could change the result.","rationale":"The reader's weakest assumption—LLM jury validity—is the same issue I would prioritize. The framework contribution (scenario, prompts, three metrics, open-source code) is real and useful; the paper is transparent about limitations and shows example data. But the central empirical claim that Gemini 2.5 models performed best overall rests on a jury that contains Gemini 2.5 Pro, creating a concrete self-preference mechanism. The paper's own sensitivity analysis of the weights shows the 'winner' is rubric-dependent, so the ranking is not a robust property of the models. My proposed test—dropping the conflicted juror and recomputing—is a low-cost, decisive check. If the ranking survives, the concern is substantially weakened; if it does not, the headline should be revised to describe the result as illustrative and rubric-specific. Therefore the appropriate verdict is CONDITIONAL, as the reader stated, and my analysis does not change that verdict.","tokens_in":12600,"tokens_out":6382,"duration_ms":71003,"concrete_test":"Replace Gemini 2.5 Pro on the jury with a model that is not among the nine candidates (e.g., GPT-4.1 or Claude Opus 4), keep the same prompts and weights, and recompute all three metric scores and the weighted overall ranking for the same 13 studies. If the top-ranked model changes or Gemini 2.5 drops out of the top group, the published 'Gemini 2.5 best overall' claim is an artifact of jury composition; if Gemini 2.5 still wins, the self-preference concern is mitigated, though a human validation subset would still be advisable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical result is computed as the mean of scores from a three-member LLM jury (o4-mini, Claude Sonnet 4, Gemini 2.5 Pro) that includes one of the models being ranked (Gemini 2.5 Pro). Because the summary scores feed directly into the weighted rubric, any systematic self-preference or provider-level bias in that juror directly inflates Gemini 2.5's scores and can change the ranking. The paper itself concedes the jury 'should be validated by human review in future applications' (Discussion, Limitations), so the ranking is presented without the validation needed to support it. The weighting rubric is also not a stable basis for 'best overall': the paper shows that changing the inference-time weight or the normalization maximum flips the winner (Results), so the headline is an artifact of the illustrative weights. The jury issue is the more load-bearing of the two, because even a fixed weighting scheme cannot rescue rankings if the judge is biased or inaccurate. Notably, the numeric-accuracy submetric is objectively verifiable by parsing the JSON results, so this concern is testable without a full human study.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RWESummary, a benchmark for evaluating LLMs on the task of summarizing structured real-world evidence (RWE) study outputs. The benchmark consists of one scenario, a dataset of 13 proprietary RWE studies from Atropos Health, and three LLM-jury-based evaluation metrics (direction of effect, numeric accuracy, and completeness). Using this framework, the authors compare nine LLMs from Anthropic, Google, and OpenAI, and report that the Gemini 2.5 models perform best overall under an illustrative weighted scoring rubric. The paper also discusses limitations including small sample size, lack of human validation of the LLM jury, and dependence on the specific prompts used.","tokens_in":12814,"tokens_out":2297,"duration_ms":24361,"significance":"If the framework is adopted and validated, RWESummary would fill a genuine gap: existing benchmarks such as MedHELM do not explicitly target structured-to-narrative RWE summarization, which is a practically important task as automated RWE generation scales. The paper provides a clear protocol, public code on GitHub, and a concrete worked example of the prompt and evaluation design, which are useful contributions. However, the headline empirical finding (Gemini 2.5 best overall) is currently fragile: it rests on an unvalidated LLM jury that includes one of the models being ranked (Gemini 2.5 Pro), a small proprietary dataset, and an illustrative weighting scheme whose parameters change the winner. The paper's own Discussion acknowledges several of these limitations, but the abstract and central framing present the model recommendation as a firm result rather than a provisional, context-dependent observation.","major_comments":[{"comment":"The three-member LLM jury includes Gemini 2.5 Pro (Methods, Evaluations), which is also one of the nine models being ranked. Since each summary's final score is the mean of the three jury scores, any self-preference or provider-level bias in Gemini 2.5 Pro's grading directly inflates the reported performance of the Gemini 2.5 models. The paper states in the Discussion that the jury 'should be validated by human review in future applications,' but the headline claim that Gemini 2.5 models performed 'best overall' is presented without such validation. This is load-bearing because all model rankings are determined solely by this jury; a different jury composition could change the ranking. The numeric-accuracy submetric is objectively checkable by parsing the JSON results, so a partial validation is feasible and should be reported.","section":"Methods, Evaluations"},{"comment":"The 'best overall' conclusion is not robust to the choice of weighting scheme. The paper itself shows that setting the inference-time weight to 4 or 5 makes Gemini 2.0 Flash tie with the other Gemini models, and a weight of 6 or higher makes Gemini 2.0 Flash preferred; changing the normalization max to 1.0 for direction and numeric accuracy makes Gemini 2.5 Flash the sole winner. Therefore the abstract's statement that 'Gemini 2.5 models performed best overall (both Flash and Pro)' is an artifact of the specific illustrative weights and normalization choices, not a stable property of the models. The claim should be tempered or accompanied by a sensitivity analysis showing which conclusions hold across reasonable parameter choices.","section":"Results, Weighted scoring rubric"},{"comment":"The benchmark uses 13 proprietary RWE studies that 'cannot be made public' (Dataset), and the summarization prompt is tailored to Atropos Health's internal tool. Since the data are not available to other researchers, RWESummary as published cannot be independently run or extended by the community, which undermines its claim to be a 'reusable' benchmark. The paper should either provide an anonymized or synthetic version of the dataset, or clearly scope the contribution as an internal evaluation framework whose data are not yet publicly available, and explain how others could apply the framework to their own data.","section":"Dataset, Discussion"}],"minor_comments":[{"comment":"There is a typo: 'machine-readible' should be 'machine-readable'.","section":"Discussion"},{"comment":"The code availability section mentions a commit but does not provide the GitHub URL or commit hash; please include the full link so readers can access the code.","section":"Methods, Code Availability"},{"comment":"The sentence 'We implemented RWESummary as a new benchmark in the extensible HELM framework and (7) believe it would be appropriate for inclusion in the MedHELM suite' has a grammatical issue; the citation '(7)' interrupts the sentence unnecessarily.","section":"Methods"},{"comment":"In the appendix, 'non-signficant' contains a ligature artifact; it should be 'non-significant'.","section":"Appendix 1"},{"comment":"Figures 1 and 2 are described but not included in the manuscript text; please ensure they are present or referenced correctly.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: RWESummary is a legitimate and clearly-described addition to the MedHELM suite for a narrow but real task—summarizing structured RWE output. The framework is the first I know of aimed specifically at this use case, and the authors are admirably transparent about its limits. The weak spot is the empirical ranking: the 'Gemini 2.5 best' claim depends on an LLM jury that includes Gemini 2.5 Pro, and the authors themselves concede the jury needs human validation. That makes the ranking provisional, not worthless.\n\nWhat's genuinely new: the scenario (structured RWE JSON → plain-language summary), the three evaluation dimensions (direction, numbers, completeness), and the adoption of the HELM/MedHELM extensible structure. The code is public, the prompts are in the appendix, and the limitations section is unusually honest. They explicitly say 13 studies is small, results are provisional, weights are illustrative, and the jury should be human-validated. That's good scientific hygiene.\n\nWhere it's soft: (1) The self-preference channel is real. Having Gemini 2.5 Pro score summaries that include Gemini-generated summaries creates a plausible bias, and the paper doesn't test its direction or magnitude. (2) The data are proprietary, so the benchmark isn't independently reproducible—though the code and prompts are, and the task itself can be re-run with public RWE output. (3) The weighting rubric: the paper shows the winner changes under alternative weights and normalizations. That's not fatal—they label it illustrative—but it undercuts the headline 'best overall' phrasing. (4) With n=13, the score differences (e.g., 0.72–0.95 for direction) may not be stable.\n\nI don't think this deserves a desk reject. It's a solid framework paper with an empirical claim that needs stronger validation. I'd send it to review with a clear request: add human validation of the jury, report the sensitivity to jury composition, and either release a shared anonymized subset or demonstrate the benchmark on public RWE data. If those are addressed, it's a useful contribution to a small but growing niche.\n\nVerdict: conditional accept, in spirit. I'd bring it to our reading group to discuss the jury-design issue, and I'd cite it if I were working on LLM evaluation for clinical summarization.","headline":"A clearly-described, honestly-scoped RWE summarization benchmark whose headline model ranking is provisional because the evaluating jury includes one of the models being ranked.","tokens_in":13338,"tokens_out":2944,"would_cite":false,"duration_ms":27857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces RWESummary, a benchmark that grades LLM-written summaries of real-world evidence studies, and reports that Gemini 2.5 models performed best overall.","keywords":["real-world evidence","LLM benchmarking","summarization","medical natural language processing","LLM jury evaluation","structured study output","RWE summarization","model selection"],"falsifier":"Run the same 13 studies' summaries through a panel of blinded human expert reviewers using the same three binary metrics and compare their scores to the LLM jury's; if human rankings disagree with the jury's, or if excluding Gemini 2.5 Pro from the jury changes Gemini 2.5's lead, the central ranking claim is not settled.","tokens_in":12406,"feed_emoji":"📊","tokens_out":6258,"duration_ms":62831,"temperature":0.7,"pith_summary":"Real-world evidence studies now pour out of large linked databases, but no benchmark measured whether LLMs can turn their structured results into accurate plain-English summaries. This paper claims RWESummary fills that gap: a scenario, a dataset of 13 real RWE studies, and three LLM-jury evaluations that check the direction of reported effects, the accuracy of numbers, and completeness for significant outcomes. Applied to nine current models, the benchmark ranked the Gemini 2.5 family first overall under the paper's example weighting, though no single model won every metric. If correct, RWESummary gives teams a reusable, extensible way to choose a summarization model for clinical evidence, rather than relying on general-purpose leaderboards.","feed_headline":"Gemini 2.5 tops new benchmark for summarizing real-world evidence","feed_subtitle":"A reusable three-check test scores LLM summaries for effect direction, numeric accuracy, and completeness.","key_machinery":"The load-bearing object is RWESummary itself: one scenario (structured RWE study output to plain-English summary), a 13-study proprietary dataset, and three LLM-jury evaluations. The jury, following the approach of the broader benchmark suite this work extends, is a panel of three distinct frontier LLMs that each return a binary score for direction of effect, numeric accuracy, and completeness, prompted separately per metric because pilot testing found joint grading less reliable. The final comparison combines these scores through a user-weightable normalization scheme, with direction of effect weighted highest. What this machinery does is convert an open-ended summarization-quality question into three checkable semantic sub-questions that can be scored automatically, making cross-model comparison reproducible without human ratings.","core_discovery":"The paper's central claim is that RWESummary is a useful foundation-model benchmark for summarizing structured real-world evidence, and that on its initial run the Gemini 2.5 models (Flash and Pro) performed best overall. The benchmark task is deliberately narrow: an LLM receives a study question, a PICOT-format design, and structured results including sample sizes, baseline covariates, and effect estimates, and must write a short scientific summary interpreting the results. Three evaluation metrics target the errors the authors observe most often: whether each effect's direction matches the data, whether every reported number is sourced correctly with reasonable rounding, and whether all significant outcomes appear. Scores come from a three-member LLM jury that rates each summary separately on each metric, and the jury's mean scores feed a normalized, weighted rubric. The paper reports that completeness was generally strong across models, while direction of effect and numeric accuracy varied more, and that the overall ranking is sensitive to the weighting choices.","pith_inferences":["Editorial inference: Because the dataset is proprietary and only 13 studies, the benchmark's internal validity is not yet matched by external validity; an obvious next step is a public or multi-site version with a larger, diverse study sample.","Editorial inference: The ranking could plausibly shift if the jury were replaced by human expert judges; a direct head-to-head between the LLM jury and human raters on the same 13 summaries would test that.","Editorial inference: The same three-error decomposition (direction, numbers, completeness) could be adapted beyond medicine, e.g., to financial or policy report summarization, wherever structured output becomes narrative text.","Editorial inference: A self-preference risk exists if a jury includes a model that is also a candidate; excluding the candidate from its own jury, or testing jury composition sensitivity, would make the benchmark more robust."],"forward_implications":["RWE summarization becomes a benchmarkable task: future models can be scored on the same scenario and metrics, and the 13-study dataset can be extended to more therapeutic areas and designs.","Teams building clinical summarization tools get a provisional default recommendation: among the nine models tested, the Gemini 2.5 family led the example-weighted overall score.","The separated three-metric grading means a user can pick a model for what matters most: effect direction, numeric fidelity, or completeness, rather than a single aggregate.","The weighting rubric makes the ranking transparent and adjustable; changing the inference-time weight or normalization choices can change which Gemini model wins, so the benchmark supports decision-making rather than a single eternal ranking."],"supporting_citations":[{"why":"Supplies the LLM-jury design and the benchmark-suite structure that RWESummary extends.","marker":"[10]"},{"why":"Supplies the extensible evaluation framework in which RWESummary is implemented.","marker":"[7]"},{"why":"Provides evidence that diverse LLM juries can replace individual judges, supporting the jury method.","marker":"[27]"},{"why":"Names BERTScore, the purely lexical metric the authors set aside in favor of semantic LLM grading.","marker":"[15]"},{"why":"Benchmarks LLMs in evidence-based medicine, establishing the broader evaluation context this task extends.","marker":"[6]"},{"why":"Evaluates LLMs on medical evidence summarization, a related but distinct task this work differentiates from.","marker":"[13]"},{"why":"Shows clinical text summarization performance, used to contrast with the structured-output-to-narrative use case.","marker":"[26]"},{"why":"Grounds the motivation that strong LLM performance on one task does not guarantee performance on related tasks, justifying a dedicated RWE benchmark.","marker":"[11]"}],"fun_headline_variants":["Gemini 2.5 tops fresh benchmark for RWE study summaries","RWESummary ranks LLMs for factual RWE summarization","Three-part test finds Gemini 2.5 best for RWE summaries","New benchmark scores LLM summaries of real-world evidence","Gemini 2.5 wins on new RWE summarization benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's model rankings assume the three-LLM jury judges summaries the way a human expert would; if the jury is biased or inaccurate, the reported order of models could change.","fun_headline_variants_meta":{"raw":{"variants":["Gemini 2.5 tops fresh benchmark for RWE study summaries","RWESummary ranks LLMs for factual RWE summarization","Three-part test finds Gemini 2.5 best for RWE summaries","New benchmark scores LLM summaries of real-world evidence","Gemini 2.5 wins on new RWE summarization benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3567,"prompt_tokens":925,"completion_tokens":2642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2551}},"tokens_in":541,"tokens_out":2642,"duration_ms":20334,"temperature":1.0,"reasoning_tokens":2551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:14:46.144745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 13 studies' summaries through a panel of blinded human expert reviewers using the same three binary metrics and compare their scores to the LLM jury's; if human rankings disagree with the jury's, or if excluding Gemini 2.5 Pro from the jury changes Gemini 2.5's lead, the central ranking claim is not settled.","supporting_citations":[],"review_version":1}