{"id":"d80ad2f1-9d5c-4f8f-bf57-5c19566f8199","arxiv_id":"2505.22787","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On 284 medical questions derived from Cochrane systematic reviews, the best of 24 LLMs, DeepSeek V3, matches expert conclusions 62.40% of the time, and all tested models struggle with uncertain or low-quality evidence.","lead":"MedEvidence is a new benchmark of 284 clinical questions taken from 100 Cochrane systematic reviews, paired with the original research studies. Tested across 24 large language models, the best model, DeepSeek V3, matches expert conclusions only 62.40% of the time, and models show systematic overconfidence and weak skepticism of low-quality evidence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'same studies' guarantee is only a 75%-by-weight guarantee; up to 25% of the evidence experts used may be missing or unusable, so the reported failure rate may partly reflect missing information rather than failed reasoning.","rationale":"The central claim requires that the benchmark actually gives LLMs the same evidence experts used. The curation pipeline explicitly relaxes this requirement to at least 75% of meta-analysis weight coming from 'valid' studies, leaving up to 25% of the evidence potentially absent or unusable. This is the least secure link in the chain: if those missing studies changed the pooled direction, a model could be correct relative to the supplied evidence and still be marked wrong. The label-conversion and source-concordance concerns are measurable but more likely to add symmetric noise; the evidence-sufficiency gap is systematically biased toward underestimating model capability. The proposed check would settle whether the 37-40% failure rate survives on questions with complete evidence. If it does, the central claim is robust; if not, the conclusion needs to be rephrased as 'LLMs fail when given incomplete subsets of the evidence.' The reader's CONDITIONAL verdict already requires such tightening, so my read does not move the verdict.","tokens_in":28958,"tokens_out":5139,"duration_ms":59907,"concrete_test":"Stratify the 284 questions by the fraction of meta-analysis weight contributed by 'valid' studies (e.g., 75-89%, 90-99%, 100%) and recompute DeepSeek V3 and GPT-4.1 accuracy within each bin. Then, for questions below 100%, augment the provided source set with the full texts (or complete numerical data) of the studies whose weight is currently missing, and re-run the same models. If accuracy in the 100%-valid bin is materially higher than in the 75-99% bin, or if augmentation closes the gap, the headline failure rate conflates missing evidence with inability to synthesize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3, Step 4 ('Question feasibility validation') admits a question when at least 75% of the meta-analysis weight comes from 'valid' studies, where validity requires the provided source to contain numerical data on both groups and on the outcome. This means that for up to 25% of the evidence weight that the Cochrane authors actually used, the LLM's input may lack the numerical information needed to reproduce the expert conclusion. The paper's central claim is that frontier LLMs fail to match SR conclusions when 'given the same studies'; but if a quarter of the pooled evidence is absent or non-numerical in the supplied abstracts/full texts, models are not actually given the same information. A model that correctly reasons from the provided subset could be scored wrong relative to a conclusion that depended on the missing weight. The benchmark does not report accuracy stratified by valid-weight fraction, so the magnitude of this confound is unmeasured. The reader's other points about label-conversion reliability and DeepSeek-generated source concordance are real but secondary; this evidence-sufficiency gap most directly threatens the 'given the same studies' wording of the main conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedEvidence, a benchmark of 284 questions derived from 100 Cochrane systematic reviews, pairing each question with the source studies cited by the review authors and a five-class expert-derived answer (higher, lower, no difference, uncertain effect, insufficient data). The authors evaluate 24 LLMs in a zero-shot setting with exact-match accuracy, zero temperature, bootstrapped confidence intervals, and both a basic and an expert-guided prompt. They report that frontier models such as DeepSeek V3 and GPT-4.1 reach only about 62% and 60% accuracy, and they analyze failure modes including long-context degradation, overconfidence, and a lack of scientific skepticism toward low-quality evidence. The central claim is that, given the same studies used by human experts, even frontier LLMs fail to match systematic review conclusions in at least 37% of cases.","tokens_in":29179,"tokens_out":4340,"duration_ms":46843,"significance":"If the benchmark's assumptions hold, MedEvidence is a useful and much-needed resource: it provides expert-grounded, closed-form evaluation for multi-document medical evidence synthesis, with 284 questions, 329 source articles, metadata on evidence certainty and full-text availability, and a public codebase and dataset. The evaluation is methodical in several respects: zero-temperature inference, exact-match scoring with valid-output checks, bootstrap confidence intervals, two prompting regimes, and a broad model spread across size, reasoning ability, and medical fine-tuning. The headline finding that current LLMs are far from saturating the benchmark would be important for the many deployed systems that already use LLMs for medical evidence synthesis. However, the significance depends on the strength of the 'same studies' guarantee and the quality of the human-curated labels; both require the additional validation requested in the major comments.","major_comments":[{"comment":"The benchmark admits a question when at least 75% of the meta-analysis weight comes from 'valid' studies, which means that for up to 25% of the evidence weight used by the Cochrane authors, the supplied sources may lack the numerical data needed to reproduce the expert conclusion. Sections 6 and 7 state that models are evaluated 'given the same studies' and that frontier LLMs fail in at least 37% of evaluated cases, but the actual guarantee is only a 75%-by-weight guarantee. The manuscript does not report the distribution of valid-weight fractions across the 284 questions, nor accuracy stratified by that fraction, so the magnitude of this missing-information confound is unmeasured. Please report such stratified results (e.g., by bins of valid-study weight) and, if the effect is material, temper the 'same studies' wording in the abstract and conclusion.","section":"§3, 'Question feasibility validation' (Step 4)"},{"comment":"The ground-truth labels are produced by human annotators, but no inter-annotator agreement measure is reported. The five answer classes include subtle boundaries (notably uncertain effect versus no difference versus insufficient data), and the per-class recall analysis in Figure 4(b) and Appendix H depends on the reliability of these conversions. Without agreement statistics (e.g., Cohen's kappa on a double-annotated subset), label noise is unquantified. Please report agreement statistics and the adjudication procedure.","section":"§3, 'Conclusion to question conversion' (Step 2)"},{"comment":"The source-concordance feature that underlies the scientific-skepticism finding in Figure 5(b) is computed by DeepSeek V3, one of the models under evaluation, with no validation against human labels. This introduces a circular dependency for that specific analysis: the claim that models lack skepticism toward low-quality findings is partly based on classifications generated by the same type of model being assessed. Please validate DeepSeek V3's single-source classifications against human annotations on a random sample, or replace this feature with an external gold standard for the analysis in Figure 5(b).","section":"§3, 'LLM-assisted metadata' / Figure 5(b)"}],"minor_comments":[{"comment":"The sentence 'we use an LLMs to assess the percentage' contains a typo and should read 'we use an LLM to assess'.","section":"§3, paragraph on LLM-assisted metadata"},{"comment":"GPT-o1 is listed as 'Generalist Non-Reasoning' but is a reasoning model; this appears to be an error in the model-type column.","section":"Appendix Table 3"},{"comment":"The main text states that performance consistently declines with token length, but Appendix Figure 15 shows that models over 100B parameters do not exhibit this decline; please qualify this claim in the main text to match the appendix.","section":"§6, 'Model performance decreases as token length increases' and Appendix G"},{"comment":"The remarks contain small typos ('reconcilliation', 'prevous') that should be corrected.","section":"Appendix O.1"},{"comment":"Exact-match evaluation counts invalid outputs as incorrect, which conflates instruction-following failures with reasoning failures; the paper does report valid-output rates, but consider also reporting accuracy conditioned on valid output in the main figure for clarity.","section":"§5.1, 'LLM evaluation'"},{"comment":"The paper says certainty is extracted 'only when it was explicitly provided by the original SR authors,' but 20.8% of questions have N/A; please clarify whether N/A means the SR did not provide a certainty rating or the annotators did not extract it.","section":"Figure 11(a) and §3, evidence certainty extraction"}],"recommendation":"major_revision","confidential_remarks":"The central measurement and the released benchmark are valuable, but the three load-bearing issues—the 75%-by-weight evidence sufficiency guarantee, missing inter-annotator agreement, and the DeepSeek-generated source concordance feature—should be addressed before publication. The first is the most important, as it directly qualifies the abstract's 'given the same studies' framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. This paper builds a benchmark that the field was missing: 284 human-curated questions from 100 Cochrane reviews, each paired with the actual source studies, plus evidence certainty and source-concordance labels. The evaluation of 24 models is broad and carefully run: exact match, zero temperature, bootstrapped CIs, prompt variants, and valid-output checks. That is real work, and the released code and data make it reusable. The main finding—frontier models top out around 62% and systematically avoid uncertainty, ignore evidence quality, and degrade on long contexts—is credible and consistent across model families. I would trust the direction of that result.\n\nThe soft spots are real but not fatal. The stress-test concern about the 75%-by-weight feasibility threshold is the most serious. A question is admitted if at least 75% of meta-analysis weight comes from 'valid' studies, so up to 25% of the evidence experts used may be missing or non-numerical in the model's input. The paper's headline claim, 'given the same studies, frontier LLMs fail... in at least 37% of cases,' overstates what was actually tested. Models were sometimes not given the same information. The authors should stratify accuracy by valid-weight fraction or report sensitivity to that threshold. Without that, the 37% failure rate is an upper bound on \"same studies\" failure, not a clean estimate.\n\nThe other concerns are secondary. No inter-annotator agreement on question conversion means label noise is unmeasured, but the conversion criteria are explicit and the 75% threshold likely absorbs some of that. Source concordance being generated by DeepSeek V3, one of the benchmarked models, is a genuine circularity, though it is used for explanatory analysis rather than the headline accuracy numbers. Reusing BIOMEDICA is fine; it is their own released resource and they disclose it.\n\nNet: this is a solid, important measurement paper that deserves serious refereeing. The core negative result—LLMs are far from matching expert SR conclusions on this kind of task—will hold up. What needs tightening before publication is the precision of the \"same studies\" claim and a sensitivity analysis around the 75% threshold. I would send it to peer review with that request.","headline":"A genuinely useful benchmark with a mostly sound negative result, but the 'same studies' claim is softer than advertised because up to 25% of meta-analytic weight can be missing from model inputs.","tokens_in":29698,"tokens_out":1004,"would_cite":true,"duration_ms":13322,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Given the same source studies as expert reviewers, the strongest evaluated LLMs fail to reproduce the review's conclusion in at least 37% of cases.","keywords":["MedEvidence","systematic reviews","medical evidence synthesis","large language models","LLM evaluation","evidence certainty","scientific skepticism","clinical decision support"],"falsifier":"Ask independent clinical experts, given only the same source studies, to answer the questions that every model answered incorrectly (53 of 284); if experts also fail to recover the review's conclusion from those sources, the benchmark's ground truth is not reachable from the supplied evidence, and the 'at least 37% failure' figure would overstate the LLM-specific deficit.","tokens_in":28758,"feed_emoji":"🧠","tokens_out":9401,"duration_ms":105577,"temperature":0.7,"pith_summary":"This paper asks a deceptively simple question: if a large language model is handed the same individual studies that human experts used to write a systematic review, can it reach the same conclusion about whether a treatment changes an outcome? To make the question testable, the authors built MedEvidence, a benchmark of 284 conclusions drawn from 100 systematic reviews, each paired with its source studies and a five-option answer key (higher, lower, no difference, uncertain effect, insufficient data). They ran 24 models of varying size, reasoning style, and medical specialization through this benchmark. Their central finding is that the best-performing model reaches 62.4% accuracy, another frontier model reaches 60.4%, and the paper therefore concludes frontier LLMs fail to match expert conclusions on at least 37% of evaluated cases. The result matters because LLM-assisted evidence-synthesis tools are already being used by clinicians and policymakers.","feed_headline":"Frontier LLMs miss at least 37% of review conclusions","feed_subtitle":"Even with the same studies experts used, the best models scored only about 62% accuracy on MedEvidence.","key_machinery":"The load-bearing object is MedEvidence itself: a set of 284 conclusion-level question-answer pairs, created by converting the 'Main Results' statements of 100 systematic reviews into five mutually exclusive labels and attaching the source studies that the review authors used. The task is deliberately stripped down, with no literature search, screening, or risk-of-bias assessment, so that a model's score isolates one skill: given the same evidence, can it reach the same direction-of-effect conclusion as the experts? The benchmark's metadata do the analytical work: author-reported evidence certainty, source concordance (the share of individual source studies whose answer agrees with the correct answer), token length, medical specialty, and full-text availability let the authors attribute failures to specific mechanisms rather than just reporting an aggregate score.","core_discovery":"MedEvidence tests whether LLMs can perform conclusion-level evidence synthesis, not just fact recall: each item gives the model the raw studies behind one systematic-review conclusion, with no access to the review itself, and asks for a forced-choice answer among five labels. The paper's central claim is that current LLMs cannot yet match expert conclusions under these conditions, and that the shortfall is systematic rather than random. Models perform worst on the 'uncertain effect' class, and the confusion patterns show models avoiding that label, confusing it with 'no difference,' and overpredicting directional answers. Model accuracy also tracks evidence certainty and source concordance more steeply than expert reasoning would predict, falling to about 41% when no source agrees with the correct answer versus about 92% when all sources agree. The paper interprets this as a lack of scientific skepticism: models are swayed by how many sources point in a direction, not by whether those sources are reliable.","pith_inferences":["Beyond the paper: since the benchmark scores only the final five-way label, it does not measure LLM performance at earlier review stages such as screening or data extraction; a model could be useful there while still failing at conclusion-drawing, and the paper does not claim otherwise.","Beyond the paper: the source-concordance feature used to diagnose 'scientific skepticism' was computed by one of the evaluated models rather than by independent humans, so that specific curve should be re-measured before it is used as a stable diagnostic.","Beyond the paper: a natural extension is to couple the benchmark with a retrieval step, since the paper deliberately isolates synthesis from search; adding retrieval would show whether the failure sits in reasoning over provided evidence or in finding the right evidence.","Beyond the paper: because performance degrades with context length, agentic systems that iteratively select and read many sources may need explicit evidence-budgeting or hierarchical summarization rather than larger context windows."],"forward_implications":["At current capability, prompting an LLM to reproduce a systematic-review conclusion from the included studies is not reliable enough for clinical deployment: the best models are wrong on more than a third of cases.","Scaling along existing axes, including more test-time reasoning, larger parameter counts beyond about 70 billion, and medical fine-tuning, does not consistently close the gap, so progress will require a different intervention.","LLM outputs on uncertain evidence are overconfident: models under-use the 'uncertain effect' answer and over-predict clear direction, which is dangerous in clinical settings where hedging is the correct expert behavior.","Because accuracy declines as input token length grows, performance depends on how evidence is packaged; full-text access helps the strongest models, while abstract-only deployments, common in practice, likely suppress accuracy.","The paper's expert-guided prompt, which explicitly asked models to grade study quality, did not remove the lack of scientific skepticism, implying that prompting alone is insufficient and models need mechanisms that actually weigh study quality."],"supporting_citations":[{"why":"Supplies open-access full texts for source studies, expanding what models see beyond abstracts.","marker":"[30]"},{"why":"Supplies abstracts for the remaining source studies, the baseline evidence available for each question.","marker":"[31]"},{"why":"Provides the retrieval filter used to select the systematic-review corpus from which all benchmark questions were built.","marker":"[32]"},{"why":"Provides the chunked-refinement procedure used when inputs exceed a model's context window, shaping the long-context results.","marker":"[42]"},{"why":"Documents that models struggle to use long in-context inputs, the prior the authors use to interpret token-length degradation.","marker":"[44]"},{"why":"Documents verbal overconfidence amplified by reinforcement learning from human feedback, used to explain models' reluctance to answer 'uncertain effect.'","marker":"[48]"},{"why":"Documents that fine-tuning can harm generalization, the basis for interpreting medical-finetuned models' worse accuracy.","marker":"[49]"}],"fun_headline_variants":["Can LLMs reason like clinical experts? Not yet, says 100-review benchmark","LLMs don't doubt weak evidence—they just count agreeing sources","Best LLMs score 62% on matching systematic review conclusions","LLMs lose scientific skepticism: overconfident, swayed by source count","MedEvidence: 24 LLMs fail to match expert systematic review verdicts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers assume that the human-built answer key is correct and that the studies given to each model cover enough of the evidence (at least 75% of the pooled weight in the meta-analysis) for the right answer to be reachable; if those conversions or selections are noisy, the accuracy figures may misstate true LLM capability.","fun_headline_variants_meta":{"raw":{"variants":["Can LLMs reason like clinical experts? Not yet, says 100-review benchmark","LLMs don't doubt weak evidence—they just count agreeing sources","Best LLMs score 62% on matching systematic review conclusions","LLMs lose scientific skepticism: overconfident, swayed by source count","MedEvidence: 24 LLMs fail to match expert systematic review verdicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4164,"prompt_tokens":1020,"completion_tokens":3144,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":3047}},"tokens_in":636,"tokens_out":3144,"duration_ms":23523,"temperature":1.0,"reasoning_tokens":3047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:00:08.953196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent clinical experts, given only the same source studies, to answer the questions that every model answered incorrectly (53 of 284); if experts also fail to recover the review's conclusion from those sources, the benchmark's ground truth is not reachable from the supplied evidence, and the 'at least 37% failure' figure would overstate the LLM-specific deficit.","supporting_citations":[{"cited_title":"Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies open-access full texts for source studies, expanding what models see beyond abstracts."},{"cited_title":"Bethesda (MD): National Center for Biotechnology Information (US), 2010-","cited_arxiv_id":null,"evidence_quote":"Supplies abstracts for the remaining source studies, the baseline evidence available for each question."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the retrieval filter used to select the systematic-review corpus from which all benchmark questions were built."},{"cited_title":"Refinedocumentschain","cited_arxiv_id":null,"evidence_quote":"Provides the chunked-refinement procedure used when inputs exceed a model's context window, shaping the long-context results."},{"cited_title":"Fine-tuning is fine, if calibrated","cited_arxiv_id":null,"evidence_quote":"Documents that fine-tuning can harm generalization, the basis for interpreting medical-finetuned models' worse accuracy."}],"review_version":1}