Pith. sign in

REVIEW 3 major objections 6 minor 55 references

Can Large Language Models Match the Conclusions of Systematic Reviews?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Given the same source studies as expert reviewers, the strongest evaluated LLMs fail to reproduce the review's conclusion in at least 37% of cases.

desk verdict A genuinely useful benchmark with a mostly sound negative result, but the 'same studies' claim is softer than advertised because up to 25% of meta-analytic weight can be missing from model inputs. read the letter →

arxiv 2505.22787 v1 pith:MVSLUBYH submitted 2025-05-28 cs.CL

Christopher Polzak , Alejandro Lozano , Min Woo Sun , James Burgess , Yuhui Zhang , Kevin Wu , Serena Yeung-Levy This is my paper
classification cs.CL
keywords MedEvidencesystematicreviewsmedicalsynthesislargelanguagemodelsLLMevaluationcertaintyscientificskepticismclinicaldecisionsupport
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a deceptively simple question: if a large language model is handed the same individual studies that human experts used to write a systematic review, can it reach the same conclusion about whether a treatment changes an outcome? To make the question testable, the authors built MedEvidence, a benchmark of 284 conclusions drawn from 100 systematic reviews, each paired with its source studies and a five-option answer key (higher, lower, no difference, uncertain effect, insufficient data). They ran 24 models of varying size, reasoning style, and medical specialization through this benchmark. Their central finding is that the best-performing model reaches 62.4% accuracy, another frontier model reaches 60.4%, and the paper therefore concludes frontier LLMs fail to match expert conclusions on at least 37% of evaluated cases. The result matters because LLM-assisted evidence-synthesis tools are already being used by clinicians and policymakers.

What carries the argument

The load-bearing object is MedEvidence itself: a set of 284 conclusion-level question-answer pairs, created by converting the 'Main Results' statements of 100 systematic reviews into five mutually exclusive labels and attaching the source studies that the review authors used. The task is deliberately stripped down, with no literature search, screening, or risk-of-bias assessment, so that a model's score isolates one skill: given the same evidence, can it reach the same direction-of-effect conclusion as the experts? The benchmark's metadata do the analytical work: author-reported evidence certainty, source concordance (the share of individual source studies whose answer agrees with the correct answer), token length, medical specialty, and full-text availability let the authors attribute failures to specific mechanisms rather than just reporting an aggregate score.

What would settle it

Ask independent clinical experts, given only the same source studies, to answer the questions that every model answered incorrectly (53 of 284); if experts also fail to recover the review's conclusion from those sources, the benchmark's ground truth is not reachable from the supplied evidence, and the 'at least 37% failure' figure would overstate the LLM-specific deficit.

Watch

Extended reading notes

Core claim

MedEvidence tests whether LLMs can perform conclusion-level evidence synthesis, not just fact recall: each item gives the model the raw studies behind one systematic-review conclusion, with no access to the review itself, and asks for a forced-choice answer among five labels. The paper's central claim is that current LLMs cannot yet match expert conclusions under these conditions, and that the shortfall is systematic rather than random. Models perform worst on the 'uncertain effect' class, and the confusion patterns show models avoiding that label, confusing it with 'no difference,' and overpredicting directional answers. Model accuracy also tracks evidence certainty and source concordance more steeply than expert reasoning would predict, falling to about 41% when no source agrees with the correct answer versus about 92% when all sources agree. The paper interprets this as a lack of scientific skepticism: models are swayed by how many sources point in a direction, not by whether those sources are reliable.

Load-bearing premise

The headline numbers assume that the human-built answer key is correct and that the studies given to each model cover enough of the evidence (at least 75% of the pooled weight in the meta-analysis) for the right answer to be reachable; if those conversions or selections are noisy, the accuracy figures may misstate true LLM capability.

Editorial extensions

If this is right

  • At current capability, prompting an LLM to reproduce a systematic-review conclusion from the included studies is not reliable enough for clinical deployment: the best models are wrong on more than a third of cases.
  • Scaling along existing axes, including more test-time reasoning, larger parameter counts beyond about 70 billion, and medical fine-tuning, does not consistently close the gap, so progress will require a different intervention.
  • LLM outputs on uncertain evidence are overconfident: models under-use the 'uncertain effect' answer and over-predict clear direction, which is dangerous in clinical settings where hedging is the correct expert behavior.
  • Because accuracy declines as input token length grows, performance depends on how evidence is packaged; full-text access helps the strongest models, while abstract-only deployments, common in practice, likely suppress accuracy.
  • The paper's expert-guided prompt, which explicitly asked models to grade study quality, did not remove the lack of scientific skepticism, implying that prompting alone is insufficient and models need mechanisms that actually weigh study quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: since the benchmark scores only the final five-way label, it does not measure LLM performance at earlier review stages such as screening or data extraction; a model could be useful there while still failing at conclusion-drawing, and the paper does not claim otherwise.
  • Beyond the paper: the source-concordance feature used to diagnose 'scientific skepticism' was computed by one of the evaluated models rather than by independent humans, so that specific curve should be re-measured before it is used as a stable diagnostic.
  • Beyond the paper: a natural extension is to couple the benchmark with a retrieval step, since the paper deliberately isolates synthesis from search; adding retrieval would show whether the failure sits in reasoning over provided evidence or in finding the right evidence.
  • Beyond the paper: because performance degrades with context length, agentic systems that iteratively select and read many sources may need explicit evidence-budgeting or hierarchical summarization rather than larger context windows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MedEvidence, a benchmark of 284 questions derived from 100 Cochrane systematic reviews, pairing each question with the source studies cited by the review authors and a five-class expert-derived answer (higher, lower, no difference, uncertain effect, insufficient data). The authors evaluate 24 LLMs in a zero-shot setting with exact-match accuracy, zero temperature, bootstrapped confidence intervals, and both a basic and an expert-guided prompt. They report that frontier models such as DeepSeek V3 and GPT-4.1 reach only about 62% and 60% accuracy, and they analyze failure modes including long-context degradation, overconfidence, and a lack of scientific skepticism toward low-quality evidence. The central claim is that, given the same studies used by human experts, even frontier LLMs fail to match systematic review conclusions in at least 37% of cases.

Significance. If the benchmark's assumptions hold, MedEvidence is a useful and much-needed resource: it provides expert-grounded, closed-form evaluation for multi-document medical evidence synthesis, with 284 questions, 329 source articles, metadata on evidence certainty and full-text availability, and a public codebase and dataset. The evaluation is methodical in several respects: zero-temperature inference, exact-match scoring with valid-output checks, bootstrap confidence intervals, two prompting regimes, and a broad model spread across size, reasoning ability, and medical fine-tuning. The headline finding that current LLMs are far from saturating the benchmark would be important for the many deployed systems that already use LLMs for medical evidence synthesis. However, the significance depends on the strength of the 'same studies' guarantee and the quality of the human-curated labels; both require the additional validation requested in the major comments.

major comments (3)
  1. [§3, 'Question feasibility validation' (Step 4)] The benchmark admits a question when at least 75% of the meta-analysis weight comes from 'valid' studies, which means that for up to 25% of the evidence weight used by the Cochrane authors, the supplied sources may lack the numerical data needed to reproduce the expert conclusion. Sections 6 and 7 state that models are evaluated 'given the same studies' and that frontier LLMs fail in at least 37% of evaluated cases, but the actual guarantee is only a 75%-by-weight guarantee. The manuscript does not report the distribution of valid-weight fractions across the 284 questions, nor accuracy stratified by that fraction, so the magnitude of this missing-information confound is unmeasured. Please report such stratified results (e.g., by bins of valid-study weight) and, if the effect is material, temper the 'same studies' wording in the abstract and conclusion.
  2. [§3, 'Conclusion to question conversion' (Step 2)] The ground-truth labels are produced by human annotators, but no inter-annotator agreement measure is reported. The five answer classes include subtle boundaries (notably uncertain effect versus no difference versus insufficient data), and the per-class recall analysis in Figure 4(b) and Appendix H depends on the reliability of these conversions. Without agreement statistics (e.g., Cohen's kappa on a double-annotated subset), label noise is unquantified. Please report agreement statistics and the adjudication procedure.
  3. [§3, 'LLM-assisted metadata' / Figure 5(b)] The source-concordance feature that underlies the scientific-skepticism finding in Figure 5(b) is computed by DeepSeek V3, one of the models under evaluation, with no validation against human labels. This introduces a circular dependency for that specific analysis: the claim that models lack skepticism toward low-quality findings is partly based on classifications generated by the same type of model being assessed. Please validate DeepSeek V3's single-source classifications against human annotations on a random sample, or replace this feature with an external gold standard for the analysis in Figure 5(b).
minor comments (6)
  1. [§3, paragraph on LLM-assisted metadata] The sentence 'we use an LLMs to assess the percentage' contains a typo and should read 'we use an LLM to assess'.
  2. [Appendix Table 3] GPT-o1 is listed as 'Generalist Non-Reasoning' but is a reasoning model; this appears to be an error in the model-type column.
  3. [§6, 'Model performance decreases as token length increases' and Appendix G] The main text states that performance consistently declines with token length, but Appendix Figure 15 shows that models over 100B parameters do not exhibit this decline; please qualify this claim in the main text to match the appendix.
  4. [Appendix O.1] The remarks contain small typos ('reconcilliation', 'prevous') that should be corrected.
  5. [§5.1, 'LLM evaluation'] Exact-match evaluation counts invalid outputs as incorrect, which conflates instruction-following failures with reasoning failures; the paper does report valid-output rates, but consider also reporting accuracy conditioned on valid output in the main figure for clarity.
  6. [Figure 11(a) and §3, evidence certainty extraction] The paper says certainty is extracted 'only when it was explicitly provided by the original SR authors,' but 20.8% of questions have N/A; please clarify whether N/A means the SR did not provide a certainty rating or the annotators did not extract it.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline accuracy numbers are checked against independent Cochrane-derived labels; auxiliary LLM-generated metadata and a BIOMEDICA self-citation are not load-bearing.

full rationale

The central claim—that frontier LLMs fail to match systematic-review conclusions in at least 37% of evaluated cases—is an empirical benchmark result, not a derivation from the models' own outputs. Ground-truth answers are human-curated from Cochrane 'Main Results' sections using a fixed taxonomy, and accuracy is computed by exact match against those labels (Section 5.1). No parameter is fitted to the benchmark, and no equation links the accuracy metric to the input sources by construction. The auxiliary metadata generated by DeepSeek V3 (source concordance and medical specialty tags) are used only for stratified analysis, not for defining correctness; even if those LLM-generated labels are noisy, they do not feed into the accuracy scores that support the main conclusion. The reuse of the authors' BIOMEDICA dataset for full-text retrieval is a data-provenance choice, not an unverified self-citation carrying a load-bearing argument. The 75%-by-weight question-feasibility threshold is an inclusion criterion rather than a circular reduction; it may weaken the strength of the 'same studies' claim, but that is a validity concern, not circularity. No uniqueness theorem, ansatz, or known result is smuggled in via self-citation. The paper's central evaluation is therefore self-contained with respect to external ground truth, and any circularity is at most minor and non-load-bearing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central accuracy numbers are direct exact-match comparisons to SR-derived labels, not fitted outputs. However, dataset inclusion thresholds and LLM-generated metadata are hand-chosen and influence the results.

free parameters (2)
  • valid_study_weight_threshold = 75%
    Hand-set threshold for considering a question answerable from the included studies; affects which 284 questions enter the benchmark.
  • max_included_references = 12
    Reviews with more than 12 references were excluded to reduce annotator burden, changing the distribution of benchmark difficulty.
assumptions (4)
  • domain assumption Cochrane systematic review conclusions are a valid gold standard for treatment effect direction.
    The benchmark treats the direction-of-effect statements in Cochrane abstracts as correct answers; the reliability of Cochrane methodology is cited but not independently verified in this paper.
  • domain assumption The five-class answer taxonomy captures the conclusions of Cochrane SRs without meaningful loss.
    Annotators discard ambiguous conclusions and force statements into higher, lower, no difference, uncertain effect, or insufficient data; this may lose nuance.
  • ad hoc to paper DeepSeek V3 single-source classifications can serve as source-level agreement labels.
    Source concordance, used to show models lack skepticism, is generated by prompting DeepSeek V3 to answer with each single source; no validation of these labels against human judgment is reported.
  • domain assumption Exact-match scoring on the answer field measures LLM ability, with invalid outputs counted as incorrect.
    The paper also reports valid-output rates, but the headline accuracy mixes format compliance with reasoning ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Match the Conclusions of Systematic Reviews?." pith.science (2026). https://pith.science/paper/MVSLUBYH

@misc{pith2026250522787,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Match the Conclusions of Systematic Reviews?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVSLUBYH}},
  note         = {Machine review of arXiv:2505.22787}
}
read the original abstract

Systematic reviews (SR), in which experts summarize and analyze evidence across individual studies to provide insights on a specialized topic, are a cornerstone for evidence-based clinical decision-making, research, and policy. Given the exponential growth of scientific articles, there is growing interest in using large language models (LLMs) to automate SR generation. However, the ability of LLMs to critically assess evidence and reason across multiple documents to provide recommendations at the same proficiency as domain experts remains poorly characterized. We therefore ask: Can LLMs match the conclusions of systematic reviews written by clinical experts when given access to the same studies? To explore this question, we present MedEvidence, a benchmark pairing findings from 100 SRs with the studies they are based on. We benchmark 24 LLMs on MedEvidence, including reasoning, non-reasoning, medical specialist, and models across varying sizes (from 7B-700B). Through our systematic evaluation, we find that reasoning does not necessarily improve performance, larger models do not consistently yield greater gains, and knowledge-based fine-tuning degrades accuracy on MedEvidence. Instead, most models exhibit similar behavior: performance tends to degrade as token length increases, their responses show overconfidence, and, contrary to human experts, all models show a lack of scientific skepticism toward low-quality findings. These results suggest that more work is still required before LLMs can reliably match the observations from expert-conducted SRs, even though these systems are already deployed and being used by clinicians. We release our codebase and benchmark to the broader research community to further investigate LLM-based SR systems.

Figures

Figures reproduced from arXiv: 2505.22787 by the authors.

Figure 1
Figure 1. Core skills evaluated by MedEvidence including: medical domain expertise across 10 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the dataset curation process for MedEvidence. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Key statistical characteristics of the questions in MedEvidence. (a) shows the dataset [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (42 more)
Figure 4
Figure 4. Figure 4: (a) Average model accuracy (and 95% CI) on MedEvidence, overlaid on the percentage [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Medically-finetuned models vs their base generalist counterparts. Pairs of medical and base models are adjacent. 95% confi￾dence intervals are calculated via bootstrap￾ping with N = 1000. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Average model accuracy as a function of model size. We observe diminishing returns beyond [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: MedEvidence cohort diagram describing selection criteria for Cochrane SRs suitable for [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 10
Figure 10. Figure 10: An example meta-analysis from a Cochrane review (figure from DOI: [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Additional statistical characteristics of MedEvidence. (a) shows the dataset distribution [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Prompt used to generate LLM responses to questions under the basic prompt setup. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Prompts used to generate LLM responses to questions under the expert-guided prompt [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Model accuracy as a function of number of relevant sources. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Model performance as a function of the number of tokens in the relevant studies, separated [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Average confusion matrices using basic prompts. (a) Average confusion matrix aggregated [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Accuracy by publication year uncertain effect no difference insufficient data lower higher Correct Answer 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy OpenBioLLM 8B Llama 4 Scout Llama 3.0 8B Qwen2.5-7B-Instruct HuatuoGPT-o1-7B OpenBioLLM 70B HuatuoGPT-o1-70B Llama 3.1 8B Llama 3…
Figure 18
Figure 18. Figure 18: Per-class recall for each individual model. 95% confidence intervals are calculated via [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Model accuracy at different evidence qualities when using the expert-guided prompt setup. [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Changes in model performance when using the basic prompt setup versus the expert [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Analyses of model behavior across questions. (a) Questions (columns) that were deemed [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: Average model accuracy across all models (and 95% confidence interval) stratified by [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: We find that most models with the poorest overall performance actually experience a [PITH_FULL_IMAGE:figures/full_fig_p026_23.png]
Figure 24
Figure 24. Figure 24: Confusion matrix for DeepSeek R1. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 33 1 6 0 1 2 2 37 4 1 2 4 6 8 48 6 7 4 3 1 4 30 0 0 5 7 15 2 8 3 0 10 20 30 40 [PIT…
Figure 25
Figure 25. Figure 25: Confusion matrix for DeepSeek V3. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 36 0 6 0 1 0 2 38 7 1 1 1 11 15 49 4 0 0 3 2 8 25 0 0 4 11 18 3 3 1 0 10 20 30 40 […
Figure 26
Figure 26. Figure 26: Confusion matrix for GPT-4.1. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_26.png]
Figure 27
Figure 27. Figure 27: Confusion matrix for GPT-4.1 mini. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 24 0 4 0 4 11 0 30 8 0 2 10 3 5 41 1 5 24 0 2 4 19 0 13 4 5 11 0 3 17 0 5 10 15 20 …
Figure 28
Figure 28. Figure 28: Confusion matrix for GPT-o1. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 21 2 4 0 0 16 1 25 5 0 1 18 6 11 25 1 0 36 0 5 3 11 0 19 3 6 9 1 0 21 0 5 10 15 20 25 30 …
Figure 29
Figure 29. Figure 29: Confusion matrix for HuatuoGPT-o1-70B. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_29.png]
Figure 30
Figure 30. Figure 30: Confusion matrix for HuatuoGPT-o1-7B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 22 4 4 0 0 13 1 32 3 1 0 13 6 12 27 0 0 34 2 2 2 11 0 21 3 6 9 0 0 22 0 5 10 15 …
Figure 31
Figure 31. Figure 31: Confusion matrix for Llama 3.0 70B [PITH_FULL_IMAGE:figures/full_fig_p036_31.png]
Figure 32
Figure 32. Figure 32: Confusion matrix for Llama 3.0 8B. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_32.png]
Figure 33
Figure 33. Figure 33: Confusion matrix for Llama 3.1 405B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 35 3 5 0 0 0 3 40 5 2 0 0 12 21 39 7 0 0 6 6 4 22 0 0 8 12 16 2 1 1 0 5 10 15 20 …
Figure 34
Figure 34. Figure 34: Confusion matrix for Llama 3.1 70B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 16 7 4 0 0 16 1 26 3 0 0 20 3 25 28 8 0 15 3 4 4 14 0 13 3 12 12 2 0 11 0 5 10 15 …
Figure 35
Figure 35. Figure 35: Confusion matrix for Llama 3.1 8B. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_35.png]
Figure 36
Figure 36. Figure 36: Confusion matrix for Llama 3.3 70B (R1-Distill). [PITH_FULL_IMAGE:figures/full_fig_p038_36.png]
Figure 37
Figure 37. Figure 37: Confusion matrix for Llama 3.3 70B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p038_37.png]
Figure 38
Figure 38. Figure 38: Confusion matrix for Llama 4 Maverick. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_38.png]
Figure 39
Figure 39. Figure 39: Confusion matrix for Llama 4 Scout. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 20 3 3 0 0 17 4 25 2 0 0 19 9 7 22 0 0 41 9 2 2 2 0 23 2 6 9 0 0 23 0 5 10 15 20 2…
Figure 40
Figure 40. Figure 40: Confusion matrix for OpenBioLLM 70B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 0 0 0 0 0 43 0 0 0 0 0 50 0 0 0 0 0 79 0 0 0 0 0 38 0 0 0 0 0 40 0 10 20 30 40 50…
Figure 41
Figure 41. Figure 41: Confusion matrix for OpenBioLLM 8B. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_41.png]
Figure 42
Figure 42. Figure 42: Confusion matrix for OpenThinker2-32B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 31 1 6 0 2 3 0 37 5 1 2 5 7 13 44 3 8 4 2 2 3 23 1 7 2 10 18 0 6 4 0 5 10 15 20…
Figure 43
Figure 43. Figure 43: Confusion matrix for QwQ-32B. higher lower no difference insufficient data uncertain effect invalid Correct higher lower no difference insufficient data uncertain effect Predicted 29 2 5 1 2 4 2 34 7 1 1 5 11 12 42 6 4 4 2 3 5 21 0 7 4 14 14 1 4 3 0 5 10 15 20 25 30 3…
Figure 44
Figure 44. Figure 44: Confusion matrix for Qwen2.5-32B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p040_44.png]
Figure 45
Figure 45. Figure 45: Confusion matrix for Qwen2.5-72B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p041_45.png]
Figure 46
Figure 46. Figure 46: Confusion matrix for Qwen2.5-7B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p041_46.png]
Figure 47
Figure 47. Figure 47: Confusion matrix for Qwen3-235B-A22B-FP8. [PITH_FULL_IMAGE:figures/full_fig_p041_47.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 36 canonical work pages

  1. [1]

    Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases

    Lutz Bornmann, Robin Haunschild, and R \"u diger Mutz. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications , 8(1):224, 2021

    work page 2021

  2. [2]

    Alejandro Lozano, Scott L Fleming, Chia-Chun Chiang, and Nigam Shah. Clinfo. ai: An open-source retrieval-augmented large language model system for answering medical questions using scientific literature. In PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024 , pages 8--23. World Scientific, 2023

    work page 2024

  3. [3]

    The emergence of large language models (llm) as a tool in literature reviews: an llm automated systematic review

    Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, and Leslie A Lenert. The emergence of large language models (llm) as a tool in literature reviews: an llm automated systematic review. arXiv preprint arXiv:2409.04600 , 2024

    arXiv 2024

  4. [4]

    How to optimize the systematic review process using ai tools

    Nicholas Fabiano, Arnav Gupta, Nishaant Bhambra, Brandon Luu, Stanley Wong, Muhammad Maaz, Jess G Fiedorowicz, Andrew L Smith, and Marco Solmi. How to optimize the systematic review process using ai tools. JCPP advances , 4(2):e12234, 2024

    work page 2024

  5. [5]

    Future of evidence synthesis: Automated, living, and interactive systematic reviews and meta-analyses

    Irbaz Bin Riaz, Syed Arsalan Ahmed Naqvi, Bashar Hasan, and Mohammad Hassan Murad. Future of evidence synthesis: Automated, living, and interactive systematic reviews and meta-analyses. Mayo Clinic Proceedings: Digital Health , 2(3):361--365, 2024

    work page 2024

  6. [6]

    Deep research system card, 2025

    OpenAI. Deep research system card, 2025. Accessed: 2025-05-15

    work page 2025

  7. [7]

    Gemini deep research – your personal research assistant, 2025

    Google. Gemini deep research – your personal research assistant, 2025. Accessed: 2025-05-15

    work page 2025

  8. [8]

    Elicit: The ai research assistant, 2025

    Elicit. Elicit: The ai research assistant, 2025. Accessed: 2025-05-15

    work page 2025

Show all 55 references
  1. [9]

    Open evidence: Ai-powered medical information platform, 2025

    OpenEvidence. Open evidence: Ai-powered medical information platform, 2025. Accessed: 2025-05-15

    2025

  2. [10]

    Food and Drug Administration

    U.S. Food and Drug Administration. Fda announces completion of first ai-assisted scientific review pilot and aggressive agency-wide ai rollout timeline, May 2025. FDA News Release

    2025

  3. [11]

    Development and testing of retrieval augmented generation in large language models--a case study report

    YuHe Ke, Liyuan Jin, Kabilan Elangovan, Hairil Rizal Abdullah, Nan Liu, Alex Tiong Heng Sia, Chai Rick Soh, Joshua Yi Min Tung, Jasmine Chiat Ling Ong, and Daniel Shu Wei Ting. Development and testing of retrieval augmented generation in large language models--a case study rep...

    2024 arXiv

  4. [12]

    Can large language models reason about medical questions? Patterns , 5(3), 2024

    Valentin Li \'e vin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns , 5(3), 2024

    2024

  5. [13]

    Medalign: A clinician-generated dataset for instruction following with electronic medical records

    Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. Medalign: A clinician-generated dataset for instruction following with electronic medical records. In...

    2024

  6. [14]

    Artificial intelligence to automate network meta-analyses: Four case studies to evaluate the potential application of large language models

    Tim Reason, Emma Benbow, Julia Langham, Andy Gimblett, Sven L Klijn, and Bill Malcolm. Artificial intelligence to automate network meta-analyses: Four case studies to evaluate the potential application of large language models. Pharmacoecon Open , 8(2):205--220, Mar 2024

    2024

  7. [15]

    Applications of the natural language processing tool chatgpt in clinical practice: Comparative study and augmented systematic review

    Nikolas Schopow, Georg Osterhoff, and David Baur. Applications of the natural language processing tool chatgpt in clinical practice: Comparative study and augmented systematic review. JMIR Med Inform , 11:e48933, Nov 2023

    2023

  8. [16]

    Riaz Qureshi, Daniel Shaughnessy, Kayden A. R. Gill, Karen A. Robinson, Tianjing Li, and Eitan Agai. Are chatgpt and large language models ``the answer''to bringing us closer to systematic review automation? Systematic Reviews , 12(1):72, 2023

    2023

  9. [17]

    Assessing the risk of bias in randomized clinical trials with large language models

    Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao, Xiaoman Wang, Ming Liu, Jhalok Ronjan Talukdar, Jinhui Tian, Kehu Yang, and Janne Estill. Assessing the risk of bias in randomized c...

    2024

  10. [18]

    Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature

    Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, et al. Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific li...

    2025 arXiv

  11. [19]

    o ws, Maria-Inti Metzendorf, Felix Heilmeyer, Waldemar Siemens, Christian Haverkamp, Daniel B \

    Judith-Lisa Lieberum, Markus T \"o ws, Maria-Inti Metzendorf, Felix Heilmeyer, Waldemar Siemens, Christian Haverkamp, Daniel B \"o hringer, Joerg J. Meerpohl, and Angelika Eisele-Metzger. Large language models for conducting systematic reviews: on the rise, but not yet ready f...

    2025

  12. [20]

    Generative artificial intelligence use in evidence synthesis: A systematic review

    Justin Clark, Belinda Barton, Loai Albarqouni, Oyungerel Byambasuren, Tanisha Jowsey, Justin Keogh, Tian Liang, Christian Moro, Hayley O’Neill, and Mark Jones. Generative artificial intelligence use in evidence synthesis: A systematic review. Research Synthesis Methods , page ...

    2025

  13. [21]

    M ed REQAL : Examining medical knowledge recall of large language models via question answering

    Juraj Vladika, Phillip Schneider, and Florian Matthes. M ed REQAL : Examining medical knowledge recall of large language models via question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 202...

    2024

  14. [22]

    H ealth FC : Verifying health claims with evidence-based medical fact-checking

    Juraj Vladika, Phillip Schneider, and Florian Matthes. H ealth FC : Verifying health claims with evidence-based medical fact-checking. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Join...

    2024

  15. [23]

    What evidence do language models find convincing?, 2024

    Alexander Wan, Eric Wallace, and Dan Klein. What evidence do language models find convincing?, 2024

    2024

  16. [24]

    Clasheval: Quantifying the tug-of-war between an llm's internal prior and external evidence, 2025

    Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an llm's internal prior and external evidence, 2025

    2025

  17. [25]

    Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm, 2024

    Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm, 2024

    2024

  18. [26]

    Untangle the knot: Interweaving conflicting knowledge and reasoning skills in large language models, 2024

    Yantao Liu, Zijun Yao, Xin Lv, Yuchen Fan, Shulin Cao, Jifan Yu, Lei Hou, and Juanzi Li. Untangle the knot: Interweaving conflicting knowledge and reasoning skills in large language models, 2024

    2024

  19. [27]

    How to write a cochrane systematic review

    Lorna K Henderson, Jonathan C Craig, Narelle S Willis, David Tovey, and Angela C Webster. How to write a cochrane systematic review. Nephrology (Carlton) , 15(6):617--624, Sep 2010

    2010

  20. [28]

    Quality of cochrane reviews

    Mark Petticrew, Paul Wilson, Kath Wright, and Fujian Song. Quality of cochrane reviews. quality of cochrane reviews is better than that of non-cochrane reviews. BMJ , 324(7336):545, Mar 2002

    2002

  21. [29]

    What is a cochrane review? Epidemiol Psychiatr Sci , 20(3):231--233, Sep 2011

    A Cipriani, T A Furukawa, and C Barbui. What is a cochrane review? Epidemiol Psychiatr Sci , 20(3):231--233, Sep 2011

    2011

  22. [30]

    Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature, 2025

    Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, Anita Rau, Xiaohan Wang, Yuhui Zhang, Alfred Seunghoon Song, Robert Tibshirani, and Serena Yeung-Levy. Biomedica: An open ...

    2025

  23. [31]

    Bethesda (MD): National Center for Biotechnology Information (US), 2010-

    Entrez Programming Utilities Help [Internet] . Bethesda (MD): National Center for Biotechnology Information (US), 2010-

    2010

  24. [32]

    Search strategy used to create the pubmed systematic reviews filter, 2019

    2019

  25. [33]

    Assessment of the strength of recommendation and quality of evidence: Grade checklist

    Camila Torres Bezerra, Antonio Jos \'e Grande, Vivianny Kelly Galv \ a o, Douglas Henrique Marin dos Santos, \'A lvaro Nagib Atallah, and Valter Silva. Assessment of the strength of recommendation and quality of evidence: Grade checklist. a descriptive study. Sao Paulo Medical...

    2022

  26. [34]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024

    2024

  27. [35]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  28. [36]

    Open Thoughts

    OpenThoughts Team. Open Thoughts . https://open-thoughts.ai, January 2025

    2025

  29. [37]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  30. [38]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025

    2025

  31. [39]

    The llama 4 herd, 2025

    AI@Meta. The llama 4 herd, 2025

    2025

  32. [40]

    Huatuogpt-o1, towards medical complex reasoning with llms, 2024

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms, 2024

    2024

  33. [41]

    Openbiollms: Advancing open-source large language models for healthcare and life sciences

    Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B, 2024

    2024

  34. [42]

    Refinedocumentschain

    LangChain. Refinedocumentschain. Accessed: 2025-05-16

    2025

  35. [43]

    An introduction to the bootstrap

    Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap . Chapman and Hall/CRC, 1994

    1994

  36. [44]

    Long context is not long at all: A prospector of long-dependency data for large language models

    Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2405.17915 , 2024

    2024 arXiv

  37. [45]

    Long-context llms struggle with long in-context learning

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. URL https://arxiv. org/abs/2404.02060 , 2024

    2024 arXiv

  38. [46]

    Large language models are overconfident and amplify human bias

    Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. Large language models are overconfident and amplify human bias. arXiv preprint arXiv:2505.02151 , 2025

    2025

  39. [47]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063 , 2023

    2023 arXiv

  40. [48]

    Taming overconfidence in llms: Reward calibration in rlhf

    Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in llms: Reward calibration in rlhf. arXiv preprint arXiv:2410.09724 , 2024

    2024 arXiv

  41. [49]

    Fine-tuning is fine, if calibrated

    Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, et al. Fine-tuning is fine, if calibrated. Advances in Neural Information Processing Systems , 37:136084--136119, 2024

    2024

  42. [50]

    Calibrated language model fine-tuning for in-and out-of-distribution data

    Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. Calibrated language model fine-tuning for in-and out-of-distribution data. arXiv preprint arXiv:2010.11506 , 2020

    2010 arXiv

  43. [51]

    Finetunebench: How well do commercial fine-tuning apis infuse knowledge into llms? arXiv preprint arXiv:2411.05059 , 2024

    Eric Wu, Kevin Wu, and James Zou. Finetunebench: How well do commercial fine-tuning apis infuse knowledge into llms? arXiv preprint arXiv:2411.05059 , 2024

    2024 arXiv

  44. [52]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025

    2025

  45. [53]

    The llama 3 herd of models, 2024

    AI@Meta. The llama 3 herd of models, 2024

    2024

  46. [54]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025

    2025

  47. [55]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.