Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Large language models streamline automated systematic review: A preliminary study

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Compared against a published review's reference standard, GPT-4 reaches 0.92 abstract-screening accuracy and 0.81 data-extraction accuracy, while Claude-3 leads in study design.

desk verdict The extraction benchmark is a real effort, but the screening claim is meaningless without a majority-class baseline and a possible pilot/test overlap threatens the headline extraction numbers. read the letter →

arxiv 2502.15702 v1 pith:EO3TGMLS submitted 2025-01-09 cs.IR cs.AIcs.CL

Xi Chen , Xue Zhang This is my paper · ORCID
classification cs.IRcs.AIcs.CL
keywords largelanguagemodelssystematicreviewGPT-4Claude-3Mistral8x7BdataextractionabstractscreeningPICOdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that modern large language models can carry out the main steps of a systematic review—study design, search-strategy formulation, literature screening, and data extraction—with enough accuracy to be useful assistants for researchers. Using a previously published systematic review as the reference standard, it tests GPT-4, Claude-3, and Mistral 8x7B on 20 included trials and 811 screened abstracts. GPT-4 achieves the highest accuracy in abstract screening (0.92) and data extraction (0.81), while Claude-3 scores best on PICO and eligibility design; the two outperform Mistral on most tasks. The authors conclude that LLMs are promising assistive tools for systematic reviews, with human oversight still required. If true, this would let researchers offload the slowest and most error-prone parts of evidence synthesis.

What carries the argument

The load-bearing object is a reference standard built from one previously published systematic review: its PICO (Population, Intervention, Comparison, Outcome) structure, eligibility criteria, search strategies, screening decisions, and extracted data are treated as ground truth, and LLM outputs are scored against them. Task-specific prompts are developed iteratively, with the screening and extraction prompts piloted on two randomized controlled trials; all models are run through application programming interfaces at temperature zero, and each task is repeated five times. This setup converts 'is the LLM doing systematic review work?' into a set of exact-match accuracy checks for screening, search, and extraction, plus human Likert-scale ratings for study design. The reference standard is what makes the cross-model comparison meaningful, and the pilot-development step is what makes the screening and extraction prompts fit the target format.

What would settle it

Check the 20 test articles against the two randomized controlled trials used to develop the screening and extraction prompts; if any overlap exists, those accuracies are inflated. A second check would be to compare a random sample of the reference standard's extracted data against the original primary papers to see whether gold-standard errors are biasing the scores.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a head-to-head performance ranking of three LLMs against a fixed reference standard drawn from an earlier systematic review on neuraxial versus general anaesthesia for hip-fracture surgery. In that test, GPT-4 significantly outperformed the other models in data extraction, with an overall accuracy of 0.81 across 1,120 data points (3,360 individual fields), versus Claude-3's 0.59 and Mistral's 0.46. GPT-4 also led abstract screening at 0.92, ahead of Mistral's 0.84 and Claude-3's 0.69, and full-text screening at 0.75. Claude-3 was best at the design stage, earning the highest overall PICO score (4.58/5) and eligibility-criteria score, while GPT-4 and Claude-3 tied on search-strategy accuracy (0.78 and 0.79). The paper's conclusion is that LLMs demonstrate potential for automating systematic-review tasks, with GPT-4 the strongest overall performer and the other two models competitive in specific stages.

Load-bearing premise

The results assume that the reference systematic review is an error-free gold standard and that the two randomized controlled trials used to develop the screening and extraction prompts were not among the 20 test articles; if either premise fails, the reported accuracies overstate real-world performance.

Editorial extensions

If this is right

  • At GPT-4's abstract-screening accuracy of 0.92, LLMs can serve as a first-pass triage step, reducing the number of abstracts human reviewers must read.
  • At GPT-4's data-extraction accuracy of 0.81, much of the structured extraction from included trials can be automated, with human reviewers checking the roughly one-in-five data points that are wrong.
  • Since no model wins every task, a practical automation pipeline would combine models: Claude-3 for protocol and PICO design and GPT-4 for screening and extraction.
  • Because full-text screening accuracy (0.75) is lower than abstract screening (0.92), the near-term automation gain is largest at the abstract stage, with full-text review still requiring more human attention.
  • Low coefficient-of-variation values for GPT-4 and Claude-3 across repeated runs mean their outputs are stable enough to be used in a reproducible workflow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the single-review test set makes the numeric accuracies preliminary; a multi-review benchmark across clinical areas would be needed to separate model capability from quirks of this one anaesthesia topic.
  • The gap between abstract screening (0.92) and full-text screening (0.75) suggests the binding constraint is long-document comprehension, so retrieval-augmented or chunked-context prompts may improve full-text accuracy faster than changing models.
  • The quantitative outcome extraction scores—GPT-4 0.80, Claude-3 0.51, Mistral 0.39—point to a specific weakness in linking numbers to outcomes and time points; testing targeted numeric-reasoning prompts would be a natural follow-up.
  • If these accuracies hold in broader tests, the economics of systematic reviews could shift: the rate-limiting step would become constructing a trustworthy reference standard, not screening or extraction themselves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper evaluates three large language models (GPT-4, Claude-3, and Mistral 8x7B) on four systematic review tasks: PICO study design formulation, eligibility criteria development, search strategy formulation, literature screening, and data extraction. Using a previously published systematic review as the reference standard, the authors report that Claude-3 performs best in PICO design, GPT-4 and Claude-3 are comparable in search strategies, GPT-4 has the highest abstract screening accuracy, and GPT-4 significantly outperforms the other models in data extraction (accuracy 0.81 vs 0.59 vs 0.46, P<0.001). The authors conclude that LLMs are promising assistive tools for systematic review automation.

Significance. If the results hold, this is a useful large-scale comparison: the data extraction evaluation spans 1,120 data points per model (10,080 manually verified fields), which is larger than most prior LLM extraction studies, and the workflow-level comparison across four SR tasks is informative. The study also provides reliability analyses (ICC, CV) and manually verified annotations. However, the main quantitative claims rest on two unresolved methodological issues: possible contamination of the test set by pilot articles used for prompt development, and an uninformative accuracy metric for a highly imbalanced screening task. These issues currently limit the practical conclusions that can be drawn.

major comments (4)
  1. [Methods, Prompt Development and Dataset] The manuscript does not state whether the two randomized controlled trials used to pilot the screening and extraction prompts were excluded from the 20-article test set. The extraction evaluation (1,120 data points per model) is the paper's headline result, so this omission is load-bearing. Please state explicitly whether the two pilot RCTs are part of the 20 test articles; if they are, repeat the extraction and screening evaluations with them removed and report whether the model ranking and significance values persist.
  2. [Literature screening, abstract screening results] With 29 inclusions among 811 abstracts, a trivial always-exclude classifier obtains 782/811 = 0.964 accuracy. All three reported abstract screening accuracies (GPT-4: 0.92, Mistral: 0.84, Claude-3: 0.69) are below this baseline, so accuracy alone does not support the claim that GPT-4 'achieved the highest accuracy' in abstract screening. Report sensitivity, specificity, and a class-balanced metric (e.g., balanced accuracy or Matthews correlation coefficient) for both screening phases.
  3. [Statistical analysis, study design and eligibility criteria] The PICO and eligibility criteria scores are based on three repeated assessments per model (n=3 per cell). The manuscript reports an overall performance P=0.002 for PICO and P=0.05 for eligibility criteria, but pairwise comparisons are only partially reported; for eligibility criteria the text claims 'Claude-3 outperformed' without presenting the post-hoc tests. With such small samples, the Kruskal-Wallis test with Bonferroni correction has low power, and the lack of effect sizes makes the practical significance unclear. Provide pairwise statistics for eligibility criteria and report effect sizes or confidence intervals.
  4. [Search strategies, Results and Discussion] Table 3 reports overall search-term accuracy of 0.78, 0.79, and 0.67 with P=0.096, yet the Discussion states that 'Claude-3 reaching as high as 90% accuracy and GPT-4 consistently achieving accuracy rates above 70%.' No 90% value appears in Table 3. Clarify whether 90% refers to one of the five repetitions and report the repetition-level results; as written, the Discussion overstates the aggregate results.
minor comments (5)
  1. [Data extraction] The text states 'The results are showed in Table 3 and Figure 6a-d' for data extraction, but the corresponding table is Table 4; fix the cross-reference.
  2. [Table 8] There are typos in Table 8: '037' should be '0.37', and the row 'Cerebral Vascular Accident' appears twice; the second instance should likely be a different outcome (e.g., Myocardial infarction).
  3. [Figure 3 caption] The Figure 3 caption refers to 'PICO design examples generated by the three LLMs', but the figure displays eligibility criteria; adjust the caption to match the content.
  4. [Throughout] The model name is written inconsistently as 'Mistral 8x7B', 'Mistral7× 8B', 'Mistral-7B× 8', and 'Mistral7×8b'; standardize to a single notation.
  5. [Abstract and Methods] The word 'Liker Scale' should be 'Likert Scale' throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the LLM outputs are compared against a fixed, externally published reference standard, and no parameter or prompt is fitted to the reported accuracies.

full rationale

The paper is an empirical benchmark, not a derivation. The reference standard consists of the authors' previously published systematic review's PICO framework, eligibility criteria, search results, screening decisions, and extracted data. The three LLMs are prompted with the research question and standard PICO or eligibility criteria, and their outputs are compared against that fixed reference standard by exact matching or Likert rating. No model parameter or prompt is fitted to the test-set accuracy numbers; the reported accuracies are simple ratios of correct matches to total items. The prior systematic review is a temporal, external artifact that does not depend on the LLM outputs, so using it as ground truth is not circular. The only adjacent concern is methodological: the paper does not state whether the two RCTs used to develop the screening and extraction prompts were excluded from the 20-article test set. That is a potential test-set contamination issue, but it is not a circularity of the kind where an output is equivalent to an input by construction or where a fitted parameter is renamed as a prediction. No specific circular step can be demonstrated from the text, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study introduces no fitted parameters or invented entities. It depends on the prior review as gold standard and on the assumption that exact-match scoring captures correctness. The main unstated cost is the quality of the reference standard and the exclusion of pilot articles from the test set.

assumptions (3)
  • domain assumption The previously published systematic review used as the reference standard is an accurate gold standard for PICO, eligibility, search, screening, and extraction.
    Methods, Dataset: the prior review's raw data, protocol, and extracted data are treated as ground truth; errors in this review would propagate to all reported accuracies.
  • domain assumption Exact match between LLM output and the reference standard is a valid measure of accuracy for search, screening, and extraction.
    Methods, LLMs' automation tasks: outputs are defined as accurate if identical to the reference; semantic equivalence or partial credit is not considered.
  • domain assumption A 5-point Likert scale scored by one investigator (the abstract says three) reliably measures the quality of PICO and eligibility criteria.
    Methods, Study Design: one investigator assessed the quality three times; no inter-rater reliability beyond repeated scoring by the same person is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large language models streamline automated systematic review: A preliminary study." pith.science (2026). https://pith.science/paper/EO3TGMLS

@misc{pith2026250215702,
  author       = {Pith},
  title        = {Pith review of: Large language models streamline automated systematic review: A preliminary study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EO3TGMLS}},
  note         = {Machine review of arXiv:2502.15702}
}
read the original abstract

Large Language Models (LLMs) have shown promise in natural language processing tasks, with the potential to automate systematic reviews. This study evaluates the performance of three state-of-the-art LLMs in conducting systematic review tasks. We assessed GPT-4, Claude-3, and Mistral 8x7B across four systematic review tasks: study design formulation, search strategy development, literature screening, and data extraction. Sourced from a previously published systematic review, we provided reference standard including standard PICO (Population, Intervention, Comparison, Outcome) design, standard eligibility criteria, and data from 20 reference literature. Three investigators evaluated the quality of study design and eligibility criteria using 5-point Liker Scale in terms of accuracy, integrity, relevance, consistency and overall performance. For other tasks, the output is defined as accurate if it is the same as the reference standard. Search strategy performance was evaluated through accuracy and retrieval efficacy. Screening accuracy was assessed for both abstracts screening and full texts screening. Data extraction accuracy was evaluated across 1,120 data points comprising 3,360 individual fields. Claude-3 demonstrated superior overall performance in PICO design. In search strategy formulation, GPT-4 and Claude-3 achieved comparable accuracy, outperforming Mistral. For abstract screening, GPT-4 achieved the highest accuracy, followed by Mistral and Claude-3. In data extraction, GPT-4 significantly outperformed other models. LLMs demonstrate potential for automating systematic review tasks, with GPT-4 showing superior performance in search strategy formulation, literature screening and data extraction. These capabilities make them promising assistive tools for researchers and warrant further development and validation in this field.

Figures

Figures reproduced from arXiv: 2502.15702 by the authors.

Figure 1
Figure 1. Design of the study [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Performance of LLMs generating PICO design [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 7
Figure 7. Performance of LLMs extracting study information [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figures from the paper (2 more)
Figure 8
Figure 8. Figure 8: Performance of LLMs extracting intervention information. a. accuracy of extracting answer in intervention information; b. accuracy of extracting quote in intervention information; c. accuracy of extracting location in intervention information. d. overall accuracy of ex…
Figure 9
Figure 9. Figure 9: Performance of LLMs extracting outcome information [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [1]

    On the Opportunities and Risks of Foundation Models for Natural Language Processing in Radiology

    Wiggins WF, Tejani AS. On the Opportunities and Risks of Foundation Models for Natural Language Processing in Radiology. Radiology: Artificial Intelligence. 2022;4(4):e220119. doi:10.1148/ryai.220119

    work page doi:10.1148/ryai.220119 2022

  2. [2]

    Extracting accurate materials data from research papers with conversational language models and prompt engineering

    Polak MP, Morgan D. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nat Commun. 2024;15(1):1569. doi:10.1038/s41467-024-45914-8

    doi:10.1038/s41467-024-45914-8 2024

  3. [3]

    Structured information extraction from scientific text with large language models

    Dagdelen J, Dunn A, Lee S, et al. Structured information extraction from scientific text with large language models. Nat Commun. 2024;15(1):1418. doi:10.1038/s41467- 024-45563-x

    doi:10.1038/s41467- 2024

  4. [4]

    Large language models in medicine

    Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930-1940. doi:10.1038/s41591- 023-02448-8

    doi:10.1038/s41591- 2023

  5. [5]

    Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions

    Cumpston M, Li T, Page MJ, et al. Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Editorial Unit, ed. Cochrane Database of Systematic Reviews. Published online October 3, 2019. doi:10.1002/14651858.ED000142

    work page doi:10.1002/14651858.ed000142 2019

  6. [6]

    Evidence based medicine: what it is and what it isn’t

    Waldegrave SRW. Evidence based medicine: what it is and what it isn’t

    work page

  7. [7]

    Systematic reviews, systematic error and the acquisition of clinical knowledge

    Mickenautsch S. Systematic reviews, systematic error and the acquisition of clinical knowledge. BMC Med Res Methodol. 2010;10(1):53. doi:10.1186/1471-2288-10-53

    work page doi:10.1186/1471-2288-10-53 2010

  8. [8]

    Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap

    Elliott JH, Turner T, Clavisi O, et al. Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap. PLoS Med. 2014;11(2):e1001603. doi:10.1371/journal.pmed.1001603

    doi:10.1371/journal.pmed.1001603 2014

Show all 48 references
  1. [9]

    Systematic review automation technologies

    Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3(1):74. doi:10.1186/2046-4053-3-74

    2014 doi

  2. [10]

    Toward systematic review automation: a practical guide to using machine learning tools in research synthesis

    Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8(1):163, s13643- 019-1074-1079. doi:10.1186/s13643-019-1074-9

    2019 doi

  3. [11]

    Using text mining for study identification in systematic reviews: a systematic review of current approaches

    O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5. doi:10.1186/2046-4053-4-5

    2015 doi

  4. [12]

    Data extraction methods for systematic review (semi)automation: Update of a living systematic review

    Schmidt L, Mutlu ANF, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. Published online October 9, 2023. doi:10.12688/f1000research.51117.2

    2023 doi

  5. [13]

    Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models

    Lai H, Ge L, Sun M, et al. Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Netw Open. 2024;7(5):e2412687. doi:10.1001/jamanetworkopen.2024.12687

    2024

  6. [14]

    Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review

    Feng L, Chiam YK, Lo SK. Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review. In: 2017 24th Asia-Pacific Software Engineering Conference (APSEC). ; 2017:41-50. doi:10.1109/APSEC.2017.10

    2017 doi

  7. [15]

    Applications of text mining within systematic reviews

    Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Syn Meth. 2011;2(1):1-14. doi:10.1002/jrsm.27

    2011 doi

  8. [16]

    Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

    Li M, Sun J, Tan X. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Syst Rev. 2024;13(1):219. doi:10.1186/s13643-024- 02609-x

    2024 doi

  9. [18]

    A question-answering framework for automated abstract screening using large language models

    Akinseloyin O, Jiang X, Palade V. A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association. 2024;31(9):1939-1952. doi:10.1093/jamia/ocae166

    2024 doi

  10. [19]

    the answer

    Qureshi R, Shaughnessy D, Gill KAR, Robinson KA, Li T, Agai E. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Syst Rev. 2023;12(1):72. doi:10.1186/s13643-023-02243-z

    2023 doi

  11. [20]

    GPT-4 Technical Report

    OpenAI, Achiam J, Adler S, et al. GPT-4 Technical Report. Published online March 4,

  12. [21]

    From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

    Vacareanu R, Negru VA, Suciu V, Surdeanu M. From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples. Published online September 10, 2024. Accessed September 14, 2024. http://arxiv.org/abs/2404.07544

    2024 arXiv

  13. [22]

    Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability Study With DocOA

    Chen X, Wang L, You M, Liu W, Fu Y, Xu J, Zhang S, Chen G, Li K, Li J. Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability Study With DocOA. J Med Internet Res. 2024 Jul 22;26:e58158. doi: 10.2196/58158. PMID: 3883...

    2024 doi

  14. [24]

    The Claude 3 Model Family: Opus, Sonnet, Haiku

    Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. 2024

    2024

  15. [25]

    Au Large | Mistral AI | Frontier AI in your hands

    AI M. Au Large | Mistral AI | Frontier AI in your hands. 2024

    2024

  16. [26]

    What are large language model settings: Temperature, top P and max tokens

    Novita. What are large language model settings: Temperature, top P and max tokens. Novita AI. https://blogs.novita.ai. Published 2023. Accessed October 14, 2024

    2023

  17. [27]

    A Review of Key Likert Scale Development Advances: 1995-

    Jebb AT, Ng V, Tay L. A Review of Key Likert Scale Development Advances: 1995-

    1995

  18. [28]

    Intraclass correlations: uses in assessing rater reliability

    Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979 Mar;86(2):420-8. doi: 10.1037//0033-2909.86.2.420. PMID: 18839484

    1979 doi

  19. [29]

    Quartile coefficient of variation is more robust than CV for traits calculated as a ratio

    Botta-Dukát Z. Quartile coefficient of variation is more robust than CV for traits calculated as a ratio. Sci Rep. 2023 Mar 22;13(1):4671. doi: 10.1038/s41598-023- 31711-8IF: 3.8 Q1 . PMID: 36949089; PMCID: PMC10033673

    2023 doi

  20. [30]

    Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

    Bernstein IA, Zhang Y (Victor), Govil D, et al. Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions. JAMA Netw Open. 2023;6(8):e2330320. doi:10.1001/jamanetworkopen.2023.30320

    2023

  21. [31]

    Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions

    Yalamanchili A, Sengupta B, Song J, et al. Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions. JAMA Netw Open. 2024;7(4):e244630. doi:10.1001/jamanetworkopen.2024.4630

    2024

  22. [32]

    Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study

    He Z, Bhasuran B, Jin Q, et al. Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study

  23. [33]

    Development of search strategies for systematic reviews in health using ChatGPT: a critical analysis

    Guimarães NS, Joviano-Santos JV, Reis MG, Chaves RRM, Observatory of Epidemiology, Nutrition, Health Research (OPENS). Development of search strategies for systematic reviews in health using ChatGPT: a critical analysis. J Transl Med. 2024;22(1):1. doi:10.1186/s12967-023-04371-5

    2024 doi

  24. [34]

    Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? Published online February 9, 2023

    Wang S, Scells H, Koopman B, Zuccon G. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? Published online February 9, 2023. Accessed November 9, 2024. http://arxiv.org/abs/2302.03495

    2023 arXiv

  25. [35]

    Potential roles of large language models in production of systematic reviews and meta-analyses

    Luo X, Chen F, Zhu D, et al. Potential roles of large language models in production of systematic reviews and meta-analyses

  26. [36]

    The role of ChatGPT in scientific communication: writing better scientific review articles

    Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles

  27. [37]

    The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation

    Gwon YN, Kim JH, Chung HS, et al. The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Med Inform. 2024;12:e51187-e51187. doi:10.2196/51187

    2024 doi

  28. [38]

    Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

    Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. J Med Internet Res. 2024;26:e48996. doi:10.2196/48996

    2024 doi

  29. [39]

    Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Research Synthesis Methods. 2024...

    2024 doi

  30. [40]

    Matsui K, Utsumi T, Aoki Y, Maruki T, Takeshima M, Takaesu Y. Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews. J Med Internet Res. 2024;2...

    2024 doi

  31. [41]

    Evaluating the Effectiveness of Large Language Models in Abstract Screening: A Comparative Analysis

    Li M, Hill UC. Evaluating the Effectiveness of Large Language Models in Abstract Screening: A Comparative Analysis

  32. [42]

    Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

    Dennstädt F, Zink J, Putora PM, Hastings J, Cihoric N. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Syst Rev. 2024;13(1):158. doi:10.1186/s13643-024-02575-4

    2024 doi

  33. [43]

    Performance of a Large Language Model in Screening Citations

    Oami T, Okada Y, Nakada T aki. Performance of a Large Language Model in Screening Citations. JAMA Netw Open. 2024;7(7):e2420496. doi:10.1001/jamanetworkopen.2024.20496

    2024

  34. [44]

    Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer

    Fink MA, Bischoff A, Fink CA, et al. Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer. Radiology. 2023;308(3):e231362. doi:10.1148/radiol.231362

    2023 doi

  35. [45]

    A Comparison of a Large Language Model vs Manual Chart Review for the Extraction of Data Elements From the Electronic Health Record

    Ge J, Li M, Delk MB, Lai JC. A Comparison of a Large Language Model vs Manual Chart Review for the Extraction of Data Elements From the Electronic Health Record. Gastroenterology. 2024;166(4):707-709.e3. doi:10.1053/j.gastro.2023.12.019

    2024 doi

  36. [46]

    Data extraction for evidence synthesis using a large language model: A proof-of-concept study

    Gartlehner G, Kahwati L, Hilscher R, et al. Data extraction for evidence synthesis using a large language model: A proof-of-concept study. Research Synthesis Methods. 2024;15(4):576-589. doi:10.1002/jrsm.1710

    2024 doi

  37. [47]

    Performance of two large language models for data extraction in evidence synthesis

    Konet A, Thomas I, Gartlehner G, et al. Performance of two large language models for data extraction in evidence synthesis. Research Synthesis Methods. 2024;15(5):818-824. doi:10.1002/jrsm.1732

    2024 doi

  38. [48]

    Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews

    Khan MA, Ayub U, Naqvi SAA, et al. Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews. Published online September 23, 2024. doi:10.1101/2024.09.20.24314108 Supplementary file 1. Quality assessment criteria for Study Design Metric Exp...

    2024 doi

  39. [2019]

    2021;12:637547

    Front Psychol. 2021;12:637547. doi:10.3389/fpsyg.2021.637547

    2021

  40. [2024]

    http://arxiv.org/abs/2303.08774

    Accessed September 14, 2024. http://arxiv.org/abs/2303.08774

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.