REVIEW 3 major objections 5 minor 17 references
Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper maps 129 NLP studies onto the five steps of evidence-based medicine, showing where the field is concentrated and where it is thin.
desk verdict Useful 5A-organized scoping review of NLP for EBM, but the PRISMA arithmetic for the headline 129-study count does not add up as printed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The organizing device is the 5A framework of evidence-based medicine (Ask, Acquire, Appraise, Apply, Assess), used as a fixed taxonomy to classify each of the 129 studies. Within that taxonomy, the PICO schema (Patient/Population, Intervention, Comparison, Outcome) is the central object that most extraction, retrieval, and synthesis work is built around; the paper treats PICO extraction and normalization as the canonical NLP task for EBM. The screening machinery is the PRISMA flow, which provides the reproducibility of the corpus, and the paper's supplementary metadata tables attach every study to a model type, a disease, and an EBM task.
What would settle it
Re-run the same five-step mapping with a broader search that adds databases such as Embase and Web of Science and terms like 'systematic review automation', 'trial matching', or 'risk of bias', and count how many additional evidence-synthesis, appraisal, and QA papers surface. If the distribution of tasks shifts materially—or if many of the 129 papers are reclassified under a different annotator—the paper's claim that extraction and evaluation dominate would be weakened.
Extended reading notes
Core claim
The paper's discovery is a taxonomy, not a new algorithm. By organizing 129 studies according to the 5A's, it shows that NLP-for-EBM research is concentrated in the early and late parts of the pipeline—searching and retrieving literature (Acquire) and applying evidence in the clinic (Apply/Assess)—with entity extraction and classification as the most frequent task across all years, peaking with the 2023 wave of large-language-model papers. It further claims that quality assessment, evidence ranking, evidence synthesis, and summarization remain thinner, and that existing benchmarks for synthesis and appraisal are rarely medical-specific. The implied finding is that the middle of the EBM workflow—appraising and synthesizing evidence—is the least automated and the most open for NLP development, even though LLMs have begun to be tested there.
Load-bearing premise
The review's map of the field is only as complete as its keyword-based search; the assumption is that requiring NLP terms and EBM terms to co-occur in titles or abstracts, searching four databases, and restricting to English-language work from 2019–2024 captured the relevant literature without systematic bias.
Editorial extensions
If this is right
- If the map is right, new NLP research for EBM should focus on evidence synthesis, appraisal, and question answering, since those steps are the least covered and LLM evaluation is still early.
- The 5A taxonomy gives a common vocabulary for comparing future systems: a new tool can be placed at a specific step and benchmarked against the corpora the review catalogues (EBM-NLP, Chia, PICO-Corpus, MS^2, Trialstreamer).
- The shift toward LLMs documented in 2022–2024 papers implies that clinicians can expect more conversational, QA-style EBM tools, but the review's challenges section says these need retrieval augmentation and source attribution before clinical use.
- The scarcity of medical-specific benchmarks for synthesis and appraisal means that progress in those steps will require dataset construction as much as model innovation.
Reading between the lines
- A testable extension the paper does not run: use the same 5A map to classify LLM-era papers published after 2024 and see whether the QA and synthesis categories grow faster than extraction—if they do, the 'under-served middle' claim would need updating within a year or two.
- The review's exclusion of non-English studies likely understates NLP-for-EBM activity in health systems where English is not the clinical language; a parallel scoping review in Chinese, Spanish, or German clinical NLP would be a natural complement.
- The emphasis on scarcity of medical-specific benchmarks suggests an opportunity: adapting existing general summarization benchmarks like CNN-DailyMail—which the paper flags as non-medical—into clinically validated test sets could shift evaluations toward the steps the field currently neglects.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This scoping review surveys NLP applications in evidence-based medicine (EBM), following PRISMA reporting. The authors report 129 included studies, organize them by the five EBM steps (Ask, Acquire, Appraise, Apply, Assess), review NLP techniques per step, summarize benchmark datasets, and discuss limitations and future directions. The manuscript includes a PRISMA flow diagram, detailed supplementary tables of included studies and benchmarks, explicit search queries, and a public GitHub repository.
Significance. If the numbers are corrected, this would be a useful structured map of a fast-moving interdisciplinary area. The paper's strengths are its systematic selection pipeline, the explicit and reproducible search strategy, the public release of study metadata, and the organization around the 5A EBM framework. The census of 129 studies, however, is the backbone of the review, so the internal arithmetic of the PRISMA flow and the presence of duplicate entries are consequential rather than cosmetic issues. The benchmark overview and task-trend summary are genuinely useful resources for researchers entering this area.
major comments (3)
- [Section 2.4 and Figure 1] The PRISMA flow arithmetic is internally inconsistent. The text states that the pool was "601 papers retrieved from databases and 9 additional sources" and that "8 duplicates" were removed; 601 + 9 - 8 = 602, yet Figure 1 reports 603 studies screened. Figure 1 also lists only 7 duplicate removals (6 manual + 1 Covidence) in the "References removed" box, not 8. Because the final count is obtained as screened (603) - 386 - 88 = 129, an alternative reading in which screened should be 593 would yield 119 included studies. The authors should report per-database hit counts, list the 9 additional sources, and provide a single consistent duplicate-removal count so that the headline number 129 is reproducible.
- [Supplementary Table 1 and reference list] The entries Kim et al. (2023a) and Kim et al. (2023b) refer to the same paper: the reference list contains two identical full citations (same title, venue, and page range), and Supplementary Table 1 lists them as two different included studies with different model/disease annotations. This duplication implies that the unique-study count is at most 128. The table should be de-duplicated, and the task-distribution statistics in Figure 2 should be recomputed accordingly.
- [Section 2.3] The screening process is described as having two annotators cross-verify study selection and metadata extraction with third-party arbitration, but no inter-annotator agreement measure is reported. Because the 129-study corpus and all downstream trends depend on the consistency of inclusion and exclusion decisions, reporting Cohen's kappa or a comparable statistic for the title/abstract and full-text screens is necessary for readers to assess the reliability of the corpus.
minor comments (5)
- [Appendix A.3] The search string line "{ebm_keywords)" has a mismatched closing parenthesis; the prose says the Boolean strategy combines any NLP term with any EBM term, but the displayed queries combine the full keyword groups with AND, not individual terms.
- [Supplementary Table 1 legend] The task legend uses the code "S" for both Abstractive Summarization and Extractive Summarization, which makes it impossible to tell which task is assigned to a given paper. A distinct code for extractive summarization would remove the ambiguity.
- [Table 1] The row label "Apply & Asses" contains a typo and should read "Apply & Assess."
- [Section 5.1] The phrase "domain-aware pertaining benefits" should be "domain-aware pre-training benefits."
- [Figure 2] The task names in Figure 2 (e.g., "Evaluation", "Summarization") are not fully aligned with the task codes in Supplementary Table 1 (e.g., "Evaluation of Performance", two separate S codes); a supplementary table of per-year counts per task would make the trends easier to verify.
Circularity Check
No significant circularity: the 129-study census and task mapping are self-contained; self-citations are corpus entries, not load-bearing premises.
full rationale
This is a scoping review, not a derivation. The central claim is a census: 129 studies matching explicit inclusion criteria (Sections 2.2 and 2.3). That count is produced by screening and metadata extraction, not by any equation whose output is assumed in its input. The five-step EBM taxonomy (Table 1) is an organizational scheme applied after selection; the finding that entity extraction/classification and evaluation dominate while evidence synthesis and appraisal are under-served is an empirical summary of the selected corpus, not a consequence of the search strategy. Self-citations such as Kang et al. (2023), Peng et al. (2023), and Zhang et al. (2024a,b) appear as included-study records or contextual examples in Supplementary Table 1 and the narrative; they are not invoked as uniqueness theorems, nor does the review rely on them to justify corpus completeness or to force any conclusion. The PRISMA arithmetic discrepancy (Section 2.4 states 601 + 9 sources and 8 duplicates removed, whereas Figure 1 reports 603 screened and 6+1 duplicates) is an internal consistency and reproducibility concern, not circularity, because the 129 figure is not the premise of an argument that reduces to itself. Similarly, the duplicated Kim et al. (2023a,b) rows in Supplementary Table 1 would affect the unique-study count but do not constitute a circular step. No fitted quantity is renamed as a prediction, and no derivation chain returns to its own assumptions.
Assumptions & free parameters
assumptions (3)
- domain assumption The search strategy (NLP keywords AND EBM keywords restricted to title/abstract) with the stated four databases captures the relevant 2019-2024 literature.
- domain assumption The two-annotator screening with third-party arbitration assigns the 129 papers to EBM tasks and metadata consistently.
- domain assumption Exclusion of non-English publications, secondary literature, and pre-2019 work does not remove a substantial share of relevant primary NLP-for-EBM research.
Cite this review
Pith. "Pith review of Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review." pith.science (2026). https://pith.science/paper/IK7ZQEMX
@misc{pith2026250522280,
author = {Pith},
title = {Pith review of: Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review},
year = {2026},
howpublished = {\url{https://pith.science/paper/IK7ZQEMX}},
note = {Machine review of arXiv:2505.22280}
}
read the original abstract
Evidence-based medicine (EBM) is at the forefront of modern healthcare, emphasizing the use of the best available scientific evidence to guide clinical decisions. Due to the sheer volume and rapid growth of medical literature and the high cost of curation, there is a critical need to investigate Natural Language Processing (NLP) methods to identify, appraise, synthesize, summarize, and disseminate evidence in EBM. This survey presents an in-depth review of 129 research studies on leveraging NLP for EBM, illustrating its pivotal role in enhancing clinical decision-making processes. The paper systematically explores how NLP supports the five fundamental steps of EBM -- Ask, Acquire, Appraise, Apply, and Assess. The review not only identifies current limitations within the field but also proposes directions for future research, emphasizing the potential for NLP to revolutionize EBM by refining evidence extraction, evidence synthesis, appraisal, summarization, enhancing data comprehensibility, and facilitating a more efficient clinical workflow.
Figures
Reference graph
Works this paper leans on
-
[3]
A probabilistic precision information retrieval model for personalized clinical trial recommendation based on heterogeneous data. In 2021 12th Inter- national Conference on Computing Communication and Networking Technologies (ICCCNT), pages 1–5. IEEE. Lara J Kanbar, Benjamin Wissel, Yizhao Ni, Nathan Pajor, Tracy Glauser, John Pestian, and Judith W Dexhei...
work page 2021
-
[5]
Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yencken
American Medical Informatics Association. Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yencken. 2024. Nicta-piboso dataset. https: //doi.org/10.57702/ne4r48m1. Dataset consists of 1,000 medical abstracts manually annotated with semantic tags based on the PICO criteria to support the automatic classification of sentences. Bevan Koopman, Tracey Wr...
-
[12]
In Proceedings of the 5th International Con- ference on Medical and Health Informatics , pages 1–6
Evaluation of applied machine learning for health misinformation detection via survey of med- ical professionals on controversial topics in pedi- atrics. In Proceedings of the 5th International Con- ference on Medical and Health Informatics , pages 1–6. ACM. Olivia Sanchez-Graillet, Christian Witte, Frank Grimm, and Philipp Cimiano. 2022. An annotated cor...
arXiv 2022
-
[13]
Transformer-based named entity recognition for parsing clinical trial eligibility criteria. ACM BCB, 2021. Shubo Tian, Pengfei Yin, Hansi Zhang, Arslan Erden- gasileng, Jiang Bian, and Zhe He. 2023. Parsing clinical trial eligibility criteria for cohort query by a multi-input multi-output sequence labeling model. In 2023 IEEE International Conference on B...
work page 2021
-
[14]
Hybrid bag of approaches to characterize se- lection criteria for cohort identification. J. Am. Med. Inform. Assoc., 26(11):1172–1180. Kunyuan Wang, Hao Cui, Yun Zhu, Xiaoyun Hu, Chang Hong, Yabing Guo, Lingyao An, Qi Zhang, and Li Liu. 2024. Evaluation of an artificial intelligence- based clinical trial matching system in chinese pa- tients with hepatoce...
work page 2024
-
[15]
Towards automated detection of contradictory research claims in medical literature using deep learn- ing approach. In 2021 Fifth International Conference on Information Retrieval and Knowledge Manage- ment (CAMP), pages 116–121. Jiayi Yuan, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu
work page 2021
-
[16]
In AMIA Annual Symposium Proceedings , volume 2024, pages 1324–1333
Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings , volume 2024, pages 1324–1333. AMIA. Kun Zeng, Zhiwei Pan, Yibin Xu, and Yingying Qu
work page 2024
-
[17]
An ensemble learning strategy for eligibility criteria text classification for clinical trial recruitment: Algorithm development and validation. JMIR Medi- cal Informatics, 8(7):e17832. Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jor- dan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas R. Campion Jr, Zhiy...
work page 2019
Show all 17 references
-
[624]
Jeongeun Kim, Mitchell Izower, and Yuri Quintana
American Medical Informatics Association. Jeongeun Kim, Mitchell Izower, and Yuri Quintana. 2023b. Parsable clinical trial eligibility criteria rep- resentation using natural language processing. In AMIA Annual Symposium Proceedings, pages 616–
-
[1996]
BMJ, 312(7023):71–72
Evidence based medicine: what it is and what it isn’t. BMJ, 312(7023):71–72. Jawad Sadek, Alex Inskip, James Woltmann, Georgina Wilkins, Christopher Marshall, Maria Pokora, Amey Vedpathak, Anastasija Jadrevska, Dawn Craig, and Michael Trenell. 2023. Scanmedicine: An online sea...
2023
-
[2018]
In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 197–207, Melbourne, Australia
A corpus with multi-level annotations of pa- tients, interventions and outcomes to support lan- guage processing for medical literature. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 197–207, Melbour...
2021
-
[2019]
Journal of the American Medical Informatics Association, 26(11):1333–1343
Dquest: dynamic questionnaire for search of clinical trials. Journal of the American Medical Informatics Association, 26(11):1333–1343. Hao Liu, Yuan Chi, Alex Butler, Yingcheng Sun, and Chunhua Weng. 2021. A knowledge base of clini- cal trial eligibility criteria. Journal of ...
2021
-
[2020]
Bioinformatics, 36(4):1234–1240
BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. Kyeryoung Lee, Zongzhi Liu, Yun Mai, Tomi Jun, Meng Ma, Tongyu Wang, Lei Ai, Ediz Calay, William Oh, Gustavo Stolovitzky, Eric Schadt, and Xiaoyan Wang
-
[2021]
COVID-19 trial graph: a linked graph for COVID-19 clinical trials. J. Am. Med. Inform. As- soc., 28(9):1964–1969. Abdelazeem Eldawlatly, Hussain Alshehri, Abdullah Alqahtani, Abdulaziz Ahmad, Fatma Al-Dammas, and Amir Marzouk. 2018. Appearance of popula- tion, intervention, co...
1964
-
[2022]
JMIR Medical Informatics, 10(12):e40743
State-of-the-art evidence retriever for preci- sion medicine: Algorithm development and valida- tion. JMIR Medical Informatics, 10(12):e40743. Qiao Jin, Zifeng Wang, Charalampos S Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken- Clarke, Elisabetta Xue, Yifan Yang, Jimeng S...
2024
-
[2023]
In 2023 IEEE International Conference on Big Data (BigData), volume 4, pages 4940–4947
Integrating a PICO clinical questioning to the QL4POMR framework for building evidence-based clinical case reports. In 2023 IEEE International Conference on Big Data (BigData), volume 4, pages 4940–4947. IEEE. Victor M Murcia, Vinod Aggarwal, Nikhil Pesaladinne, Ram Thammineni...
2023
-
[2024]
JMIR AI, 3:e50800
Optimizing clinical trial eligibility design using natural language processing models and real- world data: Algorithm development and validation. JMIR AI, 3:e50800. Chao Li, Harsha Gurulingappa, Prathamesh Karmalkar, Jana Raab, Aastha Vij, Gerard Megaro, and Christian Henke. 2...
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.