Pith. sign in

REVIEW 4 major objections 5 minor 71 references

LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a zero-shot, two-stage ranker—LLM graded relevance scoring followed by dense re-ranking—outperforms existing QA-based abstract screening methods by 5–10 percentage points in mean average precision and even beats a…

desk verdict Useful dataset and a sensible two-stage pipeline, but the headline 5-10 pp claim only holds when LGAR gets research questions that the QA baseline doesn't; on the controlled comparison the margin is 2-3 pp and on DTA it disappears entirely. read the letter →

arxiv 2505.24757 v2 pith:RDFVGYIT submitted 2025-05-30 cs.CL

Christian Jaumann , Andreas Wiedholz , Annemarie Friedrich This is my paper
classification cs.CL
keywords systematicliteraturereviewabstractscreeningzero-shotrankinglargelanguagemodelsgradedrelevancescoringdensere-rankingmeanaverageprecisionresearchquestionextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Systematic literature reviews require screening thousands of abstracts against detailed inclusion and exclusion criteria. This paper proposes LGAR, a zero-shot two-stage ranker in which a large language model first gives every abstract a graded relevance score on a 0–19 scale using the review's title, research questions, and full criteria, and a neural re-ranker (monoT5) then orders the abstracts that received the same score. The paper claims this pipeline beats the existing question-answering-based LLM ranker by 5–10 percentage points in mean average precision on the SYNERGY and TAR2019 benchmarks, and that in zero-shot mode it even outperforms a biomedical model fine-tuned for the screening task. To make such comparisons realistic, the authors manually extracted and validated the complete inclusion/exclusion criteria and research questions for 57 reviews, since existing datasets lack them. If the claims hold, screening teams can use a fully zero-shot, open-weights pipeline that produces a ranked list instead of a binary decision, reducing human workload without task-specific training.

What carries the argument

The machinery is the two-stage LGAR pipeline. The first stage is a prompted LLM used as a graded relevance scorer: the system message gives the review title and research questions and defines relevance as satisfying all inclusion criteria and no exclusion criteria, while the user message presents the paper title and abstract and asks for a single number on a configurable scale, with 0–19 found optimal. The second stage is monoT5, a dense neural re-ranker (a sequence-to-sequence transformer fine-tuned on MS MARCO) that takes the review title plus paper title and abstract as input and orders the papers inside each equal-score group, producing a complete ranking. The third supporting piece is the dataset extension: manually extracted, validated inclusion/exclusion criteria and research questions for 57 SLRs, which the paper argues are needed to evaluate LLM-based rankers on the same information a human screener would use.

What would settle it

Run a controlled comparison in which the QA baseline receives the identical prompt content as LGAR—same research questions, same criteria, same scale instructions—and in which LGAR is evaluated with only the review title (no research questions). If the QA baseline with research questions matches or beats LGAR, or if LGAR without research questions loses its advantage, the claimed architectural superiority is not supported. A second decisive check is to evaluate LGAR on systematic reviews published after the LLM training cutoff to rule out data contamination.

Watch

Extended reading notes

Core claim

The central claim is that LGAR, composed of an LLM-based graded relevance scorer and a dense re-ranker, is the best zero-shot abstract-ranking method for systematic literature reviews. In the first stage, the LLM is prompted with the review title, research questions, and the full inclusion/exclusion criteria and assigns each abstract a score on a Likert scale that the paper finds works best at 0–19; the study reports that this coarse scale outperforms the 0–4 scale recommended for general ranking tasks. In the second stage, monoT5, a cross-encoder trained on MS MARCO, re-ranks the abstracts within each score group using the review title as query, which mainly improves workload savings rather than MAP. On SYNERGY and TAR2019, LGAR reaches 40.7 and 50.6 mean average precision with title-plus-research-questions prompts, outperforming the replicated QA-based system by roughly 5–10 points, and on the Intervention and DTA subsets it outperforms a fine-tuned BioBERT ranker by 7–10 points that the paper reports. The paper also contributes a validated resource: exhaustive criteria and research questions for 57 SLRs, with a second annotator reproducing 23 of 23 research questions and 97 of 99 criteria on a 15-review validation sample.

Load-bearing premise

The result that LGAR beats the QA baseline by 5–10 points in mean average precision rests on the assumption that both systems received exactly the same information about each review; if the baseline did not also get the research questions, the gap could come from extra information rather than from LGAR's architecture.

Editorial extensions

If this is right

  • Abstract screening can be run zero-shot with open-weights LLMs and dense rankers, producing a ranked list rather than a binary label, so the human can choose any cutoff.
  • At matched recall, LGAR places relevant abstracts higher than the QA-based baseline, so screeners scan fewer irrelevant papers before reaching the same coverage.
  • The dense re-ranker's main measurable benefit is workload savings: it raises TNR@95% by roughly 7–12 points by recovering relevant papers the LLM scored too low.
  • Practitioners should use coarser graded scales (around 0–19) for abstract screening, unlike the 0–4 scale reported optimal for general ranking tasks.
  • Adding research questions to the prompt improves ranking, and the released criteria/research-question annotations let future work be evaluated under the same realistic information conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 0–19 finding departs from the 0–4 optimum for general ranking, the screening task may have a long tail of borderline relevance judgments; a testable extension is whether calibrating scores or asking the LLM for per-criterion confidence improves ranking further.
  • The paper's ablation implies the LLM stage should be tuned toward recall, deliberately keeping borderline papers in higher groups and letting monoT5 settle their order; this is an inference from the workload-savings results, not a claim the paper tests.
  • The manual extraction of research questions and criteria is the main bottleneck for applying LGAR to a new review; if the performance gain from research questions is real, automating their extraction from review protocols is a natural next step the paper does not address.
  • The paper acknowledges possible data contamination because older SLRs may appear in LLM training data; a clean test is to run LGAR on SLRs published after the model's training cutoff and check whether the MAP advantage persists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LGAR, a zero-shot two-stage abstract-ranking method for systematic literature reviews. In the first stage an instruction-following LLM assigns each candidate paper a graded relevance score on a Likert scale (0-19 in the main experiments); in the second stage a dense ranker (monoT5, title-only query) re-ranks the papers that receive the same LLM score. The authors also contribute a manually curated extension of the SYNERGY and CLEF TAR2019 datasets with inclusion/exclusion criteria and research questions for 57 SLRs, with an inter-annotator validation on 15 SLRs. Experiments compare LGAR with BM25, monoT5, a replication of the QA-based system of Akinseloyin et al. (2024), and a fine-tuned BioBERT ranker. The abstract and Section 5.4 claim that LGAR outperforms existing QA-based methods by 5-10 percentage points in MAP, and Section 5.5.4 claims a 7-10 pp advantage over a fine-tuned model. Ablations cover LLM choice, scale size, dense ranker choice, CoT prompting, self-consistency, and 2-shot variants.

Significance. The dataset extension is a genuine contribution: providing exhaustively extracted criteria and research questions for SYNERGY and TAR2019 enables more realistic and reproducible evaluation of LLM-based screening, and the authors are careful to tune their method on a separate dataset (Guo et al., 2023) rather than on the test sets. The paper also has practical value in comparing open-weights LLMs and zero-shot dense rankers in a controlled setting, and the code and data are promised publicly. If the headline performance claims were fully supported, LGAR would be a strong new zero-shot baseline for abstract screening. However, the controlled comparisons in the paper support a smaller and subset-dependent advantage, so the significance as currently stated is overstated.

major comments (4)
  1. [Abstract; Section 5.4, Table 2] The abstract and Section 5.4 claim that LGAR outperforms the QA-based method of Akinseloyin et al. (2024) by 5-10 pp. MAP. This margin is achieved only by LGAR (T+R, monoT5), whose prompt includes the SLR's research questions, while the QA baseline in Section 5.3 is described as receiving selection criteria only. The †-marked rows, which supply both systems with the same SLR and paper information, give margins of 2.8 pp. on SYNERGY (36.8 vs. 34.0) and 3.3 pp. on TAR2019 (48.4 vs. 45.1). The headline should be reworded to report the controlled, same-information comparison, or at least to state the 5-10 pp figure as the 'topline with research questions' rather than as the general advantage over the QA baseline.
  2. [Section 5.5.4, Table 4] The claim that zero-shot LGAR outperforms the fine-tuned BioBERT_ft by 7-10 pp. MAP is not supported by the DTA subset: LGAR (T, monoT5) obtains 34.5 vs. 35.4 for BioBERT_ft, a 0.9 pp. deficit, and LGAR (T+R, monoT5) obtains 38.1, a 2.7 pp. advantage. The 7-10 pp margin is entirely due to the Intervention split. The text in Section 5.5.4 should be qualified to report the subset-dependent results rather than presenting the Intervention margin as the overall outcome.
  3. [Table 7 (per-subset results)] The per-subset results in Appendix A.2 show that the advantage over the QA baseline is not consistent. On DTA, LGAR (T, monoT5) is 1.4 pp. below Akinseloyin et al. (ours) (34.5 vs. 35.9), and even LGAR (T+R, monoT5) leads by only 2.2 pp. The Qualitative and Prognosis subsets contain only 2 and 1 SLRs respectively, so any conclusions there are fragile. The paper should report these subset breakdowns in the main text and temper statements such as 'LGAR performs more robustly' and 'outperforms ... by a large margin'.
  4. [Section 5.4; Figure 6] The main comparative claims are not accompanied by any significance testing or confidence intervals. Figure 6 shows that MAP distributions overlap substantially across SLRs, and the reported margins are macro-averages over only 26 (SYNERGY) or 31 (TAR2019) SLRs. A paired permutation test or Wilcoxon signed-rank test over SLRs should be added to support the claims of superiority and robustness, especially since some subset-level comparisons go in the opposite direction.
minor comments (5)
  1. [Section 3.2] 'to an criterion' should read 'to a criterion'.
  2. [Section 5.4] 'negliglible' is a typo for 'negligible'.
  3. [Figure 4 caption] 'SNYERGY' should be 'SYNERGY'.
  4. [Table 10 caption] 'MS MACRO' should be 'MS MARCO'.
  5. [Figure 2] The prompt text contains irregular spacing, e.g., 'Y ou' and 'T ask', which should be cleaned up for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical comparisons are independently benchmarked.

full rationale

This is an empirical systems paper whose central claims are benchmark results rather than derivations from assumptions. LGAR's design choices (relevance scale 0-19, monoT5 as dense re-ranker, title-only query) were selected on a separate tuning dataset (Guo et al., 2023) and then applied unchanged to SYNERGY and TAR2019, with the selection documented in Appendix A.1 (Tables 5-6). The comparison against the QA baseline includes a dagger-marked condition in which LGAR (T, monoT5) and the replicated Akinseloyin baseline receive exactly the same SLR and paper information, so the claim that the architecture itself contributes is supported by a controlled experiment rather than by construction. No parameter is fitted to the test labels, no load-bearing claim is justified solely by a self-citation, and no known empirical pattern is merely renamed. The abstract's '5-10 pp' margin and Section 5.5.4's '7-10 pp' margin rely on less tightly controlled comparisons (LGAR with research questions versus a baseline without them, and favorable subset selection), which may overstate the controlled margins, but that is an experimental-claims issue rather than circularity. Honest non-finding: no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system rests on standard empirical assumptions in IR evaluation: proxy labels, annotation quality, and transfer of pretrained models. The most consequential is the comparison fairness assumption captured in the weakest_assumption field. No new entities are introduced.

free parameters (4)
  • relevance_scale_upper_bound = 19
    Selected via tuning on the Guo et al. (2023) dataset; 0-14 and 0-19 were nearly tied in MAP, and 0-19 was chosen to give the LLM more degrees of freedom (Appendix A.1).
  • reranker = monoT5 (3B)
    Selected via tuning on the Guo et al. dataset by comparing ColBERT, monoBERT, and monoT5 (Appendix A.1, Table 6).
  • reranker_query = title only
    monoT5 with title-only query outperformed title+research-questions on the Guo et al. tuning set on workload-saving metrics (Appendix A.1).
  • llm_backbone = Llama3.3-70B-Instruct
    Chosen as one of the strongest open-weights models at time of writing; robustness across model families is tested in Section 5.5.1.
assumptions (4)
  • domain assumption Relevance labels from full-text screening are a valid proxy for abstract-screening relevance
    Datasets provide labels from final inclusion decisions but only titles and abstracts as input; the authors acknowledge this approximation in the Limitations section.
  • domain assumption Manually extracted criteria and research questions accurately reflect each SLR's screening protocol
    Extraction was performed by the first author and validated on 15 of 57 SLRs by the second author with high agreement (Section 3.2); the remaining 42 SLRs were not independently validated.
  • domain assumption Pretrained dense rankers trained on MS MARCO transfer to biomedical abstract ranking
    monoT5 was selected on the tuning set (Guo et al.) and then applied without any domain-specific fine-tuning.
  • domain assumption LLM outputs on these SLRs are not substantially inflated by data contamination
    The authors acknowledge the possibility of contamination since many SLRs were published years ago and may appear in training data; no mitigation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews." pith.science (2026). https://pith.science/paper/RDFVGYIT

@misc{pith2026250524757,
  author       = {Pith},
  title        = {Pith review of: LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDFVGYIT}},
  note         = {Machine review of arXiv:2505.24757}
}
read the original abstract

The scientific literature is growing rapidly, making it hard to keep track of the state-of-the-art. Systematic literature reviews (SLRs) aim to identify and evaluate all relevant papers on a topic. After retrieving a set of candidate papers, the abstract screening phase determines initial relevance. To date, abstract screening methods using large language models (LLMs) focus on binary classification settings; existing question answering (QA) based ranking approaches suffer from error propagation. LLMs offer a unique opportunity to evaluate the SLR's inclusion and exclusion criteria, yet, existing benchmarks do not provide them exhaustively. We manually extract these criteria as well as research questions for 57 SLRs, mostly in the medical domain, enabling principled comparisons between approaches. Moreover, we propose LGAR, a zero-shot LLM Guided Abstract Ranker composed of an LLM based graded relevance scorer and a dense re-ranker. Our extensive experiments show that LGAR outperforms existing QA-based methods by 5-10 pp. in mean average precision. Our code and data is publicly available.

Figures

Figures reproduced from arXiv: 2505.24757 by the authors.

Figure 1
Figure 1. LGAR method for abstract screening con￾sisting of a 2-stage ranking approach: (1) the LLM ranker makes use of a realistic specification of the SLR to assign high-level graded relevance scores, (2) dense re-rankers produce a fully ordered list in scalable time. approaches focus on facilitating the selection of relevant articles, one of the most time-consuming steps (van Dinter et al., 2021). Large language models (LL… view at source ↗
Figure 2
Figure 2. Zero-shot prompts used in LGAR. Output Parsing. To make the responses as de￾terministic and reproducible as possible, we use a temperature of 0. The scores are extracted using regular expression and we verify that they are in the desired relevance scale. If score extraction fails, and when using self-consistency, the temperature is set to 0.5 to encourage more diverse results. In the rare case that the LLM does not … view at source ↗
Figure 3
Figure 3. Comparison of scales in terms of MAP and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average number of different relevance scores [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Boxplots of MAP scores of LGAR (T+R, monoT5) and Akinseloyin et al. (2024) (ours) on both datasets. of Wang et al. (2023c). We replicate the model to report all metrics in the exact same setting. Our replicated model slightly outperforms the results re￾ported by Wang e…
Figure 7
Figure 7. Figure 7: Distribution of relevance scores averaged for [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages

  1. [1]

    Opeoluwa Akinseloyin, Xiaorui Jiang, and Vasile Palade. 2024. https://doi.org/10.1093/jamia/ocae166 A question-answering framework for automated abstract screening using large language models . Journal of the American Medical Informatics Association, page ocae166

    work page doi:10.1093/jamia/ocae166 2024

  2. [2]

    Amal Alharbi, William Briggs, and Mark Stevenson. 2018. https://ceur-ws.org/Vol-2125/paper\_169.pdf Retrieving and ranking studies for systematic reviews: University of sheffield's approach to CLEF ehealth 2018 task 2 . In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018 , volume 2125 of CEUR...

    work page 2018

  3. [3]

    Vlahavas

    Antonios Anagnostou, Athanasios Lagopoulos, Grigorios Tsoumakas, and Ioannis P. Vlahavas. 2017. https://ceur-ws.org/Vol-1866/paper\_89.pdf Combining inter-review learning-to-rank and intra-review incremental training for title and abstract screening in systematic reviews . In Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin...

    work page 2017

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  5. [5]

    Hurdle, and Siddhartha Jonnalagadda

    Duy Duc An Bui, Guilherme Del Fiol, John F. Hurdle, and Siddhartha Jonnalagadda. 2016. https://doi.org/10.1016/J.JBI.2016.10.014 Extractive text summarization system to aid data extraction from full text in systematic review development . J. Biomed. Informatics, 64:265--272

    work page doi:10.1016/j.jbi.2016.10.014 2016

  6. [6]

    Andres Carvallo and Denis Parra. 2019. https://ceur-ws.org/Vol-2414/paper10.pdf Comparing word embeddings for document screening based on active learning . In Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SI...

    work page 2019

  7. [7]

    SH Cheng, C Augustin, A Bethel, D Gill, S Anzaroot, J Brun, B DeWilde, RC Minnich, R Garside, YJ Masuda, et al. 2018. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conservation Biology, 32(4):762--764

    work page 2018

  8. [8]

    Cohen, William R

    Aaron M. Cohen, William R. Hersh, K. Peterson, and Po - Yin Yen. 2006. https://doi.org/10.1197/JAMIA.M1929 Research paper: Reducing workload in systematic review preparation using automated citation classification . J. Am. Medical Informatics Assoc., 13(2):206--219

    doi:10.1197/jamia.m1929 2006

Show all 71 references
  1. [9]

    Jonathan De Bruin, Yongchao Ma, Gerbrich Ferdinands, Jelle Teijema, and Rens Van de Schoot. 2023. https://doi.org/10.34894/HE6NAQ SYNERGY - Open machine learning dataset on study selection in systematic reviews

    2023 doi

  2. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36

    2024

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. 2024. https://doi.org/10.48550/ARXIV.2407.21783 The Llama 3 Herd of Models . CoRR, abs/2407.21783

    2024 doi arXiv

  5. [13]

    Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. https://doi.org/10.1007/978-3-030-72240-1\_26 Rethink training of BERT rerankers in multi-stage retrieval pipeline . In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28...

    2021 doi

  6. [14]

    Vagrant Gautam, Miaoran Zhang, and Dietrich Klakow. 2023. https://openreview.net/forum?id=XySq36VD0U A lightweight method to generate unanswerable questions in english . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  7. [15]

    Eddie Guo, Mehul Gupta, Jiawen Deng, Ye - Jean Park, Mike Paget, and Christopher Naugler. 2023. https://doi.org/10.48550/ARXIV.2305.00844 Automated paper screening for clinical reviews using large language models . CoRR, abs/2305.00844

    2023 doi arXiv

  8. [16]

    Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Le Yan, and Yue Zhang. 2024. https://doi.org/10.48550/ARXIV.2404.11960 Generating diverse criteria on-the-fly to improve point-wise LLM rankers . CoRR, abs/2404.11960

    2024 doi arXiv

  9. [17]

    Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, and Xiangang Li. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.96 To answer or not to answer? improving machine reading comprehension model with span-based contrastive learning . In Findings of the Association for Com...

    2022 doi

  10. [18]

    Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. Clef 2019 technology assisted reviews in empirical medicine overview. CEUR Workshop Proceedings, 2380. 20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 ; Conference date: 09-09...

    2019

  11. [19]

    Elmagarmid, Ihab F

    Madian Khabsa, Ahmed K. Elmagarmid, Ihab F. Ilyas, Hossam Hammady, and Mourad Ouzzani. 2016. https://doi.org/10.1007/S10994-015-5535-7 Learning to identify relevant studies for systematic reviews using random forest and external information . Mach. Learn., 102(3):465--482

    2016 doi

  12. [20]

    Omar Khattab and Matei Zaharia. 2020. https://doi.org/10.1145/3397271.3401075 Colbert: Efficient and effective passage search via contextualized late interaction over BERT . In Proceedings of the 43rd International ACM SIGIR conference on research and development in Informatio...

    2020

  13. [21]

    Seunghee Kim and Jinwook Choi. 2014. https://doi.org/10.1016/J.JBI.2013.10.005 An svm-based high-quality article classifier for systematic reviews . J. Biomed. Informatics, 47:153--159

    2014 doi

  14. [22]

    Barbara Kitchenham and Pearl Brereton. 2013. https://doi.org/10.1016/j.infsof.2013.07.010 A systematic review of systematic review process research in software engineering . Information and Software Technology, 55(12):2049--2075

    2013 doi

  15. [23]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html Large language models are zero-shot reasoners . In Advances in Neural Informati...

    2022

  16. [24]

    Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. 2023. https://doi.org/10.1016/J.ISWA.2023.200193 An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews . Intell. Syst. Appl., 18:200193

    2023

  17. [25]

    Athanasios Lagopoulos, Antonios Anagnostou, Adamantios Minas, and Grigorios Tsoumakas. 2018. https://doi.org/10.1007/978-3-319-98932-7\_5 Learning-to-rank and relevance feedback for literature appraisal in empirical medicine . In Experimental IR Meets Multilinguality, Multimod...

    2018 doi

  18. [26]

    Grace Eunkyung Lee. 2017. https://ceur-ws.org/Vol-1866/paper\_88.pdf A study of convolutional neural networks for clinical document classification in systematic reviews: Sysreview at CLEF ehealth 2017 . In Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Foru...

    2017

  19. [27]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. https://doi.org/10.1093/BIOINFORMATICS/BTZ682 Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinform., 36(4):1234--1240

    2020 doi

  20. [28]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translatio...

    2020 doi

  21. [29]

    u ksekg \

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R \' e , Diana Acosta ...

    2023

  22. [30]

    Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O'Blenis. 2010. https://doi.org/10.1136/JAMIA.2010.004325 A new algorithm for reducing the workload of experts in performing systematic reviews . J. Am. Medical Informatics Assoc., 17(4):446--453

    2010 arXiv

  23. [31]

    Stan Matwin and Vera Sazonova. 2012. https://doi.org/10.1136/amiajnl-2012-001072 Direct comparison between support vector machine and multinomial naive bayes algorithms for medical abstract classification . Journal of the American Medical Informatics Association, 19(5):917--917

    2012 doi

  24. [32]

    Matthew Michelson and Katja Reuter. 2019. https://doi.org/10.1016/j.conctc.2019.100443 The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials . Contemporary Clinical Trials Commu...

    2019

  25. [33]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. https://arxiv.org/abs/1611.09268 MS MARCO: A human generated machine reading comprehension dataset . CoRR, abs/1611.09268

    2016 arXiv

  26. [34]

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.63 Document ranking with a pretrained sequence-to-sequence model . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708--718, Onl...

    2020 doi

  27. [35]

    Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. https://arxiv.org/abs/1901.04085 Passage re-ranking with BERT . CoRR, abs/1901.04085

    2019 arXiv

  28. [36]

    Rodrigo Frassetto Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. https://arxiv.org/abs/1910.14424 Multi-stage document ranking with BERT . CoRR, abs/1910.14424

    2019 arXiv

  29. [37]

    Eduardo A Oliveira, Maria Christina L Oliveira, Enrico A Colosimo, Daniella B Martelli, Ludmila R Silva, Ana Cristina Sim \ o es E Silva, and Herc \' lio Martelli-J \'u nior. 2022. Global scientific production in the pre-covid-19 era: An analysis of 53 countries for 22 years. ...

    2022

  30. [38]

    Mourad Ouzzani, Hossam Hammady, Zbys Fedorowicz, and Ahmed Elmagarmid. 2016. Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5:1--10

    2016

  31. [39]

    Antonia Panayi, Katherine Ward, Amir Benhadji-Schaff, A Santiago Ibanez-Lopez, Andrew Xia, and Regina Barzilay. 2023. https://doi.org/10.1186/s13643-023-02351-w Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews ....

    2023 doi

  32. [40]

    Piotr Przyby a, Austin J Brockmeier, Georgios Kontonatsios, Marie-Annick Le Pogam, John McNaught, Erik von Elm, Kay Nolan, and Sophia Ananiadou. 2018. Prioritising references for systematic reviews with robotanalyst: a user study. Research synthesis methods, 9(3):470--488

    2018

  33. [41]

    Xuan Qin, Jiali Liu, Yuning Wang, Yanmei Liu, Ke Deng, Yu Ma, Kang Zou, Ling Li, and Xin Sun. 2021. https://doi.org/10.1016/j.jclinepi.2021.01.010 Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews . Journ...

    2021 doi

  34. [42]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.97 Large language models are effective text rankers with pairwise...

    2024 doi

  35. [43]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: BM25 and beyond . Found. Trends Inf. Retr., 3(4):333--389

    2009 doi

  36. [44]

    Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, and Xingyi Song

    Ambrose Robinson, William Thorne, Ben P. Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, and Xingyi Song. 2023. https://doi.org/10.48550/ARXIV.2308.06610 Bio-sieve: Exploring instruction tuning large language models for systematic review automation . CoRR, abs/2308.06610

    2023 doi arXiv

  37. [45]

    Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen - tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/V1/2022.EMNLP-MAIN.249 Improving passage retrieval with zero-shot question generation . In Proceedings of the 2022 Conference o...

    2022 doi

  38. [46]

    Elias Sandner, Bing Hu, Alice Simiceanu, Luca Fontana, Igor Jakovljevic, Andre Henriques, Andreas Wagner, and Christian G \"u tl. 2024. https://api.semanticscholar.org/CorpusID:275956648 Screening automation for systematic reviews: A 5-tier prompting approach meeting cochrane’...

    2024

  39. [47]

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. https://doi.org/10.18653/v1/2022.naacl-main.272 C ol BERT v2: Effective and efficient retrieval via lightweight late interaction . In Proceedings of the 2022 Conference of the North Am...

    2022 doi

  40. [49]

    Harrisen Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020 b . https://doi.org/10.1145/3366423.3380185 Automatic boolean query formulation for systematic review literature search . In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020 , pages 1071--1...

    2020

  41. [50]

    Harrisen Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020 c . https://doi.org/10.1007/978-3-030-45439-5\_26 A computational approach for objectively derived systematic review search strategies . In Advances in Information Retrieval - 42nd European Conference on IR R...

    2020 doi

  42. [51]

    Benjamin Schneider. 2023. Publications output: Us trends and international comparisons. science & engineering indicators 2024. nsb-2023-33. National Science Foundation

    2023

  43. [52]

    Masood Sujau, Masako Wada, Emilie Vallee, Natalie Hillis, and Teo Susnjak. 2025. https://doi.org/10.20944/preprints202501.1513.v1 Accelerating disease model parameter extraction: An llm-based ranking approach to select initial studies for literature review automation . Preprints

    2025

  44. [53]

    Weiwei Sun, Zheng Chen, Xinyu Ma, Lingyong Yan, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. https://doi.org/10.48550/ARXIV.2311.01555 Instruction distillation makes large language models efficient zero-shot rankers . CoRR, abs/2311.01555

    2023 doi arXiv

  45. [54]

    Eugene Syriani, Istvan David, and Gauransh Kumar. 2023. https://doi.org/10.48550/ARXIV.2307.06464 Assessing the ability of chatgpt to screen articles for systematic reviews . CoRR, abs/2307.06464

    2023 doi arXiv

  46. [55]

    Raphael Tang, Xinyu Crystina Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.129 Found in the middle: Permutation self-consistency improves listwise ranking in large language models . In Proceedings of the 2024 Conference of th...

    2024 doi

  47. [56]

    Viet thi Tran, Gerald Gartlehner, Sally Yaacoub, Isabelle Boutron, Lukas Schwingshackl, Julia Stadelmaier, Isolde Sommer, Farzaneh Aboulayeh, Sivem Afach, Joerg Meerpohl, Philippe Ravaud, Paris Est, and PhD Viet-Thi Tran MD. 2023. https://api.semanticscholar.org/CorpusID:26633...

    2023

  48. [57]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://doi.org/10.485...

    2023 doi arXiv

  49. [58]

    Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, and Daniel L. Ob...

    2021 doi

  50. [59]

    Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021. https://doi.org/10.1016/j.infsof.2021.106589 Automation of systematic literature reviews: A systematic literature review . Information and Software Technology, 136:106589

    2021

  51. [60]

    Wallace, Kevin Small, Carla E

    Byron C. Wallace, Kevin Small, Carla E. Brodley, Joseph Lau, and Thomas A. Trikalinos. 2012. https://doi.org/10.1145/2110363.2110464 Deploying an interactive machine learning system in an evidence-based practice center: abstrackr . In Proceedings of the 2nd ACM SIGHIT Internat...

    2012

  52. [61]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.642 Improving text embeddings with large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

    2024 doi

  53. [62]

    Shuai Wang, Harrisen Scells, Bevan Koopman, Martin Potthast, and Guido Zuccon. 2023 a . https://doi.org/10.1145/3624918.3625322 Generating natural language queries for more effective systematic review screening prioritisation . In Proceedings of the Annual International ACM SI...

    2023

  54. [63]

    Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023 b . https://doi.org/10.1145/3539618.3591703 Can chatgpt write a good boolean query for systematic review literature search? In Proceedings of the 46th International ACM SIGIR Conference on Research and Developm...

    2023

  55. [64]

    Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023 c . https://doi.org/10.1145/3572960.3572980 Neural rankers for effective screening prioritisation in medical systematic review literature search . In Proceedings of the 26th Australasian Document Computing Symp...

    2023

  56. [65]

    Shuai Wang, Harrisen Scells, Shengyao Zhuang, Martin Potthast, Bevan Koopman, and Guido Zuccon. 2024 b . https://doi.org/10.1007/978-3-031-56027-9_25 Zero-shot generative large language models for systematic review screening automation . In Advances in Information Retrieval, p...

    2024 doi

  57. [66]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 d . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confe...

    2023

  58. [67]

    Dingjun Wu, Jing Zhang, and Xinmei Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.408 Chain of thought prompting elicits knowledge augmentation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 6519--6534, Toronto, Canada. Association ...

    2023 doi

  59. [68]

    Kraft, and Tim Menzies

    Zhe Yu, Nicholas A. Kraft, and Tim Menzies. 2018. https://doi.org/10.1007/S10664-017-9587-0 Finding better active learners for faster literature reviews . Empir. Softw. Eng., 23(6):3161--3186

    2018 doi

  60. [69]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji - Rong Wen. 2023. https://doi.org/10.48550/ARXIV.2308.07107 Large language models for information retrieval: A survey . CoRR, abs/2308.07107

    2023 doi

  61. [70]

    Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.naacl-short.31 Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels . In Proceedings of the 2024 Conference ...

    2024 doi

  62. [71]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  63. [72]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.