Pith. sign in

REVIEW 5 major objections 6 minor 59 references

A Reproducibility and Generalizability Study of Large Language Models for Query Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that Boolean queries generated by large language models for systematic reviews are not reproducible: rerunning identical prompts with different random seeds produces high-variance results, so the original reported scores…

desk verdict A useful replication-with-extension study; the seed-variance claim is real but under-supported because the reported standard deviations pool topic and seed variation rather than isolating within-topic seed instability. read the letter →

arxiv 2411.14914 v1 pith:BT5PVRRL submitted 2024-11-22 cs.IR

Moritz Staudinger , Wojciech Kusa , Florina Piroi , Aldo Lipani , Allan Hanbury This is my paper · ORCID
classification cs.IR
keywords systematicreviewsBooleanquerygenerationlargelanguagemodelsreproducibilityvariabilityretrievalevaluationPubMedopen-weightLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This study tests a specific promise of generative AI: that a chatbot can draft the Boolean search queries used in systematic literature reviews. The authors reran a previously published LLM query-generation setup across two standard test collections and seven models, including open-weight alternatives, repeating every prompt with multiple random seeds. Their central finding is that the results are neither reproducible nor reliable: scores fluctuate strongly when the same prompt is rerun with a different seed, and none of the models reliably recovered the original paper's effectiveness numbers. If this finding holds, out-of-the-box LLMs are not yet safe to trust as an automatic component of systematic-review automation, and any single-run evaluation of such queries should be regarded as unverified. The paper also documents malformed outputs and dataset problems that make the original comparisons difficult to reconstruct.

What carries the argument

The central object is the reproducibility protocol: a fixed pipeline that feeds a systematic-review title to an LLM, parses the returned Boolean query from a structured JSON response, runs it against the PubMed database, and computes precision, recall, F1, and F3 against the collection's relevance judgments. The protocol's load-bearing move is repetition: each prompt is run with several random seeds (ten for the smaller models, five for GPT-3.5-0125 and GPT-4), so the spread of scores across seeds becomes the measured quantity. That spread, not any single score, is what the paper uses to argue that LLM query generation is currently unreliable.

What would settle it

Audit the original experiments to establish the exact model version and prompt-handling procedure, then rerun each of the same prompts ten times with those exact settings on the same two collections. If the per-seed standard deviation of F1 turns out small (say under 0.05) and the original precision and recall values are recovered, the paper's central claim of unreliable, non-reproducible LLM query generation would be falsified.

Watch

Extended reading notes

Core claim

For systematic reviews, this paper reports that LLM-generated Boolean queries, evaluated by retrieving documents from PubMed and scoring precision, recall, F1, and F3 against known relevant sets, are too variable to reproduce. Re-running the same prompt with different seeds on the same topic produced large spread in every metric; on the main Seed test collection no tested model reached the effectiveness figures of the work being reproduced, while on the CLEF TAR collection the commercial models at times exceeded the reproduced precision values. The guided multi-step prompting approach deteriorated badly for several models, and across 11,200 generated queries 1,771 had incorrectly placed parentheses while a substantial share of guided runs yielded no valid query at all. The authors state plainly that none of the obtained results are reliable enough for the critical task of systematic reviews.

Load-bearing premise

The reproduction depends on the assumption that the original study used the model version and prompt format the authors inferred; if the original setup actually differed, the failure to reproduce would reflect a changed setup rather than inherent LLM instability.

Editorial extensions

If this is right

  • Researchers planning to automate systematic-review search should not yet replace expert-authored Boolean queries with a single LLM-generated query, because one run is effectively a random draw.
  • Numbers reported by the reproduced study should be treated as unstable benchmarks; future comparisons need repeated-seed means and standard deviations before claiming that one query generator outperforms another.
  • Open-weight models performed competitively with commercial models on some precision-oriented settings, but the same seed variance applies, so openness alone does not solve the reliability problem.
  • Detailed reporting of model version, prompt parsing, and dataset cleaning is necessary for any follow-up study; the paper shows that vague documentation made the original setup hard to reconstruct.
  • The small monetary cost of this repeated-seed study (about 150 USD in API calls) means that reproducibility testing is affordable and should become a standard practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if this variance result generalizes to other structured-generation tasks, then every published comparison of LLMs should report seed-to-seed variance rather than a single generated output.
  • Editorial extension: the duplicate and near-duplicate topics found in the Seed collection imply that some reported improvements on that benchmark may reflect the model echoing an example query that is effectively the answer key, rather than genuine query-writing skill.
  • Editorial extension: a focused follow-up could rerun the protocol with JSON-format output enforced for all models; the paper's own error analysis suggests that structured return modes eliminate a large class of extraction failures and may reduce the observed variance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents a reproducibility and generalizability study of LLM-based Boolean query generation for systematic reviews, reproducing and extending Wang et al. (2023) and Alaniz et al. (2023). The authors implement a pipeline that prompts GPT-3.5, GPT-4, Mistral, and Zephyr models, retrieves PubMed results, and evaluates precision, recall, F1, and F3 on the Seed and CLEF TAR collections across multiple seeds. They report that they cannot reproduce the numeric results of Wang et al., find high variance when re-running the same prompts with different seeds, compare open-source and commercial models, and conduct a failure analysis of malformed queries and dataset issues. The paper concludes that LLM-generated Boolean queries in their current form are not reliable enough for systematic review automation.

Significance. If the central claim holds, the paper provides an important cautionary result for a highly cited work, with practical implications for any pipeline that uses LLMs to generate search strategies. The authors ship code, use repeated seeded runs, report standard deviations, and quantify formatting errors, which are concrete strengths. However, the RQ1 claim is currently under-supported because the reported variability pools topic and seed effects, and the reproduction comparison relies on inferred model versions and adapted prompting. These issues are addressable with reanalysis and a more careful framing, so the study makes a useful contribution despite these gaps.

major comments (5)
  1. [Section 4.2, Tables 1-3] The standard deviations in Tables 1-3 and the boxplots in Section 4.2 are computed over the pooled set of topics and random seeds (Section 4.2: "average Recall and Precision scores across topics and random seeds"). This pooling conflates topic difficulty with seed variability, so the large SDs do not by themselves demonstrate that "rerunning the same prompts with different seeds" is unreliable, as claimed in the RQ1 answer in Section 6. A deterministic model that is very good on some topics and poor on others would yield the same pooled SD. Please report a variance decomposition, such as the mean or median of within-topic standard deviations across seeds, or a mixed-effects model with topic as a random factor, to isolate the seed contribution.
  2. [Section 3.3-3.4] The comparison to Wang et al. rests on two unverified assumptions: that the original ChatGPT experiments used GPT-3.5, inferred from the January 2023 submission deadline (Section 3.3), and that the adapted prompts, JSON return mode, and system/user prompt split preserve the original query generation behavior (Section 3.4). Since Section 5.1 itself states that the original prompt-issuing and query-extraction procedures are not described, the failure to reproduce Wang et al.'s numbers could be caused by these setup differences rather than by inherent LLM unreliability. Please reframe the RQ1 claim as conditional on these assumptions, or add a sensitivity analysis using free-text output without JSON and with the original prompt as a single user message.
  3. [Section 3.5, Tables 1-3] The text in Section 3.5 says "Thus we could not run the relevant example queries q4 and q5 for the CLEF TAR dataset," yet Tables 1-3 list q4-HQE and q5-HQE rows for CLEF TAR with values. This is a direct internal contradiction. Please clarify whether the reported CLEF TAR q4/q5 values come from a different example selection or were obtained after the statement was written, and correct the text or the tables accordingly.
  4. [Section 3.2] For CLEF TAR, the paper copies the baseline results from Wang et al. rather than reproducing them, because the original queries are not available in a PubMed-compatible format (Section 3.2). As a result, the CLEF TAR comparison cannot independently validate either the baseline or Wang et al.'s query results. Please either obtain and translate the original CLEF TAR queries (e.g., via the Polyglot tool the authors already tried) and report the reproduced baselines, or explicitly label the CLEF TAR comparison as dependent on the original paper's numbers and therefore not a full reproduction.
  5. [Section 4.1, Tables 1-3] The one-sample t-test against the single reported value from Wang et al. is used to mark results with a dagger throughout Tables 1-3. This test treats Wang et al.'s point estimate as a fixed population value, ignores the fact that Wang et al. also ran only one sample (or a small number) of queries, and does not correct for the large number of comparisons across models and query types. Please report effect sizes or confidence intervals and apply a multiple-comparison correction, or remove the significance annotations and rely on descriptive comparisons.
minor comments (6)
  1. [Section 3.3] There is a typesetting error in the model list ("Mistral-tiny (Mistral-7B-v0.2) and Mistral-small (Mixtral-8X7B-v0.1)8:."), with a stray colon and number after the closing parenthesis.
  2. [Table 4] The first model row is labeled "GPT3.5-turbo," which is not among the API models listed in Section 3.3; please clarify which model this row refers to.
  3. [Section 4.1] The statement "for six out of eight query types on the Seed dataset, we couldn't reproduce the original results" is followed later in the same section by "none of our models reached the scores reported by Wang et al.," which is stronger than the six-out-of-eight formulation; please reconcile the two statements.
  4. [Section 5.3.1] The duplicate-topic analysis would benefit from a small table or list of the identified duplicate pairs (e.g., 43/96, 7/67, 8/112) with their numbers of shared and disjoint relevant documents, since the current prose is dense and hard to verify.
  5. [Section 6] The phrase "In this Section, we summarize" uses an unnecessary capitalization of "Section"; please use lowercase.
  6. [Section 1] The word "influental" should be "influential."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical reproducibility study benchmarked against external collections and prior published baselines.

full rationale

This paper is an empirical evaluation rather than a derivation: it contains no equations, no fitted parameters, and no quantity that is constructed from its own inputs. The central RQ1 claim of high variance across seeds rests on the authors' own repeated runs with different seeds against the external Seed and CLEF TAR collections; the values taken from Wang et al. and Alaniz et al. serve only as external comparison baselines, not as ingredients of the reported variability findings. The paper does contain self-citations (e.g., Kusa et al. [27, 28, 29, 30], with [29] used to support the claim that no other suitable datasets exist), but these are not load-bearing for the main reproducibility conclusion, which would stand or fall entirely on the reported experiments. The skeptical concern that the reported standard deviations pool across topics and seeds, thereby conflating topic difficulty with seed variation, is a statistical validity issue rather than a circularity: the claim is not true by definition or by construction. No self-definitional step, fitted input renamed as prediction, or self-citation chain is exhibited, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted parameters or new entities. Its conclusions depend on assumptions about parity with the original studies and about the validity of benchmark queries and relevance judgments. The most fragile are the inferred model version and the prompt and JSON adaptations, both disclosed by the authors.

assumptions (5)
  • ad hoc to paper Wang et al. used GPT-3.5 for their ChatGPT experiments.
    Inferred from submission deadline and GPT-4 release date in Section 3.3; if the original model was different, the comparison target shifts.
  • ad hoc to paper Refined prompts and JSON return mode preserve the original query generation behavior.
    Section 3.4 says prompts were adapted for system/user split and JSON extraction; there is no check that this yields queries equivalent to the original free-text extraction.
  • ad hoc to paper Copied CLEF TAR baseline results from Wang et al. are accurate and comparable.
    Section 3.2 states baselines were copied because the queries could not be reproduced in PubMed-compatible format.
  • domain assumption Seed and CLEF TAR relevance judgments are valid ground truth.
    Sections 3.1 and 3.7 inherit assessment quality from the collections; any labeling noise is not modeled.
  • ad hoc to paper One-sample t-test against the reported scalar from Wang et al. is meaningful.
    Tables 1 to 3 flag significance against a single reported value whose distribution is unknown; this assumes the reported value is a stable point estimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Reproducibility and Generalizability Study of Large Language Models for Query Generation." pith.science (2026). https://pith.science/paper/BT5PVRRL

@misc{pith2026241114914,
  author       = {Pith},
  title        = {Pith review of: A Reproducibility and Generalizability Study of Large Language Models for Query Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BT5PVRRL}},
  note         = {Machine review of arXiv:2411.14914}
}
read the original abstract

Systematic literature reviews (SLRs) are a cornerstone of academic research, yet they are often labour-intensive and time-consuming due to the detailed literature curation process. The advent of generative AI and large language models (LLMs) promises to revolutionize this process by assisting researchers in several tedious tasks, one of them being the generation of effective Boolean queries that will select the publications to consider including in a review. This paper presents an extensive study of Boolean query generation using LLMs for systematic reviews, reproducing and extending the work of Wang et al. and Alaniz et al. Our study investigates the replicability and reliability of results achieved using ChatGPT and compares its performance with open-source alternatives like Mistral and Zephyr to provide a more comprehensive analysis of LLMs for query generation. Therefore, we implemented a pipeline, which automatically creates a Boolean query for a given review topic by using a previously defined LLM, retrieves all documents for this query from the PubMed database and then evaluates the results. With this pipeline we first assess whether the results obtained using ChatGPT for query generation are reproducible and consistent. We then generalize our results by analyzing and evaluating open-source models and evaluating their efficacy in generating Boolean queries. Finally, we conduct a failure analysis to identify and discuss the limitations and shortcomings of using LLMs for Boolean query generation. This examination helps to understand the gaps and potential areas for improvement in the application of LLMs to information retrieval tasks. Our findings highlight the strengths, limitations, and potential of LLMs in the domain of information retrieval and literature review automation.

Figures

Figures reproduced from arXiv: 2411.14914 by the authors.

Figure 1
Figure 1. Proposed pipeline to generate and evaluate boolean queries [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Recall variability for queries q4, q5 for the Seed dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Precision variability for queries q4, q5 for the Seed dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average Boolean query term length per query for the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Used PubMed search fields per Boolean query. Red line [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Percentage of incorrect query generations, per query type, [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 33 canonical work pages

  1. [1]

    Leonardo Alaniz, Cindy Vu, and Miles J Pfaff. 2023. The Utility of Artificial Intel- ligence for Systematic Reviews and Boolean Query Formulation and Translation. Plastic and Reconstructive Surgery–Global Open 11, 10 (2023), e5339

    work page 2023

  2. [2]

    Amal Alharbi and Mark Stevenson. 2017. Ranking Abstracts to Identify Relevant Evidence for Systematic Reviews: The University of Sheffield’s Approach to CLEF eHealth 2017 Task 2. In CLEF (Working Notes)

    work page 2017

  3. [3]

    Amal Alharbi and Mark Stevenson. 2020. Refining Boolean queries to iden- tify relevant studies for systematic review updates. Journal of the Amer- ican Medical Informatics Association 27, 11 (10 2020), 1658–1666. https: //doi.org/10.1093/jamia/ocaa148 arXiv:https://academic.oup.com/jamia/article- pdf/27/11/1658/34363868/ocaa148.pdf

    work page doi:10.1093/jamia/ocaa148 2020

  4. [4]

    Antonios Anagnostou, Athanasios Lagopoulos, Grigorios Tsoumakas, and Ioan- nis P Vlahavas. 2017. Combining Inter-Review Learning-to-Rank and Intra- Review Incremental Training for Title and Abstract Screening in Systematic Reviews. In CLEF (Working Notes)

    work page 2017

  5. [5]

    Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed- Source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics

    work page 2024

  6. [6]

    Brown, Patrice L

    Rohit Borah, Andrew W. Brown, Patrice L. Capers, and Kathryn A. Kaiser. 2017. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. , 12545 pages. https: //doi.org/10.1136/bmjopen-2016-012545

    doi:10.1136/bmjopen-2016-012545 2017

  7. [7]

    Andres Carvallo, Denis Parra, Hans Lobel, and Alvaro Soto. 2020. Automatic document screening of medical literature using word and text embeddings in an active learning setting. Scientometrics 125 (2020), 3047–3084

    work page 2020

  8. [8]

    Jiayi Chen, Su Chen, Yang Song, Hongyu Liu, Yueyao Wang, Qinmin Hu, Liang He, and Yan Yang. 2017. ECNU at 2017 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine. In CLEF (Working Notes)

    work page 2017

Show all 59 references
  1. [9]

    Justin Clark. 2013. Systematic reviewing: Introduction, locating studies and data abstraction. In Methods of clinical epidemiology . Springer, 187–211

    2013

  2. [10]

    A. M. Cohen, W. R. Hersh, K. Peterson, and Po Yin Yen. 2006. Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association 13, 2 (3 2006), 206–219. https: //doi.org/10.1197/jamia.M1929

    2006 doi

  3. [11]

    Giorgio Maria Di Nunzio, Federica Beghini, Federica Vezzani, and Geneviève Henrot. 2017. An Interactive Two-Dimensional Approach to Query Aspects Rewriting in Systematic Reviews. In IMS Unipd At CLEF eHealth Task 2

    2017

  4. [12]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496 (2022)

    2022 arXiv

  5. [13]

    Roberto Gozalo-Brizuela and Eduardo C Garrido-Merchan. 2023. ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv preprint arXiv:2301.04655 (2023)

    2023 arXiv

  6. [14]

    Nathalia Sernizon Guimarães, Julliane Vasconcelos Joviano-Santos, Marcela Gomes Reis, Roberta Rayra Martins Chaves, and Health Re- search (OPENS) Observatory of Epidemiology, Nutrition. 2024. Development of search strategies for systematic reviews in health using ChatGPT: a cr...

    2024

  7. [15]

    Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597 (2023)

    2023 arXiv

  8. [16]

    Elke Hausner, Charlotte Guddat, Tatjana Hermanns, Ulrike Lampert, and Siw Waffenschmidt. 2015. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.Journal of clinical epidemiology 68, 2 (2015), 191–199

    2015

  9. [17]

    Julian PT Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian A Welch. 2019. Cochrane handbook for systematic reviews of interventions . John Wiley & Sons

    2019

  10. [18]

    Noah Hollmann and Carsten Eickhoff. 2017. Ranking and Feedback-based Stop- ping for Recall-Centric Document Retrieval. In CLEF (Working Notes)

    2017

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  13. [21]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023)

    2023 arXiv

  14. [22]

    Akers Jo, Aguiar-Ibáñez Raquel, Burch Jane, Chambers Duncan, Eastwood Alison, Fayter Debra, Hempel Susanne, Light Kate, Rice Stephen, Rithalia Amber, Stewart Lesley, Stock Christian, Wilson Paul, and Woolacott Nerys. 2009. Systematic Reviews: CRD’s guidance for undertaking rev...

    2009

  15. [23]

    Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017. CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In CLEF’17

    2017

  16. [24]

    Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018. CLEF 2018 technologically assisted reviews in empirical medicine overview.CEUR Workshop Proceedings 2125 (7 2018). https://pureportal.strath.ac.uk/en/publications/clef- 2018-technologically-assisted-reviews-in...

    2018

  17. [25]

    Evangelos Kanoulas, Rene Spijker, Dan Li, and Leif Azzopardi. 2018. CLEF 2018 Technology Assisted Reviews in Empirical Medicine Overview. In CLEF 2018 Evaluation Labs and Workshop: Online Working Notes (CEUR-WS)

    2018

  18. [26]

    Sarvnaz Karimi, Justin Zobel, Stefan Pohl, and Falk Scholer. 2009. The chal- lenge of high recall in biomedical systematic search. In Proceedings of the third international workshop on Data and text mining in bioinformatics . 89–92

    2009

  19. [27]

    Wojciech Kusa, Allan Hanbury, and Petr Knoth. 2022. Automation of Cita- tion Screening for Systematic Literature Reviews Using Neural Networks: A Replicability Study. In Advances in Information Retrieval , Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Kris...

    2022 arXiv

  20. [28]

    Wojciech Kusa, Petr Knoth, and Allan Hanbury. 2023. CRUISE-Screening: Liv- ing Literature Reviews Toolbox. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (Birmingham, United Kingdom) (CIKM ’23). Association for Computing Machine...

    2023

  21. [29]

    Wojciech Kusa, Oscar E Mendoza, Matthias Samwald, Petr Knoth, and Allan Han- bury. 2023. CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews. arXiv preprint arXiv:2311.12474 (2023)

    2023 arXiv

  22. [30]

    Wojciech Kusa, Guido Zuccon, Petr Knoth, and Allan Hanbury. 2023. Outcome- Based Evaluation of Systematic Review Automation. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (Taipei, Taiwan) (ICTIR ’23). Association for Computing...

    2023

  23. [31]

    Lee and Aixin Sun

    Grace E. Lee and Aixin Sun. 2018. Seed-driven Document Ranking for System- atic Reviews in Evidence-Based Medicine. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’18) . ACM, New York, NY, USA, 455–464. https://doi.org/...

    2018

  24. [32]

    Adamantios Minas, Athanasios Lagopoulos, and Grigorios Tsoumakas. 2018. Aristotle University’s Approach to the Technologically Assisted Reviews in Empirical Medicine Task of the 2018 CLEF eHealth Lab. InCLEF (Working Notes)

    2018

  25. [33]

    Christopher Norman, Mariska Leeflang, and Aurélie Névéol. 2018. LIMSI@ CLEF eHealth 2018 Task 2: Technology Assisted Reviews by Stacking Active and Static Learning. In CLEF (Working Notes)

    2018

  26. [34]

    Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023. Chat- gpt versus traditional question answering for knowledge graphs: Current sta- tus and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466 (2023)

    2023 arXiv

  27. [35]

    Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015. Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews 4, 1 (1 2015), 5. https://doi.org/10.1186/2046-4053-4-5

    2015 doi

  28. [36]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing . Association for Computational Linguistics. https://arxiv.org/abs/1908.10084

    2019 arXiv

  29. [37]

    Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare 11 (2023), 887

    2023

  30. [38]

    Margaret Sampson and Jessie McGowan. 2006. Errors in search strategies were identified by type and frequency. Journal of clinical epidemiology 59, 10 (2006), 1057–e1

    2006

  31. [39]

    Harrisen Scells and Guido Zuccon. 2018. Searchrefiner: A query visualisation and understanding tool for systematic reviews. In Proceedings of the 27th ACM international conference on information and knowledge management . 1939–1942

    2018

  32. [40]

    Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2020. You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews. In European Conference on Information Retrieval . Springer, 399–414

    2020

  33. [41]

    Harrisen Scells, Guido Zuccon, and Bevan Koopman. 2021. A comparison of au- tomatic Boolean query formulation for systematic reviews. Information Retrieval Journal 24 (2021), 3–28

    2021

  34. [42]

    Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva. 2017. A test collection for evaluating retrieval of studies for inclusion in systematic reviews. InProceedings of the 40th International ACM SIGIR Conference on Research and Developm...

    2017

  35. [43]

    Shojania, Margaret Sampson, Mohammed T

    Kaveh G. Shojania, Margaret Sampson, Mohammed T. Ansari, Jun Ji, Steve Doucette, and David Moher. 2007. How quickly do systematic reviews go out of date? A survival analysis. Annals of Internal Medicine 147, 4 (8 2007), 224–233. A Reproducibility and Generalizability Study of ...

    2007 doi

  36. [44]

    Michael Simon, Elke Hausner, Susan F Klaus, and Nancy E Dunton. 2010. Identi- fying nurse staffing research in Medline: development and testing of empirically derived search strategies with the PubMed interface. BMC medical research methodology 10, 1 (2010), 1–8

    2010

  37. [45]

    Jaspreet Singh and Lini Thomas. 2017. IIIT-H at CLEF eHealth 2017 Task 2: Tech- nologically Assisted Reviews in Empirical Medicine. In CLEF (Working Notes)

    2017

  38. [46]

    Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of ChatGPT as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992 (2023)

    2023 arXiv

  39. [47]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca

    2023

  40. [48]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  41. [50]

    Tricco, Jamie Brehaut, Maggie H

    Andrea C. Tricco, Jamie Brehaut, Maggie H. Chen, and David Moher. 2008. Follow- ing 411 Cochrane Protocols to Completion: A Retrospective Cohort Study. PLOS ONE 3, 11 (11 2008), e3684. https://doi.org/10.1371/JOURNAL.PONE.0003684

    2008 doi

  42. [51]

    Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera

    Guy Tsafnat, Paul Glasziou, Miew K. Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera. 2014. Systematic review automation technologies. , 74 pages. https://doi.org/10.1186/2046-4053-3-74

    2014 doi

  43. [52]

    Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query Expansion with Large Language Models. arXiv preprint arXiv:2303.07678 (2023)

    2023 arXiv

  44. [53]

    Shuai Wang, Harrisen Scells, Justin Clark, Bevan Koopman, and Guido Zuccon

  45. [54]

    Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiw...

    2023

  46. [55]

    Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search. In Proceedings of the 26th Australasian Document Computing Symposium (ADCS ’22). Association for Computing ...

    2023

  47. [56]

    Eugene Yang, Sean MacAvaney, David D Lewis, and Ophir Frieder. 2022. Goldilocks: Just-right tuning of bert for technology-assisted review. In Euro- pean Conference on Information Retrieval . Springer, 502–517

    2022

  48. [57]

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih

  49. [58]

    Hye Sun Yun, Iain J Marshall, Thomas Trikalinos, and Byron C Wallace. 2023. Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews. arXiv preprint arXiv:2305.11828 (2023)

    2023 arXiv

  50. [2022]

    In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

    From little things big things grow: A collection with seed studies for medical systematic review literature search. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . 3176– 3186

  51. [2023]

    arXiv preprint arXiv:2302.13971 (2023)

    Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.