Pith. sign in

REVIEW 3 major objections 6 minor 45 references

Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Evenly sampling every paper in a study, clustering the passages, and distilling each cluster before generating the conclusion predicts study conclusions up to 10.3% F1 more accurately than prior methods, on a new Cochrane-derived benchmark.

desk verdict A genuinely useful new dataset and a plausible method, but the headline RAG comparison is not budget-controlled, so the 10.3% claim should be read with caution. read the letter →

arxiv 2505.06186 v3 pith:H3GDFOZ5 submitted 2025-05-09 cs.CL cs.AI

Massimiliano Pronesti , Joao Bettencourt-Silva , Paul Flanagan , Alessandra Pascale , Oisin Redmond , Anya Belz , Yufang Hou This is my paper · ORCID
classification cs.CLcs.AI
keywords scientificevidenceextractionretrieval-augmentedgenerationforestplotssystematicreviewsclinicalresearchquestionsfull-textbiomedicalNLPrandomizedcontrolledtrialscontradictory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper formalises a new task in automated evidence synthesis: given a clinical research question and the full texts of the papers that make up a set of studies, predict each study's conclusion—favours the intervention, favours the control, or no difference. To support the task, the authors built CochraneForest, a dataset of 202 forest plots from 48 Cochrane systematic reviews with 923 question–study pairs, restricted to plots where at least two studies reach contradictory conclusions. They then propose URCA, a retrieval-augmented generation framework that distributes the retrieval budget evenly across a study's papers, clusters the retrieved passages by meaning, uses a language model to extract query-relevant evidence from each cluster, and only then generates the conclusion; on CochraneForest it beats seven baselines across four LLMs by up to 10.3% F1, and it also improves accuracy on the open-domain sets PubMedQA and MedQA-US. The paper's message is that document-level evidence extraction is a difficult, well-defined testbed for evidence synthesis, and that the clustering step—not the uniform retrieval—carries most of URCA's advantage.

What carries the argument

The central mechanism is the URCA pipeline, a retrieval-augmented generation framework whose key steps are uniform source allocation, embedding clustering, and per-cluster LLM extraction. Uniform allocation retrieves $k_s = \lceil \min(k+\beta \log S, N_{\max})/S \rceil$ passages from each of the $S$ papers in a study, so that no single paper can dominate the context. The retrieved passages are then clustered with UMAP dimensionality reduction and a Gaussian mixture model selected by the Bayesian Information Criterion, and a language model is prompted to extract information relevant to the research question from each cluster. Finally, the extracted cluster summaries are concatenated and given to the model to produce the study conclusion. The paper's ablations establish clustering as the load-bearing component: replacing it with contiguous grouping of shuffled chunks degrades performance, while altering the order in which clusters are presented changes F1 by only about $\pm 1\%$.

What would settle it

Run URCA on the same 202 research questions without giving it the included-study set, letting it select studies from the full Cochrane corpus; if its F1 advantage over GraphRAG closes or reverses, the reported gains depend on the pre-filtered study assumption and would not transfer to open-domain evidence extraction.

Watch

Extended reading notes

Core claim

The central claim is that evidence extraction from multi-paper studies is best served by a retrieval design that treats each paper as an equally important source and defers synthesis until the retrieved content has been grouped by meaning. URCA follows this design: it allocates a computed number of passage retrievals to every paper in a study, clusters all retrieved passages with UMAP and a Gaussian mixture model, prompts a language model to extract query-relevant evidence from each cluster, and only then prompts the model to produce the study conclusion. The paper reports that this pipeline outperforms every baseline on CochraneForest—No RAG, abstract-only context, vanilla RAG with and without uniform retrieval, RAPTOR, InstructRAG, and GraphRAG—across Llama-3.1-70B, Mistral Large, GPT-3.5 Turbo, and GPT-4, with the largest gain (10.3% F1) over GraphRAG on GPT-3.5 Turbo. Ablations show that removing clustering costs 3.9–5.7 absolute F1 points, while removing uniform retrieval costs only 1.6–2.4 points. The same cluster-then-extract step, without uniform retrieval, transfers to open-domain QA, reaching 85.9% accuracy on MedQA-US and 81.1% on PubMedQA with GPT-4.

Load-bearing premise

The system is handed the set of studies that belong under each research question and the papers that make up each study, so it never has to decide which papers are relevant or whether a study belongs in the forest plot.

Editorial extensions

If this is right

  • Automated evidence synthesis could seed forest plots: given a clinical question and the included studies' full texts, a review team can obtain per-study conclusions to populate the plot and focus manual work on verification.
  • Full-text evidence is necessary for outcome-specific questions; abstract-only and no-retrieval baselines are strong but consistently below URCA, because the required nuance often lives in tables and analyses beyond the abstract.
  • Clustering is the component to invest in: it accounts for the larger share of URCA's gains on CochraneForest, while uniform retrieval mainly improves coverage of less prominent papers.
  • The cluster-then-extract design generalises beyond systematic reviews: on PubMedQA and MedQA-US, URCA (without uniform retrieval) outperforms vanilla RAG and GraphRAG, suggesting the same pipeline fits open-domain medical QA.
  • CochraneForest provides the research community with a benchmark of 923 question–study pairs centered on conflicting evidence, a regime that existing methods handle poorly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the paper reports that no annotator ever modified the conclusion label pre-selected from the study's 95% confidence interval, the labels are near-deterministic functions of reported effect sizes; a cheap quantitative baseline that reads point estimates and intervals from tables could rival or exceed RAG systems on this benchmark.
  • Because the dataset lacks rationale annotations identifying which passages support each conclusion, a natural extension is to post-hoc align the retrieved clusters (or their attention weights) to the final prediction, producing explainable evidence trails and enabling error analysis.
  • The uniform-retrieval benefit is small on CochraneForest (papers per study average 1.82), so its value may grow on corpora with more uneven multi-paper studies; a version of the benchmark sampling studies with many papers would test this prediction.
  • URCA's largest relative gain appears with GPT-3.5 Turbo (10.3% F1) and is smaller with GPT-4, suggesting weaker models benefit more from the structured, clustered context; this predicts the method will be most useful with small, open-weight models in constrained settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces COCHRANEFOREST, a dataset of 202 forest plots from 48 Cochrane systematic reviews, with 923 research-question--study pairs and full texts of 263 unique studies, annotated with study conclusions relative to a clinical question. It formalizes the task of document-level evidence extraction from studies with conflicting conclusions, and proposes URCA, a RAG framework that retrieves a source-balanced set of passages, clusters them via UMAP+GMM, extracts query-relevant evidence with an LLM, and produces a final answer. Experiments on four LLMs report F1/accuracy gains over No RAG, Abstracts, vanilla RAG (with and without uniform retrieval), RAPTOR, InstructRAG, and GraphRAG, plus additional results on PubMedQA and MedQA-US. The paper includes ablations, cluster-ordering tests, inter-annotator agreement, and a qualitative example.

Significance. If the empirical claims hold, COCHRANEFOREST is a useful and genuinely document-level benchmark that goes beyond abstract-level evidence extraction, and URCA is a simple, explainable RAG baseline with a plausible design rationale (source-balanced retrieval plus query-guided clustering). The authors provide several strengths: a substantial annotation effort with IAA reporting, ablations of the two main components, tests on two external QA benchmarks, and a qualitative trace of the pipeline. However, the central comparative claim is currently undercut by a retrieval-budget confound: URCA appears to place more passages in context than the baselines, so the reported F1 gains may not reflect the method's intrinsic merit. The missing hyperparameter details and absence of any variance estimate further weaken the claim of consistency. The contribution is promising and within the journal's scope, but the comparison must be made fair and reproducible before the headline result can be accepted.

major comments (3)
  1. [§5.1, Algorithm 1] The comparison in Table 3 is not retrieval-budget-controlled. The setup states that 'by default, we use the top 10 retrieved passages in all the approaches under comparison' (Section 5.1), but Algorithm 1 sets k_s = ceil(min(k + β·log(S), N_max)/S) and therefore retrieves S·k_s passages. For k=10, S≥2 and β>0, this is strictly greater than 10, so URCA receives more retrieved evidence than the top-10 baselines. Since β, N_max, and the resulting average context size are never reported (Appendix F only says other hyperparameters were left at defaults), the reader cannot tell whether the F1 gains come from the method or from extra context. The 'w/o Uniform Retrieval' ablation in Table 5 changes the number of chunks as well as the source allocation, so it does not isolate the uniform-retrieval mechanism. Please re-run the experiments with a matched total budget (equal number of passages or equal token budget per query), report β/N_max and average context sizes, and include a URCA variant that uses exactly k passages in total.
  2. [Appendix F, Algorithm 1] URCA is not reproducible from the information given. The paper does not report β or N_max, which directly control how many passages Algorithm 1 retrieves, nor does it specify the embedding model, passage chunk size/overlap, UMAP parameters, or the random seed used for UMAP and GMM clustering. The statement in Appendix F that 'all the other hyperparameters were left to the default value' is insufficient because these parameters are not defaults of any named library that can be identified from the paper. Since clustering is reported as the largest contributor to performance (Table 5), the stochastic components of clustering need fixed seeds or repeated runs. Please provide a complete configuration table or a link to a released code repository.
  3. [§5.2, Tables 3 and 5] All reported numbers are single point estimates with no error bars, confidence intervals, or significance tests. Several of the decisive comparisons are small in absolute terms: on Llama-3.1-70B, URCA's F1 is 66.1 versus GraphRAG's 65.6, and the uniform-retrieval ablation differences are between 1.6 and 2.4 F1 points. Given the stochasticity of UMAP/GMM clustering and the use of API-based LLMs, a few repeated runs with different seeds, or a paired significance test, are needed to support the claim that URCA 'consistently outperforms' the baselines across all four models.
minor comments (6)
  1. [Abstract, §5.2] The phrase 'up to 10.3% in F1 score' should be made unambiguous: the 10.3% is a relative improvement over GraphRAG on GPT-3.5-Turbo, not 10.3 absolute F1 points; please state both relative and absolute values.
  2. [§5.1] The sentence 'by default, we use the top 10 retrieved passages in all the approaches under comparison' is inconsistent with the retrieval formula in Algorithm 1; please define the value of k used for URCA and clarify how the top-10 budget applies to the uniform-retrieval variants.
  3. [§3.3, Table 2] The Fleiss κ of 0.06 for Task 1 is reported alongside cosine similarity 0.95, but the paper does not explain why the two metrics diverge so sharply; since pairwise φ values show that annotators changed 33–100% of the items, a short discussion of what the κ is measuring would help readers judge the reliability of the research-question annotations.
  4. [§3.2] The text notes that no annotator modified the pre-selected conclusion label (Task 2); this means the conclusion labels were effectively derived deterministically from confidence intervals, so the reported inter-annotator agreement does not validate the label assignment itself. Please state this explicitly.
  5. [Appendix F] There is a typo in 'gpt-3.5.-turbo' (should be 'gpt-3.5-turbo').
  6. [§3.1 / Appendices] The paper says the dataset is released but provides no URL or access mechanism for the dataset or code; please add repository links or a clear availability statement.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: URCA's F1 gains are empirical, externally benchmarked, and not forced by construction; the only mild concern is that some CochraneForest research questions were generated by a model from the same family later evaluated.

full rationale

The paper's central claim is an empirical comparison, not a derivation from fitted inputs. URCA uses off-the-shelf retrievers and LLMs, RAPTOR-style clustering, and prompt-based extraction; no parameter is fitted to the CochraneForest labels, and the ground-truth conclusions are defined externally by 95% confidence intervals extracted from Cochrane forest plots (Appendix B), not by URCA's outputs. The method is additionally tested on MedQA-US and PubMedQA, where it also outperforms baselines, providing independent non-circular support. The only mild circularity-adjacent issue is that Section 3.2 states research questions were generated by prompting llama-3.1-70b, and Llama-3.1-70B is one of the evaluated models; this could make the test inputs slightly model-familiar, but the labels are not model-generated and the gains replicate across GPT-4, GPT-3.5-Turbo, and Mistral Large, so it is not load-bearing. The retrieval-budget imbalance noted by a skeptical reader (URCA's uniform rule can place more than k passages in context) is a comparison-fairness concern, not a circular reduction of the result to its own inputs. Citations to co-authored prior work (O'Doherty et al. 2024; Al Khatib et al. 2021) are related-work context only and are not used to justify URCA's design or performance.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results are empirical; the only 'inputs' beyond standard LLM/RAG machinery are the dataset construction choices and a few unreported hyperparameters (beta, Nmax, UMAP seed). The label derivation from confidence intervals is an important domain assumption that the paper states but does not independently validate.

free parameters (3)
  • beta (scaling factor for uniform retrieval) = not reported
    Controls the per-source retrieval allocation ks = ceil(min(k + beta * log(S), Nmax) / S); the paper does not report the value used in the experiments.
  • Nmax (maximum retrieved passages cap) = not reported
    Caps the total number of passages in the uniform retrieval formula; no value is given in the paper.
  • UMAP random state / clustering seed = not reported
    UMAP is stochastic; no seed is reported, so clustering and therefore results may vary between runs.
assumptions (4)
  • domain assumption The Cochrane Database of Systematic Reviews is a valid and representative source for training and evaluating evidence extraction.
    The dataset is built entirely from Cochrane reviews, and the paper generalizes from this subset to the task of biomedical evidence extraction (Section 3.1).
  • domain assumption A study's conclusion is correctly determined by whether the 95% confidence interval of its effect estimate crosses the null threshold (0 for mean differences, 1 for ratios).
    Appendix B defines the three labels this way, and Task 2 in Section 3.2 pre-selects labels from the CI; annotators never modified them.
  • domain assumption The off-the-shelf retriever returns passages that contain the evidence needed to infer the conclusion.
    URCA and all RAG baselines depend on retrieval quality; no retriever training or re-ranking is performed (Section 5.1).
  • domain assumption The LLM output can be unambiguously mapped to one of the three predefined conclusion labels.
    The paper does not describe a parsing strategy or report how off-label outputs were handled in the F1 computation (Section 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies." pith.science (2026). https://pith.science/paper/H3GDFOZ5

@misc{pith2026250506186,
  author       = {Pith},
  title        = {Pith review of: Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3GDFOZ5}},
  note         = {Machine review of arXiv:2505.06186}
}
read the original abstract

Extracting scientific evidence from biomedical studies for clinical research questions (e.g., Does stem cell transplantation improve quality of life in patients with medically refractory Crohn's disease compared to placebo?) is a crucial step in synthesising biomedical evidence. In this paper, we focus on the task of document-level scientific evidence extraction for clinical questions with conflicting evidence. To support this task, we create a dataset called CochraneForest, leveraging forest plots from Cochrane systematic reviews. It comprises 202 annotated forest plots, associated clinical research questions, full texts of studies, and study-specific conclusions. Building on CochraneForest, we propose URCA (Uniform Retrieval Clustered Augmentation), a retrieval-augmented generation framework designed to tackle the unique challenges of evidence extraction. Our experiments show that URCA outperforms the best existing methods by up to 10.3% in F1 score on this task. However, the results also underscore the complexity of CochraneForest, establishing it as a challenging testbed for advancing automated evidence synthesis systems.

Figures

Figures reproduced from arXiv: 2505.06186 by the authors.

Figure 1
Figure 1. An example from COCHRANEFOREST. The research question is annotated based on systematic re￾view context and forest plot, where each row represents a study and its conclusion about the question. However, the process of producing systematic reviews is both time-consuming and costly. A 2019 study estimated that on average conducting a systematic review takes 1–2 years and costs over $141,000 (Michelson and Reuter, 2019)… view at source ↗
Figure 2
Figure 2. Overview of the URCA framework. Retrieval size is first distributed across all papers. Retrieved chunks [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Coverage rate of multiple sources with and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance with contiguous grouping versus [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: illustrates the filtering process used to construct the COCHRANEFOREST dataset. After downloading the full Cochrane CDSR archive, we applied multiple filtering stages described in Sec￾tion 3.1 to ensure that only reviews with complete data and high-quality forest plots…
Figure 6
Figure 6. Figure 6: Qualitative example from COCHRANEFOREST. URCA clusters the retrieved passages and filters out irrelevant information from each cluster [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Prompt templates for research question generation (a), knowledge extraction (b), and answer finalisation [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The forest plot annotation interface [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages

  1. [1]

    Khalid Al Khatib, Tirthankar Ghosal, Yufang Hou, Anita de Waard, and Dayne Freitag. 2021. https://doi.org/10.18653/v1/2021.sdp-1.7 Argument mining for scholarly document processing: Taking stock and looking ahead . In Proceedings of the Second Workshop on Scholarly Document Processing, pages 56--65, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.sdp-1.7 2021

  2. [2]

    M Alessio, G Faggioli, N Ferro, FM Nardini, R Perego, et al. 2024. Improving RAG systems via sentence clustering and reordering. In CEUR WORKSHOP PROCEEDINGS, volume 3784, pages 34--43

    work page 2024

  3. [3]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  4. [4]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    2024

  5. [5]

    Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. https://doi.org/10.1145/3637528.3671458 Bias and unfairness in information retrieval systems: New challenges in the LLM era . In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 6437–6447. ACM

    arXiv 2024

  6. [6]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130

    arXiv 2024

  7. [7]

    Joseph L. Fleiss. 1971. https://api.semanticscholar.org/CorpusID:143544759 Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378--382

    1971

  8. [8]

    Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022. New and improved embedding model. OpenAI Blog. Available online: https://openai. com/blog/new-and-improved-embedding-model (accessed on 28 November 2023)

    work page 2022

Show all 45 references
  1. [9]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929--3938. PMLR

    2020

  2. [10]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  3. [11]

    Julian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch, editors. 2024. https://www.training.cochrane.org/handbook Cochrane Handbook for Systematic Reviews of Interventions , version 6.5 (updated august 2024)...

    2024

  4. [12]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research

    2022

  5. [13]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. http://jmlr.org/papers/v24/23-0037.html Atlas: Few-shot learning with retrieval augmented language models . Jour...

    2023

  6. [14]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  7. [15]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019 doi

  8. [16]

    Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2024. REALTIME QA : what's the answer right now? Advances in Neural Information Processing Systems, 36

    2024

  9. [17]

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP . arXiv preprint arXiv:2212.14024

    2022 arXiv

  10. [18]

    Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. https://doi.org/10.1002/jrsm.1715 Can large language models replace humans in systematic reviews? evaluating GPT -4's efficacy in screening and extracting data from peer-reviewed and gr...

    2024 doi

  11. [19]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  12. [20]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV -embed: Improved techniques for training LLM s as generalist embedding models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2025

  13. [21]

    Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. https://doi.org/10.18653/v1/N19-1371 Inferring which medical treatments work from reports of clinical trials . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Com...

    2019 doi

  14. [22]

    VI Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Proceedings of the Soviet Physics Doklady, 10(8):707--710

    1966

  15. [23]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Proc...

    2020

  16. [24]

    Iain J Marshall and Byron C Wallace. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic reviews, 8:1--10

    2019

  17. [25]

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. https://doi.org/10.21105/joss.00861 UMAP : Uniform manifold approximation and projection . Journal of Open Source Software, 3(29):861

    2018 doi

  18. [26]

    Matthew Michelson and Katja Reuter. 2019. The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemporary clinical trials communications, 16:100443

    2019

  19. [27]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. https://doi.org/10.18653/v1/2023.eacl-main.148 MTEB : Massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages...

    2023 doi

  20. [28]

    Benjamin Nye, Ani Nenkova, Iain Marshall, and Byron C. Wallace. 2020. https://doi.org/10.18653/v1/2020.acl-demos.9 T rialstreamer: Mapping and browsing medical evidence in real-time . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: S...

    2020 doi

  21. [29]

    James O ' Doherty, Cian Nolan, Yufang Hou, and Anya Belz. 2024. https://doi.org/10.18653/v1/2024.acl-srw.42 Beyond abstracts: A new dataset, prompt design strategy and method for biomedical synthesis generation . In Proceedings of the 62nd Annual Meeting of the Association for...

    2024 doi

  22. [30]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  23. [31]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. RAPTOR : Recursive abstractive processing for tree-organized retrieval. In the Twelfth International Conference on Learning Representations

    2024

  24. [32]

    Chantal Shaib, Millicent Li, Sebastian Joseph, Iain Marshall, Junyi Jessy Li, and Byron Wallace. 2023. https://doi.org/10.18653/v1/2023.acl-short.119 Summarizing, simplifying, and synthesizing medical evidence using GPT -3 (with varying success) . In Proceedings of the 61st An...

    2023 doi

  25. [33]

    Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei W Koh. 2025. Scaling retrieval-based language models with a trillion-token datastore. Advances in Neural Information Processing Systems, 37:91260--91299

    2025

  26. [34]

    Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25 A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation ...

    2006

  27. [35]

    Daniel Sosa, Malavika Suresh, Christopher Potts, and Russ Altman. 2023. https://doi.org/10.18653/v1/2023.acl-short.61 Detecting contradictory COVID -19 drug efficacy claims from biomedical literature . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  28. [36]

    George Tsatsaronis, Michael Schroeder, Georgios Paliouras, Yannis Almirantis, Ion Androutsopoulos, Eric Gaussier, Patrick Gallinari, Thierry Artieres, Michael R Alvers, Matthias Zschunke, et al. 2012. Bio ASQ : A challenge on large-scale biomedical semantic indexing and questi...

    2012

  29. [37]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. https://doi.org/10.18653/v1/2024.findings-acl.813 F resh LLM s: Refreshing large language models with search engine augmentation . In Fi...

    2024 doi

  30. [38]

    Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö. Arık. 2024. https://arxiv.org/abs/2410.07176 Astute RAG : Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models . Preprint, arXiv:2410.07176

    2024 arXiv

  31. [39]

    RAT : Retrieval augmented thoughts elicit context-aware reasoning and verification in long-horizon generation

    Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. RAT : Retrieval augmented thoughts elicit context-aware reasoning and verification in long-horizon generation. In NeurIPS 2024 Workshop on Open-World Agents

    2024

  32. [40]

    Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025. https://openreview.net/forum?id=P1qhkp8gQT Instruct RAG : Instructing retrieval-augmented generation via self-synthesized rationales . In the Thirteenth International Conference on Learning Representations

    2025

  33. [41]

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. https://arxiv.org/abs/2405.15556 Certifiably robust RAG against retrieval corruption . Preprint, arXiv:2405.15556

    2024

  34. [42]

    Hye Yun, Iain Marshall, Thomas Trikalinos, and Byron Wallace. 2023. Appraising the potential uses and harms of LLM s for medical systematic reviews. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics

    2023

  35. [43]

    Hye Sun Yun, David Pogrebitskiy, Iain James Marshall, and Byron C Wallace. 2024. Automatically extracting numerical results from randomized controlled trials with large language models. In Machine Learning for Healthcare Conference. PMLR

    2024

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.