Pith. sign in

REVIEW 3 major objections 6 minor 56 references

Constructing and Evaluating Declarative RAG Pipelines in PyTerrier

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PyTerrier-RAG lets researchers build and evaluate sequential and iterative RAG pipelines declaratively in PyTerrier, and swap retrievers by renaming one variable in the pipeline expression.

desk verdict Genuinely useful RAG extension for PyTerrier, but the printed code in the key demo doesn't run, so the paper's central claim is not demonstrated as written. read the letter →

arxiv 2506.10802 v1 pith:F627W5S6 submitted 2025-06-12 cs.IR

Craig Macdonald , Jinyuan Fang , Andrew Parry , Zaiqiao Meng This is my paper
classification cs.IR
keywords RetrievalAugmentedGenerationPyTerrierDeclarativepipelinesQuestionansweringIterativeRAGIRCoTNeuralrerankingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This demo paper argues that retrieval-augmented generation (RAG) fits naturally into PyTerrier's declarative pipeline architecture, and introduces PyTerrier-RAG to make that fit concrete. The extension adds RAG-specific data types, answer-generating reader components, preprocessed question-answering datasets, prebuilt indices, and exact-match, F1, and ROUGE evaluation measures, so that a RAG system is expressed as a short pipeline such as retriever, then reranker, then context builder, then reader. The worked example compares BM25 and dense E5 retrieval feeding a Fusion-in-Decoder reader on Natural Questions, with the whole experiment evaluated by a single pt.Experiment call. The point is that RAG researchers can reuse the entire PyTerrier ecosystem of sparse, learned-sparse, and dense retrievers and neural rerankers without writing glue code or configuration files.

What carries the argument

The load-bearing object is PyTerrier's transformer pipeline over a typed relational data model: each stage maps one relation type to another (retrieval $Q\to R$, reranking $R\to R$, context building $R\to Q_c$, reading $Q_c\to A$), and operators such as >> (then), + (score sum), | (set union), and % (rank cutoff) compose them into a new transformer. PyTerrier-RAG extends this model with answer types $A$ and $GA$, the query-with-context type $Q_c$, a Reader class with pluggable backends, a Concatenator that turns retrieved documents into context, and an IRCOT class that implements iterative retrieval-generation loops. The declarative construction is what carries the argument: because a pipeline is an expression, not a script, retrieval components can be interchanged and experiments evaluated through the same pt.Experiment mechanism as classical IR.

What would settle it

Run the Section 5 worked example end-to-end in a clean environment: load the nq dataset, build the BM25 and E5 retrieval pipelines, attach the Fusion-in-Decoder reader with the specified model identifier, and call pt.Experiment for F1 and EM on the dev topics; if the pipelines error or the experiment returns no F1 or EM table because contexts or answers are malformed, the central claim of easy, working declarative RAG pipelines is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that RAG pipelines are naturally declarative, and that PyTerrier-RAG supplies the missing data model and components to express them. Sequential RAG becomes retriever, then reranker, then Concatenator, then reader, where the new Reader transformer maps a query-with-context type $Q_c$ to an answer type $A$, and a Backend abstraction lets local transformer models, sequence-to-sequence models, or API-based models be dropped in by changing one constructor. Iterative RAG is supported through an IRCOT wrapper that loops retriever and reader until a user-supplied exit condition on the generated answer is met. The same experiment API used for classical retrieval then evaluates answer quality using gold answers and textual-overlap measures, so swapping BM25 for E5 or adding MonoT5 is a one-variable change in the pipeline expression.

Load-bearing premise

The load-bearing premise is that the externally hosted artifacts, namely the prebuilt indices for Wikipedia and the fine-tuned reader models the worked example calls, exist, load correctly, and produce the results the code snippets imply, since the paper defers all actual output to notebook links rather than showing an executed run.

Editorial extensions

If this is right

  • Retriever-reader combinations become a controlled variable: comparing BM25 with E5 or dense versus sparse retrieval on answer quality is a one-line change to the pipeline expression.
  • PyTerrier retrieval plugins for SPLADE, ColBERT, dense encoders, and MonoT5 or duoT5 rerankers slot into RAG pipelines without new interfaces.
  • Iterative and adaptive RAG architectures reduce to pipeline objects with exit conditions, making multi-hop QA methods like IRCoT comparable under identical evaluation.
  • QA evaluation no longer requires intermediate result files: pt.Experiment computes Exact Match, F1, ROUGE, and BERTScore directly from pipeline outputs, with significance testing and prefix precomputation available.
  • Ten preprocessed datasets spanning general QA, multi-hop QA, fact checking, and dialogue generation lower the barrier to reproducing RAG experiments in a common environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors leave implicit is that the framework turns retriever-reader interaction into a controlled variable: the same pipeline expression can be re-run with each stage replaced, making causal studies of retrieval quality on answer quality straightforward.
  • The exit-condition loop used for IRCoT could plausibly express newer adaptive and self-reflective RAG methods as declarative pipelines, which would make those methods comparable under identical evaluation.
  • Because the data model is extensible, the same machinery could be pointed at multi-hop and fact-checking tasks beyond question answering; the preprocessed datasets for HotpotQA, MuSiQue, FEVER, and Wizard of Wikipedia are already present, and only task-specific measures would need to be added.
  • An implicit assumption worth testing empirically is whether retrieved documents should be ranked by estimated relevance at all; the paper's closing remark suggests RAG contexts may be lossy, and the framework allows direct experiments varying context order, size, and noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PyTerrier-RAG, an extension of the PyTerrier retrieval platform for declaratively constructing and evaluating retrieval-augmented generation (RAG) pipelines. It extends the PyTerrier data model with answer and gold-answer types, adds reader and backend components including a Fusion-in-Decoder reader, supports iterative RAG via an IRCoT implementation, offers ten preprocessed QA-style datasets and prebuilt indices, and integrates QA evaluation measures such as EM and F1 into the existing pt.Experiment API. The central claim is that RAG pipelines can be expressed succinctly using PyTerrier operators and evaluated declaratively, with a worked Natural Questions example comparing BM25 and E5 retrieval before a FiD reader.

Significance. The contribution is a software and demo contribution rather than a new algorithmic result. If the advertised functionality works as described, it is a useful addition to the IR experimental ecosystem: it lowers the barrier for RAG experimentation by reusing PyTerrier's operator notation, plugin ecosystem, and experiment framework, and it provides preprocessed datasets and indices. The paper is transparent about the ecosystem, names the relevant artifacts, and points to open-source code and notebooks. However, the verification burden is carried almost entirely by code snippets, and those snippets contain syntax errors and undefined variables, so the current version does not yet demonstrate the central ease-of-use claim.

major comments (3)
  1. [Section 5, first code block, lines 8–9] The worked example is syntactically incomplete: `bm25_fid = bm25 %` and `e5_fid = e5 %` are missing the rank-cutoff operand that the `%` operator requires (see Table 1), producing a Python SyntaxError. Because this example is the manuscript's primary demonstration of "easy-to-build pipelines" and all output is deferred to live notebooks, the paper as printed does not verify that the advertised pipelines execute. Please correct the snippets to include the intended cutoff (e.g., `% 10`) and add at least one executed output table or a machine-checked trace.
  2. [Section 4.3, IRCoT snippet] The variable `monoT5` is used in `retriever=bm25 >> monoT5` but is never defined; earlier in Section 4.2 the component is written as `MonoT5()`. This inconsistency makes the iterative RAG example non-runnable as printed and should be aligned, together with any required imports that a reader would need to reproduce the snippet.
  3. [Sections 1 and 5] The abstract and Section 1 claim "state-of-the-art LLM readers" and Section 5 promises a demonstration of succinctness, but the manuscript contains no executed experimental output or comparison: the EM/F1 table described in Section 5 is only said to appear in external notebooks. For a demo paper this is acceptable in principle, but the central capability claim would be materially strengthened by including the small NQ comparison, or at least a minimal reproducibility verification, inside the paper itself.
minor comments (6)
  1. [Table 3] The numeric columns are formatted without separators, making entries such as "079,16808,75703,610" hard to read; please use clear thousands separators or separate columns.
  2. [References] References [7] and [8] are duplicates of the same REANO paper, and references [9] and [10] are duplicates of the same TRACE paper; the duplicate entries should be merged.
  3. [Reference [16]] The DOI string contains a typo: "arvix" should be "arxiv".
  4. [Section 4.5] The paragraph uses both "prefix-computation" and "precomputation" for the same mechanism; please standardize the terminology.
  5. [Section 4.2] The backend examples show `HuggingFaceBackend`, `Seq2SeqLMBackend`, and `OpenAIBackend` but the paper does not state required package versions or how API keys are configured; a short note would improve reproducibility.
  6. [Section 5] The paper says example output is shown in the provided live notebooks, but no notebook link or figure is included in the text; adding a direct URL or a small output figure would help readers assess the expected result.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a systems/demo paper with no derivation-to-prediction chain; self-citations are contextual and non-load-bearing.

full rationale

This is a demonstration paper describing a software extension, not a paper that derives predictions from fitted parameters or from a uniqueness theorem. There is no equation whose output is identical to its input, no parameter fitted on one subset and then reported as a prediction on a closely related subset, and no load-bearing importation of a self-cited result to force a choice. The central claim is that PyTerrier-RAG makes RAG pipelines easy to build and evaluate, supported by code snippets, datasets, indices, and references to the existing PyTerrier platform. The self-citations to PyTerrier [31,32], REANO [7,8], TRACE [9,10], and precomputation [30] are used as ecosystem context or examples of expressible pipelines; they are not used as evidence that PyTerrier-RAG's claimed capabilities exist. The pre-built indices and readers are externally hosted artifacts, but the paper does not define the framework's claimed functionality in terms of those artifacts' outputs. Issues such as the syntactically incomplete worked-example lines (`bm25 %` and `e5 %`) are correctness or reproducibility concerns about whether the demonstration is executable as printed, not circularity: the claimed capability is not made true by definition or by a self-citation. No circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented physical entities. The framework introduces software abstractions (Reader, Concatenator, IRCOT, datatypes A/GA/Qc), but these are implementation components with no independent scientific evidence burden. The central claim rests on assumptions about prior software, datasets, and metrics.

assumptions (4)
  • domain assumption PyTerrier's relational operator semantics correctly compose pipeline stages as described in [31].
    Section 3 relies on prior PyTerrier semantics, which the paper does not re-derive or verify.
  • domain assumption The ten pre-processed benchmark datasets (NQ, TriviaQA, etc.) and their gold answers have been correctly transformed into PyTerrier relations.
    Section 4.4 asserts compatibility without showing validation of the preprocessing.
  • domain assumption The DPR-based EM and F1 implementations are correct and match the original DPR code.
    Section 4.5 states the measures are 'based on implementations provided in the DPR repository', which the paper does not reproduce or test.
  • domain assumption External HuggingFace artifacts ('pyterrier/ragwiki-terrier', 'pyterrier/ragwiki-e5.flex', 'terrierteam/t5fid_base_nq') will remain available and function as stated.
    Section 5's worked example depends on these hosted artifacts; no fallback or version pinning is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructing and Evaluating Declarative RAG Pipelines in PyTerrier." pith.science (2026). https://pith.science/paper/F627W5S6

@misc{pith2026250610802,
  author       = {Pith},
  title        = {Pith review of: Constructing and Evaluating Declarative RAG Pipelines in PyTerrier},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F627W5S6}},
  note         = {Machine review of arXiv:2506.10802}
}
read the original abstract

Search engines often follow a pipeline architecture, where complex but effective reranking components are used to refine the results of an initial retrieval. Retrieval augmented generation (RAG) is an exciting application of the pipeline architecture, where the final component generates a coherent answer for the users from the retrieved documents. In this demo paper, we describe how such RAG pipelines can be formulated in the declarative PyTerrier architecture, and the advantages of doing so. Our PyTerrier-RAG extension for PyTerrier provides easy access to standard RAG datasets and evaluation measures, state-of-the-art LLM readers, and using PyTerrier's unique operator notation, easy-to-build pipelines. We demonstrate the succinctness of indexing and RAG pipelines on standard datasets (including Natural Questions) and how to build on the larger PyTerrier ecosystem with state-of-the-art sparse, learned-sparse, and dense retrievers, and other neural rankers.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 15 canonical work pages

  1. [8]

    Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2024. REANO: Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (...

    work page doi:10.18653/v1/2024.acl-long.115 2024

  2. [10]

    Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2024. TRACE the Evidence: Constructing Knowledge-Grounded Reasoning Chains for Retrieval-Augmented Generation. InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computa...

    work page 2024

  3. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi

  4. [2]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. InProceedings of the 2013 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, ...

    work page 2013

  5. [3]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. InProceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, ACL 2017, Van- couver, Canada, July 30 - August 4, Volume 1: Long Papers, Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Lingu...

    doi:10.18653/v1/p17-1171 2017

  6. [4]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The Power of Noise: Redefining Retrieval for RAG Systems. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, U...

    arXiv 2024

  7. [5]

    Kaustubh D. Dhole. 2024. PyTerrier-GenRank: The PyTerrier Plugin for Rerank- ing with Large Language Models.CoRRabs/2412.05339 (2024). https://doi.org/ 10.48550/arXiv.2412.05339 arXiv:2412.05339

    work page Pith review arXiv doi:10.48550/arxiv.2412.05339 2024

  8. [6]

    Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In7th International Conference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum? id=r1l73iRqKm

    work page 2019

Show all 56 references
  1. [11]

    Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. InSIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, Virtual Event, Canada, July ...

    2021

  2. [12]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. CoRRabs/2312.10997 (2023). https://doi.org/10.48550/arXiv.2312.10997 arXi...

    2023 doi arXiv

  3. [13]

    Mitko Gospodinov, Sean MacAvaney, and Craig Macdonald. 2023. Doc2Query-: When Less is More. InAdvances in Information Retrieval - 45th European Con- ference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Pro- ceedings, Part II (Lecture Notes in Computer...

    2023 doi

  4. [14]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. InProceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online)...

    2020 doi

  5. [15]

    Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April...

    2021 doi

  6. [16]

    Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research.CoRRabs/2405.13576 (2024). https://doi.org/10.48550/arvix.2405.13576 arXiv:2405.13576

    2024 doi arXiv

  7. [17]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-Scale Similar- ity Search with GPUs.IEEE Trans. Big Data7, 3 (2021), 535–547. https: //doi.org/10.1109/TBDATA.2019.2921572

    2021

  8. [18]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehen- sion. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Ca...

    2017 doi

  9. [19]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ...

    2020 doi

  10. [20]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Ke- shav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

  11. [21]

    Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, Ch...

    2020

  12. [22]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and...

    2019 doi

  13. [23]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in N...

    2020

  14. [24]

    Hang Li, Ahmed Mourad, Shengyao Zhuang, Bevan Koopman, and Guido Zuc- con. 2021. Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls.CoRRabs/2108.11044 (2021). arXiv:2108.11044 https://arxiv.org/abs/2108.11044

    2021 arXiv

  15. [25]

    2021.Pretrained Trans- formers for Text Ranking: BERT and Beyond

    Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021.Pretrained Trans- formers for Text Ranking: BERT and Beyond. Morgan & Claypool Publishers. https://doi.org/10.2200/S01123ED1V01Y202108HLT053

    2021 doi

  16. [26]

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2024. RA-DIT: Retrieval-Augmented Dual Instruc- tion Tuning. InThe Twelfth International Conferenc...

    2024

  17. [27]

    Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval.Foundations and Trends in Information Retrieval3, 3 (2009), 225–331

    2009

  18. [28]

    Sean MacAvaney. 2025. Artifact Sharing for Information Retrieval Research. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3726302.3730147 SIGIR ’25, July 13–18, 2025, Padua, Italy Craig...

    2025

  19. [29]

    Sean MacAvaney and Craig Macdonald. 2022. A Python Interface to PISA!. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Cartere...

    2022

  20. [30]

    Sean MacAvaney and Craig Macdonald. 2025. On Precomputation and Caching in Information Retrieval Experiments with Pipeline Architectures. InProceedings of WOWS workshop at ECIR 2025. https://doi.org/10.48550/arXiv.2504.09984

    2025 doi arXiv

  21. [31]

    Craig Macdonald and Nicola Tonellotto. 2020. Declarative Experimentation in Information Retrieval using PyTerrier. InICTIR ’20: The 2020 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Norway, September 14-17, 2020, Krisztian Balog, Vi...

    2020

  22. [32]

    Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval. InCIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Austra...

    2021

  23. [33]

    Malkov and Dmitry A

    Yury A. Malkov and Dmitry A. Yashunin. 2016. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.CoRR abs/1603.09320 (2016). arXiv:1603.09320 http://arxiv.org/abs/1603.09320

    2016 arXiv

  24. [34]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Han- naneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. InProceedings of the 61st Annual Meeting of the Association for Comp...

    2023

  25. [35]

    Mackenzie, and Torsten Suel

    Antonio Mallia, Michal Siedlaczek, Joel M. Mackenzie, and Torsten Suel. 2019. PISA: Performant Indexes and Search for Academia. InProceedings of the Open- Source IR Replicability Challenge co-located with 42nd International ACM SIGIR Conference on Research and Development in I...

    2019

  26. [36]

    Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. CoRRabs/2101.05667 (2021). arXiv:2101.05667 https://arxiv.org/abs/2101.05667

    2021 arXiv

  27. [37]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. InFindings of the Association for Computational Linguis- tics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor...

    2023 doi

  28. [38]

    David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Stéphane Clinchant, and Vassilina Nikoulina. 2024. BERGEN: A Benchmarking Library for Retrieval-Augmented Generation. InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florid...

    2024

  29. [39]

    Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. RAGChecker: A Fine-grained Framework...

    2024

  30. [40]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing Retrieval-Augmented Large Language Models with Itera- tive Retrieval-Generation Synergy. InFindings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, Decemb...

    2023 doi

  31. [41]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024 doi

  32. [42]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal

  33. [43]

    Fangzheng Tian, Debasis Ganguly, and Craig Macdonald. 2025. Is Relevance Propagated from Retriever to Generator in RAG?. InAdvances in Information Re- trieval - 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6-10, 2025, Proceedings, Part I (L...

    2025

  34. [45]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  35. [46]

    Pan, and Kam-Fai Wong

    Hongru Wang, Wenyu Huang, Yang Deng, Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z. Pan, and Kam-Fai Wong. 2024. UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Sys- tems.CoRRabs/2401.13256 (2024). https://doi.org/10.48550/ARXIV....

    2024 doi arXiv

  36. [47]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text Embeddings by Weakly- Supervised Contrastive Pre-training.CoRRabs/2212.03533 (2022). https: //doi.org/10.48550/ARXIV.2212.03533 arXiv:2212.03533

    2022 doi arXiv

  37. [48]

    Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021. Pseudo- Relevance Feedback for Multiple Representation Dense Retrieval. InICTIR ’21: The 2021 ACM SIGIR International Conference on the Theory of Information Re- trieval, Virtual Event, Canada, July 11, 2021,...

    2021

  38. [49]

    Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. InProceedings of the 40th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, ...

    2017

  39. [50]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge- Intensive Multi-Step Questions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jor...

    2023 doi

  40. [51]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Mod- els. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Ope...

    2023

  41. [52]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  42. [53]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Pr...

    2023

  43. [55]

    Cohen, Rus- lan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Rus- lan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Lang...

    2018

  44. [2018]

    FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, ...

    2018 doi

  45. [2020]

    In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

    BERTScore: Evaluating Text Generation with BERT. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=SkeHuCVFDr

    2020

  46. [2022]

    MuSiQue: Multihop Questions via Single-hop Question Composition.Trans. Assoc. Comput. Linguistics10 (2022), 539–554. https://doi.org/10.1162/TACL_A_ 00475

    2022 doi

  47. [2023]

    https://doi.org/10.48550/ARXIV.2310

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.CoRRabs/2310.03714 (2023). https://doi.org/10.48550/ARXIV.2310. 03714 arXiv:2310.03714

    2023 doi arXiv

  48. [2024]

    InThe Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self- Reflection. InThe Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https: //openreview.net/forum?id=hSyW5go0v8

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.