UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses

Gabriel Iturra-Bocaz; Ivica Kostric; Krisztian Balog; Mariam Arustashvili; Weronika {\L}ajewska

REVIEW 4 major objections 4 minor 1 cited by

UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that its nugget-based GINGER pipeline, with original-query-plus-few-rewrites retrieval, delivers grounded responses and shows clear diminishing returns on adding more documents to reranking and generation.

desk verdict Solid challenge report with one clean retrieval result and an overclaimed saturation effect that its own Table 2 contradicts. read the letter →

arxiv 2506.22210 v1 pith:NSWKXCUC submitted 2025-06-27 cs.IR

Weronika {L}ajewska , Ivica Kostric , Gabriel Iturra-Bocaz , Mariam Arustashvili , Krisztian Balog This is my paper

classification cs.IR

keywords retrieval-augmentedgenerationinformationnuggetsqueryrewritingGINGERLiveRAGchallengecontextcurationresponserecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a retrieval-augmented generation system can get more out of retrieved documents by first distilling them into atomic units of relevant information, called information nuggets, and then clustering, ranking, and summarizing those nuggets instead of feeding whole passages to the generator. On the retrieval side, it claims that the best recall comes from keeping the original query in the search string and adding only a few diverse sub-query rewrites: a single rewrite hurts, and more than about five rewrites adds nothing. On the generation side, it claims that response quality improves only up to a point as more documents are used for reranking and generation, so the system can cut its most expensive stages and stay within a two-hour processing budget without losing measurable quality. A sympathetic reader would care because this points to a concrete efficiency-quality frontier for real-time answering systems.

What carries the argument

The object that carries the argument is the information nugget: a minimal, atomic unit of relevant information, extracted as a verbatim span from a retrieved passage. GINGER's generation side is built on four operations on these nuggets: detecting them in the top $m$ reranked passages, clustering them into query facets with BERTopic, ranking the facet clusters with DuoT5, and summarizing each top cluster into one sentence that a final LLM pass rephrases into fluent prose without adding content. The retrieval side is built on an intermediate-answer query rewrite, reciprocal rank fusion of BM25 and dense embeddings, and MonoT5 plus DuoT5 reranking. The nugget representation is what lets the pipeline drop redundant context while keeping facts, sources, and length constraints visible at every step.

What would settle it

Re-run the configuration comparison (for example $k=20$, $k=40$, $k=50$ with $m=10$ or $m=20$) on a test set where each query has many judged passages and human-verified nuggets; if the broadest reranking setup then clearly outscores the narrower ones, the reported saturation is an artifact of sparse reference nuggets rather than a fixed property of the pipeline.

Watch

Extended reading notes

Core claim

The central claim is that GINGER, a three-stage pipeline for document retrieval, context curation, and response generation, produces grounded, attributable, length-limited answers whose quality tracks retrieval recall but saturates early with respect to reranking and generation effort. The paper's headline evidence is that concatenating the original query with three to five LLM-generated rewrites raises Recall@500 on the TREC RAG'24 test collection from 0.320 to about 0.397-0.400, whereas using a single rewrite alone drops recall to 0.217. It also reports that raising the number of passages sent to pairwise reranking and generation from $(k=20, m=5)$ through $(k=50, m=20)$ improves the automatic nugget score only from 0.350 to 0.406, with the $(k=40, m=10)$ configuration achieving 0.397 in roughly half the time of the best configuration. The paper concludes that the practical operating point is a broad retrieval pool produced by the original query plus a small number of rewrites, followed by targeted reranking and generation on a limited set of passages.

Load-bearing premise

The comparisons among response configurations assume that the automatic nugget-based scorer, which builds its gold standard from only the two ground-truth documents provided per synthetic query, fairly measures answer quality; the paper itself notes this undervalues well-grounded answers to open-ended queries when the reference nuggets are sparse.

Editorial extensions

If this is right

Combining the original query with three to five diverse rewrites raises retrieval recall from 0.320 to roughly 0.400, while any single rewrite alone harms recall.
Response quality measured by the automatic nugget score saturates: the fastest configuration tested $(k=20, m=5)$ scores 0.350, and the heaviest $(k=50, m=20)$ scores 0.406, with intermediate configurations nearly matching it in far less time.
A broad retrieval pool plus a narrow, nugget-level generation context is the operating point that balances completeness against the two-hour LiveRAG time budget.
The expensive pairwise-reranking and per-cluster summarization stages are the places to cut when latency matters, because the quality curve is flat beyond their early cutoffs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the sparse-nugget scoring bias is real, then the optimal reranking cutoff for open-ended queries is likely larger than the paper's reported optimum, because the extra relevant content that a broader rerank pool retrieves is exactly what the two-document gold nuggets fail to credit.
A testable extension would hold retrieval fixed and vary only the nugget-detection prompt or the clustering granularity; the paper's design predicts that response scores move mainly with the number of facets covered, not with more source documents.
The nugget-cluster-summarize-refine template could be ported to other LLMs and retrievers, and the reported saturation suggests that system builders should tune the number of reranked items before tuning the generator.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful systems report from the LiveRAG challenge. The one solid empirical contribution is that concatenating the original query with a handful of LLM-generated rewrites beats the original alone on Recall@500 (0.400 vs 0.320 on TREC RAG'24). The response-quality saturation claim in the abstract does not survive contact with its own Table 2.

What's genuinely new is modest: the authors take their GINGER pipeline (SIGIR'25) and their multi-query rewriting method (SIGIR'24), combine them, and sweep the number of rewrites and two document cutoffs (k for DuoT5 reranking, m for nugget detection). The system description is transparent, prompts are in the appendix, and the retrieval evaluation uses external TREC RAG'24 judgments. That part is solid.

The soft spot is Table 2. The best V_strict (0.406) is at the largest settings tested (k=50, m=20). k=20,m=10 scores 0.404 and k=40,m=10 scores 0.397. No error bars, no significance tests, one run on 100 synthetic queries. The m=5 vs m=10 gap (0.350 vs 0.404) actually shows more generation documents help, not saturate. So the abstract's 'increasing the number of documents... reduces effectiveness' is not supported.

To the authors' credit, they acknowledge the AutoNuggetizer limitation in Section 4.3: nuggets extracted from only two ground-truth documents can under-score well-grounded broad responses. But they then use that same constrained evaluation to justify a trade-off claim. That is the load-bearing inconsistency. The fact the best score occurs at the broadest k,m also aligns with the sparsity bias favoring narrower configs, so the direction of the claimed effect is suspect.

Minor: the DataMorgana set is small and unreleased, results are single-run. Self-citation is fair here — GINGER and the multi-query rewriting method are their published systems, and they cite related external work.

Who this is for: engineers building RAG pipelines who want a concrete example of query rewriting plus nugget-based generation, and anyone using AutoNuggetizer should read Section 4.3. It deserves a serious referee — the retrieval result is clean and the response-evaluation caveat is important. I'd accept it for review, with a request to fix the saturation claim and add statistical support.

Referee Report

4 major / 4 minor

Summary. The paper describes the UiS-IAI submission to the SIGIR'25 LiveRAG Challenge. The system, GINGER, is a modular retrieval-augmented generation pipeline composed of three stages: document retrieval (query rewriting with Falcon, sparse and dense retrieval, reciprocal rank fusion, and MonoT5/DuoT5 reranking), context curation (nugget detection, BERTopic clustering, facet cluster ranking), and response generation (cluster summarization and fluency refinement). The authors evaluate query rewriting on TREC RAG'24 using Recall@500, and response generation on 100 DataMorgana queries using the AutoNuggetizer V_strict measure under different reranking (k) and generation (m) cutoffs. They report that concatenating the original query with multiple rewrites improves recall over the original query alone, and they claim that increasing the number of documents used for reranking and generation beyond a certain point reduces effectiveness without improving response quality, motivating the final submission configuration k=40, m=10.

Significance. If the findings hold, the paper offers a useful proof-of-concept that nugget-based context curation can be combined with multi-query rewriting in a time-constrained RAG challenge. The authors provide concrete results on an established retrieval benchmark, make all prompts available in the appendix, and are candid about the limitations of the AutoNuggetizer evaluation. However, the main generalization about the trade-off between document count and response quality is not supported by the reported numbers, and the evaluation protocol is too coarse to support the central claim. The paper would be strengthened by revised claims, statistical support, and a more careful treatment of the evaluation bias it identifies.

major comments (4)

[Abstract / Table 2] The abstract states that 'increasing the number of documents used for reranking and generation beyond a certain point reduces effectiveness, without improving response quality.' Table 2 does not exhibit this pattern: the highest V_strict (0.406) occurs at the largest tested settings (k=50, m=20); increasing m from 5 to 10 (at k=20) improves V_strict from 0.350 to 0.404; and the only comparison that isolates k (k=20,m=10 vs k=40,m=10) shows a difference of 0.007 (0.404 vs 0.397), which is not shown to be meaningful. The claimed saturation or decline is therefore not established. The abstract and Section 4.3 should be revised to report a more cautious conclusion, such as 'no clear improvement beyond moderate cutoffs', or additional experiments should be provided to support the claimed trade-off.
[Section 4.2 / Table 2] No variance information is reported for V_strict. With 100 queries, the differences among 0.397, 0.404, and 0.406 are likely within noise, so statements such as 'slightly higher' and 'similar scores' are not supported beyond the point estimates. The authors should provide per-query score distributions, bootstrap confidence intervals, or a paired significance test (e.g., Wilcoxon signed-rank) for the key comparisons, or explicitly state that the configurations are not statistically separable.
[Section 4.3] The paper itself concedes that AutoNuggetizer derives vital nuggets from only the two ground-truth documents per DataMorgana query, and that this can under-score well-grounded responses with sparse reference nuggets while rewarding responses that happen to align exactly with the reference nuggets. This bias is exactly in the direction that would make broader k and m configurations look non-beneficial, because a response that covers relevant content outside the two reference documents receives no credit. Since V_strict is the only response-quality metric in Table 2, the central efficiency/quality trade-off claim rests on a metric the authors describe as constrained. Supplementary analysis, such as manual judgments on a sample or evaluation with additional reference passages, is needed to show that the ranking of configurations is not an artifact of the narrow nugget set.
[Section 4.3] The decision to submit k=40, m=10 despite k=20, m=10 having a higher V_strict (0.404 vs 0.397) and a similar time estimate (41 vs 42 minutes) is justified by 'topic coverage and response diversity', but no measurement of diversity or coverage is reported. Either provide a concrete diversity or coverage metric supporting the choice, or present the decision as an engineering choice motivated by robustness concerns rather than as an outcome of the evaluated quality metric.

minor comments (4)

[Section 3.1, Eq. (1)] The notation q'_i is used but not formally defined, and the text says the final search string is created by concatenating the rewrites with the original query; it is unclear whether this is a single long query string or a set of queries whose results are fused, given that the retrieval stage uses reciprocal rank fusion.
[Table 1, Footnote 13] The statement that findings on TREC RAG'24 are 'expected to generalize' to the DataMorgana test collection is a hypothesis; it should be explicitly framed as such rather than presented as a definite conclusion.
[Section 4.3 / Table 2] The 'Time estimate' column is not defined; specify the hardware configuration, batch sizes, and number of parallel processes assumed for these estimates, since Section 3.4 describes different GPU allocations during the challenge.
[References] There are typographical errors in the bibliography, e.g., 'ALC' in reference [17] should be 'ACL'; a careful proofread of the reference list is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's results are empirical measurements against external benchmarks, not derivations from fitted inputs.

full rationale

The paper's load-bearing claims are empirically evaluated. Query rewriting effectiveness is measured as Recall@500 on the TREC RAG'24 collection using trec_eval with the original-query baseline (Section 4.2, Table 1); this is an external, published test collection, not a quantity defined by the pipeline. Response quality is measured with the AutoNuggetizer framework from TREC RAG'24, with V_strict computed against vital nuggets derived from ground-truth passages (Section 4.2, Table 2). No parameter is fitted to the evaluation set and then renamed as a prediction; the configuration choices (k and m) are ablated rather than optimized against V_strict, and the paper explicitly notes the best-scoring configuration was not selected due to time constraints. The self-citations to GINGER [21] and the authors' multi-query rewriting work [19] describe components that are then independently tested in this paper; the prior GINGER result on TREC RAG'24 is an external benchmark claim, and no argument in the current paper reduces to an unverified self-citation. The conceded AutoNuggetizer limitation (Section 4.3) is a measurement-validity caveat—sparse ground-truth nuggets may under-score well-grounded responses—but this is not a circularity: the evaluation metric is external to the method and its known bias does not make the pipeline's output equivalent to the metric's input. Accordingly, the derivation chain is self-contained with respect to the circularity patterns considered.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper contains no mathematical derivation; its conclusions rest on hand-chosen pipeline cutoffs (l, k, m) and domain assumptions about evaluation artifacts. The DataMorgana-generated 100-query set is the sole basis for the V_strict numbers, and TREC RAG recall is assumed to transfer to the LiveRAG corpus. The LLM's obedience to the extraction and fluency prompts is also assumed. The system itself introduces no invented entities.

free parameters (3)

number of query rewrites l = 3 (final submission); 5 gave best recall 0.400
l is varied in Table 1 (1, 3, 5, 10); the central claim that a few rewrites suffice depends on this hand-chosen value, and the best recall is at l=5 while the final submission uses l=3.
pairwise reranking cutoff k = 40 (final submission)
k is varied in Table 2 (20, 40, 50); k=50/m=20 gives best V_strict 0.406, but k=40 was selected for time constraints, even though k=20/m=10 scored 0.404, slightly above the selected configuration.
response generation cutoff m = 10 (final submission)
m is varied in Table 2 (5, 10, 20); higher m improves quality only slightly (m=20 with k=50 gives 0.406 vs 0.404 at m=10), and m=10 was selected to limit Falcon API calls.

assumptions (3)

domain assumption DataMorgana-generated questions and their two ground-truth passages are a valid, unbiased evaluation set for LiveRAG response quality
Section 4.1: the 100-instance test set is generated by the authors via the DataMorgana API; all V_strict scores in Table 2 rest on this reference set, and the paper itself notes that two-document ground truth may be sparse for open-ended queries.
domain assumption Recall@500 on TREC RAG'24 estimates retrieval effectiveness on the LiveRAG corpus
Sections 4.1 to 4.3: TREC RAG'24 uses a different collection (MS MARCO v2.1) than the challenge corpus; the paper transfers these results to the LiveRAG pipeline without a direct comparison, as acknowledged in footnote 13.
domain assumption The Falcon3-10B model performs prompt-instructed nugget extraction and summarization faithfully and deterministically
Sections 3.2 to 3.3 and Appendix B: the pipeline assumes the LLM copies spans exactly 'without changing any part of the original text' and does not add content in summarization; no verification of extraction faithfulness is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses." pith.science (2026). https://pith.science/paper/NSWKXCUC

@misc{pith2026250622210,
  author       = {Pith},
  title        = {Pith review of: UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSWKXCUC}},
  note         = {Machine review of arXiv:2506.22210}
}

read the original abstract

Retrieval-augmented generation (RAG) faces challenges related to factual correctness, source attribution, and response completeness. The LiveRAG Challenge hosted at SIGIR'25 aims to advance RAG research using a fixed corpus and a shared, open-source LLM. We propose a modular pipeline that operates on information nuggets-minimal, atomic units of relevant information extracted from retrieved documents. This multistage pipeline encompasses query rewriting, passage retrieval and reranking, nugget detection and clustering, cluster ranking and summarization, and response fluency enhancement. This design inherently promotes grounding in specific facts, facilitates source attribution, and ensures maximum information inclusion within length constraints. In this challenge, we extend our focus to also address the retrieval component of RAG, building upon our prior work on multi-faceted query rewriting. Furthermore, for augmented generation, we concentrate on improving context curation capabilities, maximizing the breadth of information covered in the response while ensuring pipeline efficiency. Our results show that combining original queries with a few sub-query rewrites boosts recall, while increasing the number of documents used for reranking and generation beyond a certain point reduces effectiveness, without improving response quality.

Figures

Figures reproduced from arXiv: 2506.22210 by the authors.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SIGIR 2025 -- LiveRAG Challenge Report
cs.CL 2025-07 conditional novelty 3.0 of 10

In the SIGIR 2025 LiveRAG Challenge, all 25 active RAG teams beat the no-RAG baseline on LLM-judged correctness, and LLM scores correlated with human scores at r=0.88.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages · cited by 1 Pith paper

[1]

Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad

work page
[2]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023. WikiHowQA: A Comprehensive Benchmark for Multi- Document Non-Factoid Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’23). 5291–5314

work page 2023
[3]

Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses MS MARCO: A Human Generated MAchine Reading Comprehension Dataset. arXiv:1611.09268 [cs.CL]

arXiv 2016
[4]

Cormack, Charles L

Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’09)

work page 2009
[5]

Dhole and Eugene Agichtein

Kaustubh D. Dhole and Eugene Agichtein. 2024. GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation. In Advances in Information Retrieval: 46th European Conference on Information Retrieval (ECIR ’24). 326––335

work page 2024
[6]

Naghmeh Farzi and Laura Dietz. 2024. TREMA-UNH at TREC: RAG Systems and RUBRIC-style Evaluation. In The Thirty-Third Text REtrieval Conference Proceedings (TREC ’24)

work page 2024
[7]

Simone Filice, Guy Horowitz, David Carmel, Zohar Karnin, Liane Lewin-Eytan, and Yoelle Maarek. 2025. Generating Diverse Q&A Benchmarks for RAG Evalua- tion with DataMorgana. arXiv:2501.12789 [cs.CL]

arXiv 2025
[8]

Maik Fröbe, Lukas Gienapp, Harrisen Scells, Eric Oliver Schmidt, Matti Wieg- mann, Martin Potthast, and Matthias Hagen. 2024. Webis at TREC 2024: Biomed- ical Generative Retrieval, Retrieval-Augmented Generation, and Tip-of-the- Tongue Tracks. In The Thirty-Third Text REtrieval Conference Proceedings (TREC ’24)

work page 2024

Show all 43 references

[9]

Matteo Gabburo, Nicolaas Paul Jedema, Siddhant Garg, Leonardo F. R. Ribeiro, and Alessandro Moschitti. 2024. Measuring Retrieval Complexity in Question Answering Systems. In Findings of the Association for Computational Linguistics: ACL 2024 (ACL ’24’). 14636–14650

2024
[10]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-Augmented Genera- tion for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL]

2023 arXiv
[11]

Lukas Gienapp, Harrisen Scells, Niklas Deckers, Janek Bevendorff, Shuai Wang, Johannes Kiesel, Shahbaz Syed, Maik Fröbe, Guido Zuccon, Benno Stein, Matthias Hagen, and Martin Potthast. 2024. Evaluating Generative Ad Hoc Information Retrieval. In Proceedings of the 47th Interna...

2024
[12]

Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News Summarization and Evaluation in the Era of GPT-3. arXiv:2209.12356 [cs.CL]

2023 arXiv
[14]

Yizheng Huang and Jimmy Huang. 2024. A Survey on Retrieval-Augmented Text Generation for Large Language Models. arXiv:2203.05794 [cs.CL]

2024 arXiv
[15]

Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL ’21). 874–880

2021
[16]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (2023), 1–38

2023
[17]

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

2024
[18]

Bevan Koopman and Guido Zuccon. 2023. Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness. In Findings of the Association for Computational Linguistics: EMNLP 2023 (EMNLP ’23) . 15012– 15022

2023
[19]

Ivica Kostric and Krisztian Balog. 2024. A Surprisingly Simple yet Effective Multi- Query Rewriting Method for Conversational Passage Retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24) . 2271–2275

2024
[20]

Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022. Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade- off in Abstractive Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguist...

2022
[21]

Weronika Łajewska and Krisztian Balog. 2025. GINGER: Grounded Information Nugget-Based Generation of Responses. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25)

2025
[22]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings ...

2020
[23]

Nelson Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiability in Generative Search Engines. In Findings of the Association for Computational Linguistics: EMNLP 2023 (EMNLP ’23) . 7001–7025

2023
[24]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

2024
[25]

Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented Language Models: a Survey. arXiv:2302.07842 [cs.CL]

2023 arXiv
[26]

Golbus, and Javed A

Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, and Javed A. Aslam. 2012. IR system evaluation using nugget-based test collections. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM ’12) . 393–402

2012
[27]

Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The Expando-Mono- Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. arXiv:2101.05667 [cs.IR]

2021 arXiv
[28]

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! arXiv:2312.02724 [cs.IR]

2023 arXiv
[29]

Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411.09607 [cs.IR]

2024 arXiv
[30]

Zackary Rackauckas. 2024. RAG-Fusion: a New Take on Retrieval-Augmented Generation. International Journal on Natural Language Computing 13, 1 (2024), 37–47

2024
[31]

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-Context Retrieval-Augmented Lan- guage Models. Transactions of the Association for Computational Linguistics 11 (2023), 1316–1331

2023
[32]

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, and Michael Collins. 2021. Measuring Attribution in Natural Language Generation Models. Computational Linguistics 49, 4 (2021), 777–840

2021
[33]

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: Retrieval-Augmented Black-Box Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

2023
[34]

Chilton, and Kathleen McKeown

Melanie Subbiah, Sean Zhang, Lydia B. Chilton, and Kathleen McKeown. 2024. Reading Subtext: Evaluating Large Language Models on Short Story Summariza- tion with Writers. arXiv:2403.01061 [cs.CL]

2024 arXiv
[35]

Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors. In Proceedings of the 61st Annual Meetin...

2023
[36]

Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. 2022. Inves- tigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries. In Proceedings of the 2022 Conference of th...

2022
[37]

Voorhees

Ellen M. Voorhees. 2003. Overview of the TREC 2003 Question Answering Track. In The Twelfth Text REtrieval Conference Proceedings (TREC ’03)

2003
[38]

Conroy, Neil Molino, Julia Yang, and Mike Green

Yue Wang, John M. Conroy, Neil Molino, Julia Yang, and Mike Green. 2024. Laboratory for Analytic Sciences in TREC 2024 Retrieval Augmented Generation Track. In The Thirty-Third Text REtrieval Conference Proceedings (TREC ’24)

2024
[39]

Orion Weller, Kyle Lo, David Wadden, Dawn Lawrie, Benjamin Van Durme, Arman Cohan, and Luca Soldaini. 2024. When do Generative Query and Docu- ment Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets. In Findings of the Association for Computational...

2024
[40]

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv:2310.04408 [cs.CL]

2023 arXiv
[41]

Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, and Jing Xiao. 2023. PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter. In Find- ings of the Association for Computational Lingui...

2023
[42]

Trippas, Jeff Dalton, and Filip Radlinski

Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2023. Con- versational Information Seeking. Foundations and Trends in Information Retrieval 17, 3-4 (2023), 244–456

2023
[43]

Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö Arik. 2024. Chain of Agents: Large Language Models Collaborating on Long- Context Tasks. arXiv:2406.02818 [cs.CL] Weronika Łajewska, Ivica Kostric, Gabriel Iturra-Bocaz, Mariam Arustashvili, and Kriszt...

2024 arXiv
[2023]

Proceedings of the 4th New Frontiers in Summarization Workshop (2023), 68–74

From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting. Proceedings of the 4th New Frontiers in Summarization Workshop (2023), 68–74

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad

work page

[2] [2]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023. WikiHowQA: A Comprehensive Benchmark for Multi- Document Non-Factoid Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’23). 5291–5314

work page 2023

[3] [3]

Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses MS MARCO: A Human Generated MAchine Reading Comprehension Dataset. arXiv:1611.09268 [cs.CL]

arXiv 2016

[4] [4]

Cormack, Charles L

Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’09)

work page 2009

[5] [5]

Dhole and Eugene Agichtein

Kaustubh D. Dhole and Eugene Agichtein. 2024. GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation. In Advances in Information Retrieval: 46th European Conference on Information Retrieval (ECIR ’24). 326––335

work page 2024

[6] [6]

Naghmeh Farzi and Laura Dietz. 2024. TREMA-UNH at TREC: RAG Systems and RUBRIC-style Evaluation. In The Thirty-Third Text REtrieval Conference Proceedings (TREC ’24)

work page 2024

[7] [7]

Simone Filice, Guy Horowitz, David Carmel, Zohar Karnin, Liane Lewin-Eytan, and Yoelle Maarek. 2025. Generating Diverse Q&A Benchmarks for RAG Evalua- tion with DataMorgana. arXiv:2501.12789 [cs.CL]

arXiv 2025

[8] [8]

Maik Fröbe, Lukas Gienapp, Harrisen Scells, Eric Oliver Schmidt, Matti Wieg- mann, Martin Potthast, and Matthias Hagen. 2024. Webis at TREC 2024: Biomed- ical Generative Retrieval, Retrieval-Augmented Generation, and Tip-of-the- Tongue Tracks. In The Thirty-Third Text REtrieval Conference Proceedings (TREC ’24)

work page 2024