Pith. sign in

REVIEW 4 major objections 6 minor 57 references

Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that multi-round retrieval-augmented generation can be made about twice as fast by reusing cached document representations and verifying drafted phrases in parallel, without losing answer quality.

desk verdict Plausible acceleration scheme for adaptive RAG with a real soft spot: quality preservation is only shown for one of four baselines, and even that table has mixed results. read the letter →

arxiv 2505.12731 v2 pith:JIFZOF72 submitted 2025-05-19 cs.AI

Jie Ou , Jinyu Guo , Shuaihong Jiang , Zhaokun Wang , Libo Qin , Shunyu Yao , Wenhong Tian This is my paper · ORCID
classification cs.AI
keywords adaptiveretrieval-augmentedgenerationKVcachereuseinstruction-drivendeduplicationparalleldecodingspeculativemulti-hopquestionansweringinferenceaccelerationrepresentationreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-round (adaptive) retrieval-augmented generation repeatedly retrieves and re-encodes documents that largely overlap from round to round, wasting computation. This paper proposes IDR2, a model-agnostic wrapper that reuses cached key-value representations of documents seen in earlier rounds, uses generated instructions to tell the model which cached content is relevant, and verifies several draft tokens at once during decoding. The authors report that the wrapper accelerates the prefilling stage by 2.79 times and decoding by 2.33 times on average, cutting end-to-end latency by about 2 times across four A-RAG methods and four datasets, while keeping answer quality at baseline level and sometimes improving it. If the claim holds, adaptive RAG becomes practical for latency-sensitive applications because the extra retrieval rounds no longer double or triple the cost.

What carries the argument

The load-bearing object is the key-value (KV) cache shared across retrieval iterations. CICS treats the KV pairs of a document as a reusable representation, so a document that appears again in a later round is loaded from memory instead of being re-encoded. IDGR then uses explicit linguistic instructions, listing document IDs and relevance scores, to make the model attend to the current round's relevant cached content and ignore redundant or contaminated parts. IGPG builds an n-gram-based draft language model from the retrieved documents and, at each autoregressive step, proposes a block of draft tokens that the LLM accepts or rejects in a single forward pass, achieving parallel generation whenever retrieved text overlaps with the answer.

What would settle it

Run IDR2 on a multi-hop QA set where the same document appears in consecutive rounds but the second round's question requires reading a different part of that document, and compare exact match and F1 against the unchanged A-RAG baseline; if the cached-and-instructed run is statistically worse on any dataset, the 'maintaining equal generation quality' claim fails. A more direct check is to replace the reused KV pairs with freshly recomputed pairs while keeping the IDGR instructions identical and measure output divergence: material divergence means the cached representation, not the instructions, is carrying the quality loss.

Watch

Extended reading notes

Core claim

The paper's central claim is that the redundant computation in adaptive RAG is largely caused by overlapping documents across retrieval rounds and can be removed without sacrificing quality. It presents a three-part mechanism: CICS stores the key-value representations of every retrieved document and reuses them when the same document reappears; IDGR adds an automatically generated natural-language instruction that identifies relevant and irrelevant cached documents and their relevance rankings, counteracting the context contamination that self-attention introduces into cached representations; and IGPG constructs draft token sequences from phrase fragments already present in the retrieved documents and has the LLM verify them in parallel, reducing the number of autoregressive steps. Experiments across four A-RAG baselines, four datasets, and LLaMA-2/Vicuna models at 7B and 13B scales report 1.75x to 4.72x prefilling speedups, 1.49x to 4.00x decoding speedups, and 1.31x to 3.53x end-to-end speedups while maintaining generation quality.

Load-bearing premise

The load-bearing premise is that a document's cached key-value representation, computed in an earlier round's context, can be made just as useful as a freshly computed representation through instruction-based guidance.

Editorial extensions

If this is right

  • End-to-end latency of multi-round RAG drops by about half, so systems can afford more retrieval and verification rounds within the same response-time budget.
  • The speedup grows with the overlap between adjacent retrieval rounds; DRAGIN, whose query refinement increases query similarity, shows the largest prefilling gain, up to 4.72x.
  • Because IDR2 is a wrapper, it can be applied to existing A-RAG methods without retraining or changing their retrieval decisions.
  • Quality recovery comes from the instruction module: without IDGR, reusing cached representations lowers exact match from 22.5 to 20.3 on 2WikiMultihopQA with LLaMA2-7B, and IDGR raises it to 25.4.
  • The same acceleration pattern holds for both BM25 and dense SGPT retrievers, suggesting the method does not depend on a particular retrieval signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the cache policy adaptive: estimate the predicted overlap between rounds and only invoke CICS when the overlap is high enough to offset the lookup and instruction overhead.
  • IGPG's draft source and standard speculative decoding are complementary; retrieved documents offer high-quality drafts for grounded phrases like names and dates, while a small draft model covers free-form tokens, so combining them could yield larger or more robust speedups.
  • The quality-preservation claim likely hinges on the model's instruction-following strength; the same IDGR directives may need stronger wording or finer-grained relevance cues for weaker instruction-tuned or base models.
  • Cross-task transfer of instruction phrasing is an open testable question: the paper uses document IDs and relevance rankings, but question-specific instructions such as 'focus on the part that answers X' may further improve quality on multi-hop tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces IDR2, a model-agnostic acceleration framework for Adaptive RAG. It has three components: CICS reuses cached KV representations of documents that reappear across retrieval rounds, IDGR adds instruction-based relevance/irrelevance cues to the prompt to mitigate contamination from cached representations, and IGPG constructs n-gram phrase fragments from retrieved documents and verifies them in parallel to reduce autoregressive decoding steps. Experiments are run with LLaMA2-7B/13B and Vicuna-13B on 2WikiMultihopQA, HotpotQA, StrategyQA, and IIRC, and report prefilling/decoding/end-to-end speedups for FLRAG, FSRAG, FLARE, and DRAGIN. The authors claim 2.79x prefilling, 2.33x decoding, and 2.0x end-to-end average acceleration with equal generation quality.

Significance. If the claims held, the work would be a useful practical contribution to serving multi-round RAG. Strengths include direct speedup measurements on multiple A-RAG methods and open-source LLMs, a three-module decomposition with an ablation isolating each module's effect (Table 5), and attention to retriever variation (Table 4). The paper does not fit parameters to match results, so circularity is not a concern. However, the significance is currently limited by incomplete evidence for the quality-preservation half of the central claim and by an under-specified measurement protocol.

major comments (4)
  1. [Section 4.5, Table 2] The claim that IDR2 preserves generation quality across A-RAG approaches is not supported for FLRAG, FSRAG, or FLARE: Table 1 reports speedup only for these baselines, and Table 2 reports quality metrics (EM/F1/Acc) only for DRAGIN. Because the abstract and Section 1 assert 'maintaining equal generation quality' for various A-RAG approaches, quality tables for all four baselines are needed; without them the central composite claim is only half verified.
  2. [Section 4.5, Table 2] Even for the DRAGIN baseline, the quality evidence is mixed: LLaMA2-13B HotpotQA EM falls from 31.6 to 29.9, Vicuna-13B IIRC EM falls from 23.90 to 21.59 and F1 from 28.11 to 26.28, and LLaMA2-7B StrategyQA accuracy falls from 65.10 to 62.50. Since no error bars, test-set sizes, or significance tests are reported, the statement that quality is 'maintained' is not established; the authors should report variances or confidence intervals, the number of samples per metric, and a statistical comparison.
  3. [Sections 3.1-3.2, Table 5] The cached KV representations reused by CICS are context-dependent by the paper's own description in Section 3.2 ('the Key-Value representation of each document incorporates information from previously processed documents through self-attention'). Reusing them across rounds introduces a systematic representation mismatch; IDGR instructions can bias attention but cannot erase already-encoded cross-document information. The ablation in Table 5 shows CICS alone lowers EM from 22.5 to 20.3, so the 'equal quality' claim rests on IDGR's correction. The paper should include a direct test of this correction, e.g., comparing IDR2 against a variant that recomputes cached K/V in the current context on at least one dataset, and discuss whether the correction is robust across tasks and models.
  4. [Section 4.4, Table 1] The speedup measurements lack a precise protocol: the paper does not state how many test examples were timed, how many repeated runs were averaged, whether GPU clocks, batching, and sequence lengths were controlled, or whether prefilling and decoding times were measured at the framework level; no variance or confidence intervals are reported. In addition, the abstract's average prefilling speedup of 2.79x does not match the unweighted average of the Table 1 prefilling ratios (approximately 2.7x); please clarify the weighting or correct the number.
minor comments (6)
  1. [Figure 1(b)] Define the overlap ratio precisely and report the overlap for each of rounds 2 and 3 separately; the caption is ambiguous about the denominator and sample size.
  2. [Equations (1)-(2)] The notation for A<t and the cache update is confusing: Kt/Vt are used both for document representations and for the generated-token cache; please disambiguate the two uses.
  3. [Equation (5)] Clarify the ranges of indices j and the meaning of the barred token ar a^{j-1}; the current notation is hard to follow because the draft sequence and the verified sequence are not clearly distinguished.
  4. [Table 2] The table has formatting issues (e.g., '22.633.02' and '18.522.59' appear to merge two numbers), and the note 'underlining indicates better results' is not visible in the table; please reformat.
  5. [Sections 1 and 6] Section 1 says '2.0 times on average' end-to-end acceleration while Section 6 says 'up to 2.0x acceleration'; Table 1 ranges from 1.31x to 3.53x, so the intended claim should be made consistent.
  6. [Section 5.2] The case study reports wall-clock times without stating whether they come from the same measurement protocol as Table 3; please align the reporting or note that the case study is illustrative only.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: speedups are direct measurements, quality comparisons are empirical, and no fitted parameter or self-citation chain is renamed as a prediction.

full rationale

This is an empirical systems paper with no derivation chain that reduces to its own inputs. The acceleration numbers (e.g., 2.79x prefilling, 2.33x decoding, 2.0x end-to-end) are wall-clock speedup measurements against reproduced baselines, not quantities derived from fitted parameters. The IDGR module's instructions use retriever relevance scores or model-based scoring as inputs, but these are not fitted to the reported quality or speed outcomes, so there is no fitted input being relabeled as a prediction. CICS reuses cached key-value representations by construction, and the paper explicitly acknowledges that those representations are contaminated by prior-round documents (Section 3.2), which is a robustness limitation rather than a circular step. The main evidentiary weakness is that quality metrics are reported only for DRAGIN (Table 2), while acceleration is reported for all four A-RAG baselines; this leaves the 'maintaining equal generation quality' claim under-supported for FLRAG, FSRAG, and FLARE, and the mixed quality results in Table 2 are a correctness/evidence concern, not a circularity concern. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The conclusion that the workflow is accelerated is independently measurable and not forced by definition. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The system relies on three domain assumptions: KV-cache reuse remains approximately valid or correctable under changing context; generated tokens largely appear as n-grams in retrieved documents; and LLMs can follow instructions to filter cached content without re-encoding. The first is documented as imperfect in the paper (Section 3.2, Table 5), making it the most fragile. There are no invented entities. The unstated draft length M and n-gram window N are free hyperparameters.

free parameters (2)
  • Draft length M = not reported
    IGPG builds M-length draft sequences for parallel verification; M determines the maximum number of tokens generated per forward pass, but no value or tuning procedure is given (Section 3.3).
  • N-gram window N = not reported
    The approximate draft language model uses an N-token context of the form P(x_t|x_{t-N+1},...,x_{t-1}); N is never specified, which affects draft quality and speedup (Section 3.3).
assumptions (3)
  • domain assumption Cached KV representations from a previous round remain approximately valid when concatenated with new documents in the current round.
    CICS reuses Ko_t, Vo_t from earlier rounds (Eq. 3). Section 3.2 admits cached representations are contextually contaminated by prior documents and need IDGR to correct quality.
  • domain assumption A large fraction of generated tokens appears as n-grams in the retrieved documents.
    IGPG relies on matching phrase fragments from D_t (Figure 3 motivates this with 2WikiMultihopQA). If acceptance drops on other datasets, the decoding speedup diminishes.
  • domain assumption LLMs can follow explicit instructions to ignore or prioritize cached content without recomputing it.
    IDGR injects natural-language instructions into the prompt (Eq. 4); this assumes instruction-following is reliable enough to offset the context contamination from CICS.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps." pith.science (2026). https://pith.science/paper/JIFZOF72

@misc{pith2026250512731,
  author       = {Pith},
  title        = {Pith review of: Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIFZOF72}},
  note         = {Machine review of arXiv:2505.12731}
}
read the original abstract

Retrieval-augmented generation (RAG) has emerged as a pivotal method for expanding the knowledge of large language models. To handle complex queries more effectively, researchers developed Adaptive-RAG (A-RAG) to enhance the generated quality through multiple interactions with external knowledge bases. Despite its effectiveness, A-RAG exacerbates the pre-existing efficiency challenges inherent in RAG, which are attributable to its reliance on multiple iterations of generation. Existing A-RAG approaches process all retrieved contents from scratch. However, they ignore the situation where there is a significant overlap in the content of the retrieval results across rounds. The overlapping content is redundantly represented, which leads to a large proportion of repeated computations, thus affecting the overall efficiency. To address this issue, this paper introduces a model-agnostic approach that can be generally applied to A-RAG methods, which is dedicated to reducing the redundant representation process caused by the overlapping of retrieval results. Specifically, we use cache access and parallel generation to speed up the prefilling and decoding stages respectively. Additionally, we also propose an instruction-driven module to further guide the model to more effectively attend to each part of the content in a more suitable way for LLMs. Experiments show that our approach achieves 2.79 and 2.33 times significant acceleration on average for prefilling and decoding respectively while maintaining equal generation quality.

Figures

Figures reproduced from arXiv: 2505.12731 by the authors.

Figure 1
Figure 1. (a) The pipeline of A-RAG. (b) Analysis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of our IDR2. the same color indicates the same document and representation. The a i t denotes the i th generated token and m rep￾resents the number of generated tokens. The Eq.(2) denoted the autoregression process, which can only generate one token at each step. It needs to be exe￾cuted through multiple steps to obtain the complete At . CICS stores the Kt , Vt from Eq.(1), which is the representation f… view at source ↗
Figure 3
Figure 3. The x-axis represents the length of consecutive [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The analysis of speedup for different numbers of retrieved documents. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The detailed process of A-RAG based on a specific example, with LLaMA-13B. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 15 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  3. [3]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations

  4. [4]

    Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. 2023. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5910--5924

    work page 2023

  5. [5]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206--2240. PMLR

    2022

  6. [6]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

  7. [7]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  8. [8]

    James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. 2020. Iirc: A dataset of incomplete information reading comprehension questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1137--1147

    2020

Show all 57 references
  1. [9]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  2. [10]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346--361

    2021

  3. [11]

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pages 3690--3699. PMLR

    2020

  4. [12]

    Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions. arXiv preprint arXiv:2410.12837

    2024 arXiv

  5. [13]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625

    2020

  6. [14]

    Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. 2022. Token dropping for efficient bert pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3774--3784

    2022

  7. [15]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Comp...

    2024

  8. [16]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969--7992

    2023

  9. [17]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations

    2019

  10. [18]

    Gyuwan Kim and Kyunghyun Cho. 2021. Length-adaptive transformer: Train once with length drop, use anytime with search. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Languag...

    2021

  11. [19]

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2024. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784--794

    2022

  13. [21]

    M Komeili. 2021. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566

    2021 arXiv

  14. [22]

    Jun Kong, Jin Wang, Liang-Chih Yu, and Xuejie Zhang. 2022. Accelerating inference for pretrained language models by unified multi-perspective early exiting. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4677--4686

    2022

  15. [23]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  17. [25]

    Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In The Twelfth International Conference on Learning Representations

  18. [26]

    Huanshuo Liu, Bo Chen, Menghui Zhu, Jianghao Lin, Jiarui Qin, Hao Zhang, Yang Yang, and Ruiming Tang. 2024. Retrieval-oriented knowledge for click-through rate prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1441--1451

    2024

  19. [27]

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. 2024. Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text. arXiv preprint arXiv:2410.07590

    2024 arXiv

  20. [28]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023

  21. [29]

    Superposition prompting: Improving and accelerating retrieval-augmented generation

    Thomas Merth, Qichen Fu, Mohammad Rastegari, and Mahyar Najibi. Superposition prompting: Improving and accelerating retrieval-augmented generation. In Forty-first International Conference on Machine Learning

  22. [30]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings...

    2024

  23. [31]

    Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904

    2022 arXiv

  24. [32]

    Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024. When do llms need retrieval augmentation? mitigating llms' overconfidence helps retrieval augmentation. arXiv preprint arXiv:2402.11457

    2024 arXiv

  25. [33]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  26. [34]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023

  27. [35]

    Accelerating llm inference with staged speculative decoding

    Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models@ ICML2023

  28. [36]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. arXiv preprint arXiv:2403.10081

    2024 arXiv

  29. [37]

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pages 2464--2469. IEEE

    2016

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo...

    2023

  32. [40]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 97--110. IEEE

    2021

  33. [41]

    Ruobing Wang, Daren Zha, Shi Yu, Qingfei Zhao, Yuxuan Chen, Yixuan Wang, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, et al. 2024 a . Retriever-and-memory: Towards adaptive note-enhanced retrieval-augmented generation. arXiv preprint arXiv:2410.08821

    2024 arXiv

  34. [42]

    Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, et al. 2024 b . Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223

    2024 arXiv

  35. [43]

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246--2251

    2020

  36. [44]

    Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023 a . Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487

    2023 arXiv

  37. [45]

    Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding

    Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding. Transactions on Machine Learning Research

  38. [46]

    Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. 2023 b . Improving the reliability of large language models by leveraging uncertainty-aware in-context learning. arXiv preprint arXiv:2310.04782

    2023 arXiv

  39. [47]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  40. [48]

    Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. 2024. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. arXiv preprint arXiv:2406.19215

    2024 arXiv

  41. [49]

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023 a . Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168

    2023 arXiv

  42. [50]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2023 b . H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Pro...

    2023

  43. [51]

    Accelerating iterative retrieval-augmented language model serving with speculation

    Zhihao Zhang, Alan Zhu, Lijie Yang, Yihua Xu, Lanting Li, Phitchaya Mangpo Phothilimthana, and Zhihao Jia. Accelerating iterative retrieval-augmented language model serving with speculation. In Forty-first International Conference on Machine Learning

  44. [52]

    Zihan Zhang, Meng Fang, and Ling Chen. 2024. Retrievalqa: Assessing adaptive retrieval-augmented generation for short-form open-domain question answering. arXiv preprint arXiv:2402.16457

    2024 arXiv

  45. [53]

    Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023. Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  46. [54]

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330--18341

    2020

  47. [55]

    Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774

    2021 arXiv

  48. [56]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.