Pith. sign in

REVIEW 2 major objections 4 minor 111 references

CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoinRAG claims that reusing fine-grained 'nugget' KV caches, rather than full chunks, improves RAG answer quality under a 100 ms latency budget, outperforming the strongest chunk-level baseline by an average 5.3% F1.

desk verdict Genuinely new nugget-level KV cache reuse with solid ablations, but the headline 5.3% F1 gain is statistically fragile and the KV-slice approximation is plausible yet under-tested. read the letter →

arxiv 2608.07458 v1 pith:BNKKPFA4 submitted 2026-08-07 cs.CL cs.AIcs.IRcs.LG

Gyuwan Kim , Cheoneum Park , Tao Yang This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords retrieval-augmentedgenerationKVcachereuselong-contextquestionansweringinformationnuggetslatency-accuracyParetofrontiermulti-hopQAprefilllatencyoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoinRAG claims that the accuracy-latency Pareto frontier of long-context retrieval-augmented generation (RAG) can be improved by operating on fine-grained 'information nuggets' rather than full text chunks. It precomputes key-value (KV) caches for each document chunk offline, extracts nuggets as token spans, and at inference time retrieves query-relevant nuggets and slices the cached KV tensors for those spans, stitching them into a compact prefix with rotary position alignment. Under a strict 100 ms P99 time-to-first-token budget, this yields an average 5.3% relative F1 improvement over the strongest chunk-level cache-reuse baseline across three multi-hop QA datasets, with a roughly 1.8x shorter active context. The design shifts the bottleneck from online prefill computation to one-time offline corpus encoding, and the paper argues that the gain comes from removing noise and redundancy while preserving document-level conditioning through cache slicing.

What carries the argument

The central mechanism is the contextualized nugget KV cache: each document chunk is encoded once in a full forward pass, and each extracted nugget is stored as a token span [si, ei] within that chunk's cached key-value tensors. At inference, the selected nugget caches are sliced as C[si:ei], rotated by an offset Δi that reassigns contiguous positions via the RoPE rotation operator, and concatenated into a single prefix cache. This lets the model attend to a compact sequence of nuggets as if it were a fresh continuous prefix, without re-encoding any raw text. Two-stage retrieval (first top-kc chunks by dense embedding, then top-k nuggets within those chunks) supplies the query-specific spans, and nugget-aware fine-tuning trains the model on the exact same stitched-cache construction to close the training-inference gap.

What would settle it

Run the same selected nuggets through two configurations: (1) CoinRAG's cache-sliced composition, and (2) a fresh forward pass that encodes the concatenated nugget text with the same order and positions, with all other components (retrieval, model, prompt) held fixed. If the fresh encoding attains equal or higher F1 on HotpotQA, 2WikiMQA, and MuSiQue at comparable context length, the claim that cache slicing preserves context worth is falsified; if the fresh encoding is worse, slicing adds value beyond faithful replication.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a RAG system can beat chunk-level KV cache reuse on answer quality while staying within a 100 ms P99 TTFT budget by representing retrieved evidence as query-selected nugget spans of a precomputed chunk cache. Concretely, on LongBench multi-hop QA (HotpotQA, 2WikiMQA, MuSiQue), the best CoinRAG configuration reaches F1 scores of 51.4, 44.5, and 32.3 under the budget, against 49.1, 42.2, and 27.4 for the strongest chunk-level competitor, an average improvement of 5.3%. The paper also shows that even with no latency limit, the average F1 gain over all baselines is 5.2%, while the active context length is up to 10.1x shorter than standard RAG. These empirical results are attributed to three mechanisms working together: two-stage chunk-then-nugget retrieval that narrows the candidate pool, context-preserving KV slicing that avoids re-encoding isolated span text, and nugget-aware fine-tuning that aligns the training distribution with the stitched-cache inference structure.

Load-bearing premise

The approach assumes that slicing a precomputed chunk's key-value cache at an offset span and rotating its positions produces a representation equivalent to freshly encoding the composed nugget sequence in one continuous context.

Editorial extensions

If this is right

  • Under a 100 ms P99 TTFT service-level agreement, CoinRAG's best configuration achieves higher F1 than the strongest chunk-level baseline on all three multi-hop QA datasets, with an average relative gain of 5.3%.
  • The active prefix context is about 1.84x shorter than the best chunk-level baseline under the same budget, which directly reduces the KV cache memory footprint per request and raises the number of concurrent requests a server can hold.
  • When the latency limit is removed, CoinRAG still improves average F1 by 5.2% over the strongest chunk-level baseline, while using 6.8x shorter context on average, indicating that removing noisy context outweighs the loss from missing cross-chunk interactions.
  • Position alignment during cache composition contributes most when the latency budget is tight: under a 75 ms P99 budget it improves F1 by 3.0–8.5% across datasets, and becomes comparable to no alignment beyond 100 ms.
  • Nugget-aware fine-tuning is a load-bearing component: ablating it reduces peak F1 by 6.3 to 11.3 points on the three datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's core assumption—that sliced chunk-cache representations behave like fresh encodings of the composed nugget sequence—could be tested directly by comparing CoinRAG's assembled prefix against a fresh forward pass over the same nugget text. If the fresh encoding matches or exceeds F1, the gain would come from context compression rather than from cache-reuse fidelity; if it is lower, slicin
  • The failure examples suggest an exploitable bias: when supporting and distractor nuggets share surface terms, the model tends to follow frequency. A testable extension is re-ranking selected nuggets with a cross-encoder or applying a frequency-debiasing heuristic at composition time, which could recover some of the missed answers.
  • Because the offline cache is tied to a specific model checkpoint and positional-encoding topology, the method's practical reach depends on how often the backbone changes. An incremental-re-encoding scheme for changed chunks, noted by the paper, would make the method viable for evolving corpora.
  • If the slicing approximation holds for other RoPE-based models, the same nugget-cache composition could be applied to long-document summarization or open-domain QA with different retrieval backbones, potentially extending the latency-accuracy frontier beyond the three LongBench tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents CoinRAG, a RAG system that avoids online encoding of long retrieved contexts by precomputing per-chunk KV caches offline, extracting fine-grained 'nuggets' as token spans, retrieving the most query-relevant nuggets in two stages, and composing their sliced KV caches into a single prompt prefix using RoPE position rotation, with optional nugget-aware fine-tuning. The central claim is that under a P99 TTFT budget of 100 ms, CoinRAG achieves higher answer F1 than chunk-level KV-cache-reuse baselines on three LongBench multi-hop QA datasets, improving over the strongest baseline TurboRAG by 4.7%, 0.5%, and 14.6% (average 5.3%), while also shortening active context length and improving the accuracy-length Pareto frontier.

Significance. If the empirical claims hold, this is a practically useful contribution to efficient long-context RAG: it directly targets an interactive latency SLA, reports detailed engineering trade-offs (disk storage, I/O, offline extraction cost, fine-tuning cost), includes ablations that isolate each component, and retrains TurboRAG and KVLink under matched data and hyperparameters. The paper is also careful in its limitations section about offline costs, checkpoint coupling, retrieval recall, and the absence of cross-chunk attention. I do not see a circularity problem in the F1 comparisons; however, the headline statistical evidence and the validation of the KV-composition approximation are not yet commensurate with the strength of the claims.

major comments (2)
  1. [§4.2, Table 3, Appendix E] The headline average gain is a comparison of per-dataset maxima, not of fixed systems. Appendix E sweeps kc over 9 values and k over 12 values for CoinRAG (108 configurations), while chunk baselines sweep 9 kc values (CacheBlend additionally sweeps 11 recomputation ratios), and Table 3 reports, for each method, the configuration with the highest F1 at the 100 ms budget on the same 200-question evaluation sets used to report the results. The 2WikiMQA difference is 42.4 vs. 42.2 F1, roughly one question in 200, and no confidence intervals, paired bootstrap intervals, or significance tests are reported. Because Section 5 quantifies gains to one decimal place, please add per-question paired bootstrap CIs and either pre-register the configuration or select it on a held-out validation split; otherwise the central Pareto-frontier claim is not statistically established.
  2. [§2.4 and §4.3] The central KV-composition mechanism is not validated against the correct oracle. The statement in §2.4 that sliced KV states are 'identical to what they would be under a fresh encoding of the chunk' is about the original chunk-level cache, not about the composed sequence of nuggets from different chunks; cross-slice attention is absent and RoPE rotation only relocates positions. The Isolated Nugget Encoding ablation in §4.3 is a reasonable component study, but it does not measure the approximation error introduced by composing slices rather than encoding the same nugget sequence fresh in one pass. Please add a direct control that encodes the exact composed nugget sequence in a forward pass (reporting F1 with latency unconstrained, and if feasible under a matched budget) and report the difference; this determines whether the mechanism itself is sound or whether the observed gains come from other components such as fine-tuning and prompt design.
minor comments (4)
  1. [§1] The sentence 'it consistently outperforms Standard RAG and Standard CAG with a 5.3% (41.7 vs. 39.6) answer quality improvement' is inaccurate: Table 3 shows 41.7 vs. 39.6 is CoinRAG vs. TurboRAG, while Standard RAG averages 29.9; please correct the sentence or name the right baseline.
  2. [Table 3] The CoinRAG rows are garbled in the rendered text (e.g., '1, 3051.472 87 618'); please fix the table formatting so each column is legible.
  3. [§4.1 and Appendix E] Please state explicitly whether all baselines use the identical answer-generation prompt; if CoinRAG uses a different prompt from the baselines, prompt differences would confound the F1 comparison.
  4. [§5 and Abstract] The paper mixes 'relative improvement' and 'F1 improvement' (5.3% vs. +5.3 points); please use one convention consistently throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical comparisons, not derivations that assume their own conclusions.

full rationale

CoinRAG's central claim is an empirical F1 comparison on LongBench under a 100 ms P99 TTFT budget. The method's components (offline nugget extraction, two-stage retrieval, contextualized KV slicing, position rotation, and nugget-aware fine-tuning) are implemented and evaluated against external baselines; none of the reported numbers is derived from an equation that assumes the conclusion. The sliced-KV approximation in Section 2.4 is a stated design assumption tested by the 'Isolated Nugget Encoding' ablation, and the composition formula Cctx = Cp ⊕ Rot(Cb1[s1:e1]; Δ1) ⊕ ... is a construction, not a circular proof. Fine-tuning on the training splits of the same three benchmarks is standard supervised practice and is shared by the trained baselines (TurboRAG, KVLink), so it does not by itself force the headline result. There are no load-bearing self-citations and no imported uniqueness theorems; RoPE rotation is prior external work. The main legitimate concern is statistical fragility: hyperparameters kc and k are swept on the same 200-question eval sets and the best configuration is reported without significance testing or confidence intervals, so the 5.3% average gain may not be robust. That is a reliability and correctness issue, not circularity, because the reported F1 values are observed outcomes rather than quantities constructed to equal the method's inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirically supported but not fully verified premises: that RoPE-rotated KV slices faithfully represent composed contexts, that the proprietary extractor produces high-recall nuggets, and that two-stage dense retrieval surfaces the needed evidence. These are domain assumptions about the behavior of the model and the retriever, not mathematical axioms.

free parameters (2)
  • fuzzy match threshold tau = 0.7
    Section 2.2 (Algorithm 1, line 9) and Appendix E: candidates are accepted only if similarity exceeds 0.7; this affects which nuggets are retained and thus the final context composition.
  • per-dataset retrieval counts kc and k = varies by dataset and budget (e.g., HotpotQA kc=1, k=30 under 100 ms)
    Table 3 and Section 4.2 pick the best configuration for each latency budget from a sweep; this is a selection over hyperparameters rather than a fixed system setting, and it influences the claimed F1.
assumptions (3)
  • domain assumption RoPE-rotated KV cache slices are a faithful stand-in for a fresh encoding of the composed nugget sequence
    Section 2.4 relies on this to justify concatenating slices from different chunks; the equivalent fresh encoding would have different attention histories, so this is an approximation that is only indirectly validated.
  • domain assumption The offline nugget extractor (GPT-4o-mini) yields a high-recall set of evidence spans
    Section 2.2 and Appendix C; if recall is low, the composed context misses needed evidence and F1 drops, as the Limitations section acknowledges.
  • domain assumption Dense two-stage retrieval (chunk then nugget) recalls the relevant evidence
    Section 2.3; the method inherits retrieval errors, and the authors state in Limitations that answer quality is bounded by retrieval recall.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG." pith.science (2026). https://pith.science/paper/BNKKPFA4

@misc{pith2026260807458,
  author       = {Pith},
  title        = {Pith review of: CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNKKPFA4}},
  note         = {Machine review of arXiv:2608.07458}
}
read the original abstract

Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This paper optimizes the Pareto frontier under low prefill latency constraints while maximizing accuracy by proposing CoinRAG (Contextualized Information Nugget KV Cache Reuse for Long-Context RAG). The name metaphorically reflects our core mechanism: much like assembling small tokens (or "coins") to accumulate a larger value, CoinRAG compositionally reuses offline-computed, fine-grained nugget caches to form a learned contextual representation efficiently in a more semantically relevant but compact manner. Specifically, instead of full-chunk encoding, CoinRAG identifies query-relevant semantic units within retrieved chunks through two-stage retrieval and seamlessly assembles their sliced KV representations with a chunk-level context. Extensive evaluations on LongBench multi-hop question answering tasks demonstrate that CoinRAG significantly reduces operational costs and outperforms the other baselines with a new Pareto frontier and an average 5.3% relative improvement in answer quality (F1) under a standard fast prefill latency budget.

Figures

Figures reproduced from arXiv: 2608.07458 by the authors.

Figure 1
Figure 1. Comparison of context construction during the prefill stage across RAG paradigms. Gray blocks represent [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Pareto frontiers: accuracy vs. latency budget. The x-axis denotes Time-to-First-Token (TTFT) P99 latency [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Pareto frontiers: accuracy vs. length budget. The x-axis denotes a prefill context length limit in token [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: F1-score versus TTFT P99 latency budget (ms) comparing CoinRAG against four ablations, each removing [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qwen2-7B-Instruct answer generation prompt. The instruction block is computed once as the system prefix. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Full system prompt used for GPT-4o-mini nugget extraction. The user content follows the format [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: A step-by-step example of the nugget extraction algorithm of CoinRAG in Algorithm [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Step-by-step illustration of the CoinRAG inference pipeline (HotpotQA, [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Three success case types of CoinRAG on HotpotQA. All retrieval scores and IDs are from an actual [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Three failed case types of CoinRAG on HotpotQA, mirroring the success cases in Figure [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

111 extracted references · 48 canonical work pages

  1. [1]

    18th USENIX symposium on operating systems design and implementation (OSDI 24) , pages=

    Taming \ Throughput-Latency \ tradeoff in \ LLM \ inference with \ Sarathi-Serve \ , author=. 18th USENIX symposium on operating systems design and implementation (OSDI 24) , pages=

  2. [2]

    The Twelfth International Conference on Learning Representations , year=

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection , author=. The Twelfth International Conference on Learning Representations , year=

  3. [3]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  4. [4]

    2024 , eprint=

    BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation , author=. 2024 , eprint=

    2024

  5. [5]

    arXiv preprint arXiv:2511.02749 , year=

    Using Span Queries to Optimize for Cache and Attention Locality , author=. arXiv preprint arXiv:2511.02749 , year=

    arXiv

  6. [6]

    Companion Proceedings of the ACM on Web Conference 2025 , pages=

    Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages=

    2025

  7. [7]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Benchmarking Large Language Models in Retrieval-Augmented Generation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  8. [8]

    First Conference on Language Modeling , year=

    Understanding Retrieval Augmentation for Long-Form Question Answering , author=. First Conference on Language Modeling , year=

Show all 111 references
  1. [9]

    International Conference on Learning Representations , year=

    8-bit Optimizers via Block-wise Quantization , author=. International Conference on Learning Representations , year=

  2. [10]

    Advances in Neural Information Processing Systems , volume=

    QLoRA: Efficient Finetuning of Quantized LLMs , author=. Advances in Neural Information Processing Systems , volume=

  3. [11]

    European Conference on Information Retrieval , pages=

    Incorporating Q&A Nuggets Into Retrieval-Augmented Generation , author=. European Conference on Information Retrieval , pages=. 2026 , organization=

    2026

  4. [12]

    arXiv preprint arXiv:2404.16130 , year=

    From Local to Global: A Graph RAG Approach to Query-Focused Summarization , author=. arXiv preprint arXiv:2404.16130 , year=

    arXiv

  5. [13]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Enabling Large Language Models to Generate Text with Citations , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  6. [15]

    Proceedings of Machine Learning and Systems , volume=

    Prompt cache: Modular attention reuse for low-latency inference , author=. Proceedings of Machine Learning and Systems , volume=

  7. [16]

    arXiv preprint arXiv:2407.21783 , year=

    The Llama 3 Herd of Models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  8. [17]

    Transactions on Machine Learning Research , year=

    A Survey on Large Language Model Acceleration based on KV Cache Management , author=. Transactions on Machine Learning Research , year=

  9. [18]

    Proceedings of the 28th International Conference on Computational Linguistics , pages=

    Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , author=. Proceedings of the 28th International Conference on Computational Linguistics , pages=

  10. [19]

    International Conference on Learning Representations , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations , year=

  11. [20]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  12. [21]

    Advances in Neural Information Processing Systems , volume=

    HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  13. [22]

    ACM Transactions on Computer Systems , publisher=

    RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation , author=. ACM Transactions on Computer Systems , publisher=

  14. [23]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  15. [24]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Dense Passage Retrieval for Open-Domain Question Answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  16. [25]

    Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    Bridging the Training-Inference Gap for Dense Phrase Retrieval , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    2022

  17. [26]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    QPaug: Question and Passage Augmentation for Open-Domain Question Answering of LLMs , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  18. [27]

    Transactions of the Association for Computational Linguistics , volume=

    Natural Questions: A Benchmark for Question Answering Research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  19. [28]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  20. [29]

    Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    GINGER: Grounded Information Nugget-Based Generation of Responses , author=. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  21. [30]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    Latent Retrieval for Weakly Supervised Open Domain Question Answering , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

  22. [31]

    Learning Dense Representations of Phrases at Scale , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  23. [32]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Phrase Retrieval Learns Passage Retrieval, Too , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    2021

  24. [33]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Compressing Context to Enhance Inference Efficiency of Large Language Models , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  25. [34]

    arXiv preprint arXiv:2505.06569 , year=

    MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAG , author=. arXiv preprint arXiv:2505.06569 , year=

    arXiv

  26. [35]

    Transactions of the Association for Computational Linguistics , volume=

    Lost in the Middle: How Language Models Use Long Contexts , author=. Transactions of the Association for Computational Linguistics , volume=

  27. [36]

    Advances in neural information processing systems , volume=

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , author=. Advances in neural information processing systems , volume=

  28. [37]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  29. [38]

    International Conference on Learning Representations , volume=

    Block-Attention for Efficient Prefilling , author=. International Conference on Learning Representations , volume=

  30. [39]

    Proceedings of Machine Learning and Systems , volume=

    Efficiently Scaling Transformer Inference , author=. Proceedings of Machine Learning and Systems , volume=

  31. [40]

    arXiv preprint arXiv:2411.09607 , year=

    Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework , author=. arXiv preprint arXiv:2411.09607 , year=

    2024 arXiv

  32. [41]

    Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    The great nugget recall: Automating fact extraction and rag evaluation with large language models , author=. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  33. [42]

    2024 , eprint=

    Qwen2 Technical Report , author=. 2024 , eprint=

    2024

  34. [43]

    arXiv preprint arXiv:2511.21631 , year=

    Qwen3-VL Technical Report , author=. arXiv preprint arXiv:2511.21631 , year=

    arXiv

  35. [44]

    The Twelfth International Conference on Learning Representations , year=

    RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval , author=. The Twelfth International Conference on Learning Representations , year=

  36. [45]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    2018

  37. [46]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

  38. [47]

    Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    Retrieval Augmentation Reduces Hallucination in Conversation , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    2021

  39. [48]

    Neurocomputing , volume=

    RoFormer: Enhanced transformer with Rotary Position Embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  40. [49]

    Transactions of the Association for Computational Linguistics , volume=

    MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  41. [50]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    C-Pack: Packed Resources For General Chinese Embeddings , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  42. [51]

    The Twelfth International Conference on Learning Representations , year=

    RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation , author=. The Twelfth International Conference on Learning Representations , year=

  43. [52]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    2018

  44. [54]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  45. [55]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    CompAct: Compressing Retrieved Documents Actively for Question Answering , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  46. [56]

    First Conference on Language Modeling , year=

    RAFT: Adapting Language Model to Domain Specific RAG , author=. First Conference on Language Modeling , year=

  47. [57]

    The Thirteenth International Conference on Learning Representations , year=

    SiReRAG: Indexing Similar and Related Information for Multihop Reasoning , author=. The Thirteenth International Conference on Learning Representations , year=

  48. [58]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  49. [59]

    Advances in Neural Information Processing Systems , volume=

    SGLang: Efficient Structured Language Model Programs , author=. Advances in Neural Information Processing Systems , volume=

  50. [60]

    2025 , eprint=

    Efficient and Scalable Agentic AI with Heterogeneous Systems , author=. 2025 , eprint=

    2025

  51. [61]

    ISCA , year =

    Patel, Pratyush and Choukse, Esha and Zhang, Chaojie and Shah, Aashaka and Goiri, Íñigo and Maleki, Saeed and Bianchini, Ricardo , title =. ISCA , year =

  52. [62]

    2025 , eprint=

    Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving , author=. 2025 , eprint=

    2025

  53. [64]

    2025 , note=

    Recommended Web Performance Timings: How long is too long? , author =. 2025 , note=

    2025

  54. [65]

    Response Time , author =. , note=

  55. [66]

    2023 , note=

    Response Time , author =. 2023 , note=

    2023

  56. [67]

    14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) , year =

    Arpan Gujarati and Reza Karimi and Safya Alzayat and Wei Hao and Antoine Kaufmann and Ymir Vigfusson and Jonathan Mace , title =. 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) , year =

  57. [68]

    and Gonzalez, Joseph E

    Crankshaw, Daniel and Wang, Xin and Zhou, Giulio and Franklin, Michael J. and Gonzalez, Joseph E. and Stoica, Ion , title =. Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation , pages =. 2017 , isbn =

    2017

  58. [70]

    Text Retrieval Conference , year=

    Overview of the TREC 2003 Question Answering Track , author=. Text Retrieval Conference , year=

    2003

  59. [71]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    QuAC: Question answering in context , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  60. [72]

    Transactions of the Association for Computational Linguistics , volume=

    Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  61. [73]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    doc2dial: A goal-oriented document-grounded dialogue dataset , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  62. [74]

    Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    MultiDoc2Dial: Modeling dialogues grounded in multiple documents , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    2021

  63. [75]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  64. [76]

    Proceedings of the 16th ACM workshop on artificial intelligence and security , pages=

    Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection , author=. Proceedings of the 16th ACM workshop on artificial intelligence and security , pages=

  65. [77]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the associatio...

    2024

  66. [78]

    Brian J Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang. 2025. Don't do rag: When cache-augmented generation is all you need for knowledge tasks. In Companion Proceedings of the ACM on Web Conference 2025, pages 893--897

    2025

  67. [79]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024 a . https://arxiv.org/abs/2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2402.03216

    2024 arXiv

  68. [80]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024 b . Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    2024

  69. [81]

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2174--2184

    2018

  70. [82]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: a low-latency online prediction serving system. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation, NSDI'17, page 613–627, ...

    2017

  71. [83]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36:10088--10115

    2023

  72. [84]

    Laura Dietz, Bryan Li, Gabrielle Liu, Jia-Huei Ju, Eugene Yang, Dawn Lawrie, William Walden, and James Mayfield. 2026. Incorporating q&a nuggets into retrieval-augmented generation. In European Conference on Information Retrieval, pages 301--309. Springer

    2026

  73. [85]

    Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. Multidoc2dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6162--6176

    2021

  74. [86]

    Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118--8128

    2020

  75. [87]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 a . Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--6488

    2023

  76. [88]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023 b . Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  77. [89]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325--338

    2024

  78. [90]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial inte...

    2023

  79. [91]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. https://www.usenix.org/conference/osdi20/presentation/gujarati Serving DNNs like clockwork: Performance predictability from the bottom up . In 14th USENIX Symposium ...

    2020

  80. [92]

    LI Haoyang, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, HU Nicole, Wei Dong, Li Qing, and Lei Chen. 2025. A survey on large language model acceleration based on kv cache management. Transactions on Machine Learning Research

    2025

  81. [93]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625

    2020

  82. [94]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princip...

    2023

  83. [95]

    Weronika ajewska and Krisztian Balog. 2025. Ginger: Grounded information nugget-based generation of responses. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2723--2727

    2025

  84. [96]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural informati...

    2020

  85. [97]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  86. [98]

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. 2025. Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6599--6612

    2025

  87. [99]

    Dongyang Ma, Yan Wang, and Tian Lan. 2025. Block-attention for efficient prefilling. In International Conference on Learning Representations, volume 2025, pages 63774--63788

    2025

  88. [100]

    Robert B. Miller. 1968. https://doi.org/10.1145/1476589.1476628 Response time in man-computer conversational transactions . In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I, AFIPS '68 (Fall, part I), page 267–277, New York, NY, USA. Association...

    1968

  89. [101]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606--624

    2023

  90. [102]

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2025. The great nugget recall: Automating fact extraction and rag evaluation with large language models. In Proceedings of the 48th International ACM SIGI...

    2025

  91. [103]

    PubNub . 2023. Response time. https://medium.com/@PubNub/how-fast-is-real-time-human-perception-and-technology-fe7246381935. Accessed: 2026-05-15

    2023

  92. [104]

    Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249--266

    2019

  93. [105]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784--3803

    2021

  94. [106]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  95. [107]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539--554

    2022

  96. [108]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (vo...

    2023

  97. [109]

    Voorhees

    Ellen M. Voorhees. 2004. https://api.semanticscholar.org/CorpusID:215762892 Overview of the trec 2003 question answering track . In Text Retrieval Conference

    2004

  98. [110]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. https://arxiv.org/ab...

    2024 arXiv

  99. [111]

    Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025. Kvlink: Accelerating large language models via efficient kv cache reuse. arXiv preprint arXiv:2502.16002

    2025

  100. [112]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  101. [113]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems,...

    2025

  102. [114]

    Jeong-Min Yun, Yuxiong He, Sameh Elnikety, and Shaolei Ren. 2015. https://doi.org/10.1145/2766462.2767708 Optimal aggregation policy for reducing tail latency of web search . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Informati...

    2015

  103. [115]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others. 2024. Sglang: Efficient structured language model programs. Advances in Neural Information Processing Systems, 37...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.