Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

C2KV claims that KV caches can be made compressed, position-agnostic, and directly composable at concatenation time, enabling up to 17× inference speedup on long-context workloads without retraining the base model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:11 UTC pith:4BA4ZXGQ

load-bearing objection A novel sidecar-extractor design for composable KV reuse with public code, but the 'by construction' claim overstates what the mask actually guarantees, and the benchmark overlap plus missing speedup breakdown mean the evidence needs a serious revision. the 4 major comments →

arxiv 2607.17715 v1 pith:4BA4ZXGQ submitted 2026-07-20 cs.CL

C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Chuheng Du , Junyi Chen , Hanlin Tang , Kan Liu , Tao Lan , Lin Qu , Chaoyue Niu , Shengzhong Liu
show 2 more authors
Guihai Chen Fan Wu
This is my paper
classification cs.CL
keywords KV cache reuseKV cache compressionnon-prefix reuseposition-agnostic representationcomposable KV cachestructured attentionlong-context inferencesidecar extractor
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-context LLM serving is dominated by the cost of storing and moving key-value (KV) caches, not by prefill computation, but existing reuse methods either recompute portions of the context or alter the model to stitch blocks together. C2KV proposes to fix this at the representation level: a lightweight sidecar extractor reads each document once and emits compressed, position-agnostic KV blocks that can be concatenated in any order at inference time. The extractor uses learnable memory tokens and a structured attention mask to isolate the compressed representations from their source context, then trains them jointly with a concatenation objective so they stay accurate after recombination. If the central claim holds, document encoding becomes a one-time offline cost and request-time preparation reduces to loading precomputed blocks, yielding large latency and bandwidth savings while keeping the base model frozen.

Core claim

The paper's central claim is that composability for KV reuse is a learnable property of the representation itself, not an approximation to be fixed at serving time. C2KV learns a compressed KV cache manifold in which each document is reduced to a small number of C2Token key-value pairs that are explicitly designed to be position-agnostic: the Structured Information Flow mask ensures original tokens never see the C2Tokens, each C2Token only reads its own block plus a shared sink block, and C2Tokens accumulate causally. Because the extracted pairs are stored without positional embeddings, they can be assigned new positions at concatenation time and stitched directly with no blending or recompu

What carries the argument

Structured Information Flow (SIF): the attention mask that governs the C2Extractor. It enforces three constraints—original tokens attend only to original tokens, each C2Token attends only to its own token block plus the sink block, and C2Tokens attend causally to preceding C2Tokens—so that the extracted C2Token KVs are decoupled from their original context. The compression-concatenation co-training objective then makes these position-agnostic blocks merge-ready.

Load-bearing premise

The extractor's per-block C2Token representations are genuinely context-independent, even though each document's later blocks causally encode information from earlier blocks; the paper asserts this structural isolation yields position-agnostic representations without formally demonstrating that reordering or interleaving extracted segments cannot corrupt that baked-in context.

What would settle it

Train C2KV on a multi-document QA corpus, then for a fixed query, evaluate the model's answer under all permutations of the document order (or with one document split into blocks whose order is shuffled). If the variance in task scores across permutations is comparable to a plain-concatenation baseline, or if reversing the document order changes the generated answer, the position-agnostic claim is falsified. A more direct check: measure the KL divergence between the model's next-token distributions when a single document's C2KV is placed at the beginning versus the end of a long context; high

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Non-prefix KV reuse becomes a load-only operation: time-to-first-token consists of fetching compressed KV blocks and re-assigning positional embeddings, with no blending or token recomputation.
  • Decode-time latency no longer scales linearly with context length; the paper shows the per-token latency curve flattens between 16k and 128k tokens.
  • A frozen, unmodified base model can be augmented with a sidecar module of roughly 10% additional parameters, avoiding per-model architectural changes.
  • Dynamic-ratio training produces a single extractor that works across compression budgets, including an unseen 10× ratio.
  • The approach transfers across model families and parameter scales, maintaining accuracy near full recompute on multi-document QA and summarization tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The position-agnostic claim is likely bounded by information flow across segments: because C2Tokens only accumulate context causally within a document, the first block of a later document carries no memory of earlier documents, so cross-document dependencies may still require the sink block or global mechanisms the paper does not ablate deeply.
  • A natural test is to measure order invariance directly: for a fixed query and set of documents, the output distribution should be nearly identical under document permutations if the manifold is truly position-agnostic; the paper reports aggregate accuracy but not per-instance permutation variance.
  • The same sidecar extraction could be applied to streaming or incremental settings by re-extracting only new segments and concatenating, but the paper explicitly assumes offline document identification and extraction.
  • Since the SIF mask already constrains information to a low-dimensional subspace, further layer-wise or adaptive compression budgets—or quantizing the C2KV blocks—could compound the memory savings without retraining the extractor.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes C²KV, a method for non-prefix KV cache reuse with compression. It augments a frozen base LLM with a lightweight sidecar extractor that inserts learnable C2Token embeddings and per-layer QKV projection heads. A structured attention mask (SIF) is used to allow original tokens to attend only to original tokens, and C2Tokens to attend only to their own block (and a sink block) plus earlier C2Tokens. The resulting per-document C2KV segments are stored and later concatenated directly at inference with only positional reassignment, avoiding blending or recomputation. The extractor is trained by SFT on multi-document QA data (HotpotQA, 2WikiMultiHopQA, LongMagpie) using a compression-concatenation co-training objective. Experiments cover LongBench subsets, RULER, TTFT, and decode latency on Qwen3-4B, Llama3.1-8B, Qwen2.5-7B, and Qwen3-14B, comparing with full recompute, naive reuse, EPIC, CacheBlend, and Block-Attention. The central claims are that C²KV significantly reduces KV storage/transfer costs and preserves generation quality while supporting flexible non-prefix reuse.

Significance. If validated, the contribution is practically significant: it directly targets the storage and memory-bandwidth bottleneck of long-context serving rather than only prefill compute, and it does so without modifying the frozen base model. The paper includes a public code release, evaluates across several model families, and tests compression ratios including a dynamic-ratio setting and RULER long-context retrieval. The proposed architecture is interesting and the co-training idea is a reasonable way to make compressed representations concatenation-aware. However, the core claim that the SIF mask makes extracted KVs position-agnostic and composable 'by construction' is not established; at best the experiments show that co-training can learn approximate order tolerance. In addition, key evaluation tasks overlap with the training data, and the reported accuracy numbers are single-run without variance. The paper is promising but needs stronger validation of its central mechanism before the claims can be accepted.

major comments (4)
  1. [§3.3, Eqs. (8)–(9), §3.3.3] The claim that SIF yields 'context-independent, composable, and directly reusable' C2KVs, and that reuse works 'by construction rather than approximation', is not supported by the masking scheme. Original tokens still attend causally to all preceding original tokens (Eq. (8) only blocks original→C2 attention), so the original-token hidden states in block B(j) at layer ℓ already encode information from blocks 1..j−1. Since C2Token c_j is allowed to read those block states (Eq. (9)), and the 'Causal accumulation across blocks' rule additionally lets c_j attend to earlier C2Tokens, the extracted C2KV for a later block is a function of the order and positions of earlier blocks. Thus the position-agnostic/composable property is not guaranteed by construction; it is at best an approximate property learned through co-training. Please either provide a formal argument for the claim or weaken it,
  2. [§4.1.1, Appendix B, Table 5] The extractor is SFT-trained on HotpotQA, 2WikiMultiHopQA, and LongMagpie, and the LongBench evaluation includes HotpotQA and 2WikiMQA. The accuracy results on those subsets are therefore in-domain, which weakens the headline 'preserving generation quality' claim as a statement of general composable reuse. Please report results separately for evaluation tasks that are disjoint from the training tasks, or at least clearly disclose the overlap when presenting aggregate claims.
  3. [Tables 1, 2, 4, 5] All accuracy tables report single runs without standard errors, confidence intervals, or significance tests. Many differences between C²KV and Full Recompute are small (around 0.01–0.02 F1/ROUGE), so the claim of 'preserving generation quality' is under-specified without an estimate of run-to-run variability. This is especially important because the proposed method is stochastic in training and the evaluation is on relatively small LongBench subsets.
  4. [§4.2, Figure 7, Appendix C] The TTFT comparison uses 4× compression for C²KV while baselines use uncompressed KVs, and TTFT excludes offline extraction and system/query prefill for all methods. This is a reasonable design choice for isolating the storage/bandwidth benefit, but the abstract's 'up to 17× inference speedup' is not traceable to a single end-to-end measurement reported in the paper. Please provide a concrete breakdown of the 17× figure (which operations are in the numerator/denominator) and, ideally, an end-to-end latency measurement for a representative workload including decode.
minor comments (7)
  1. [§3.3.2] The 'sink block B(0)' is referenced in Eq. (9) but is never defined in §3.3.1 or elsewhere. Clarify how it is constructed and why it provides necessary global context.
  2. [§3.4.1] The sentence 'compression and KV cache concatenation are ,' is incomplete; the missing predicate should be filled in.
  3. [Figure 3] Typo in caption: 'Lantecy' should be 'Latency'.
  4. [§4.6 vs. Table 2] The text refers to 'C2KV-4x-Res', but Table 2 has no 'Res' row for Qwen3-14B. Please align the text with the table or add the missing row.
  5. [§4.1.1 and References] LongBench is cited as [40], but reference [40] is titled 'LongCite' and does not appear to be the LongBench paper. Please correct the citation.
  6. [Eq. (5)] Equation (5) contains garbled control characters and an unusual norm expression; please re-set it in proper LaTeX.
  7. [Table 4] The abbreviation 'w/o T.' in the row 'C2KV-Res. (4x) w/o T.' is not explained; define it in the table caption.

Circularity Check

2 steps flagged

Partial circularity: 'reuse by construction' is actually a trained objective, and the core quality claims are evaluated on the same QA families used for SFT.

specific steps
  1. self definitional [§3.3.3 (Comparison with Anchor Token-based Compression) and §3.4 (Composable KV Learning)]
    "This asymmetry is critical: it allows the extracted C2KVs to be context-independent, composable, and directly reusable across requests. ... Our design replaces semantic compression with structural KV extraction, enabling reuse by construction rather than approximation."

    The 'by construction' claim does not follow from the mask equations. In §3.3.2, original tokens still attend to all preceding original tokens under the standard causal mask (Eq. 8 only forbids original→C2 attention), and the 'Causal accumulation across blocks' rule lets each C2Token attend to earlier C2Tokens. Therefore a C2KV extracted at block j is a function of earlier blocks, so position-agnostic composability is not entailed by SIF; it is the property that §3.4's concatenation-supervised LSFT is explicitly trained to produce. Calling this trained behavior a construction-level property makes the conclusion equal to the training target.

  2. fitted input called prediction [§4.1.1 (Dataset) and Appendix B.2 / Table 5]
    "We train the C2Extractor using multi-document supervised fine-tuning (SFT) question-answering data ... HotpotQA [37], 2WikiMultiHopQA [13], and LongMagpie [9]. ... We evaluate C2KV on LongBench datasets covering multi-document QA and long-form summarization, including HotpotQA, 2WikiMQA, MuSiQue ..."

    The extractor is SFT-trained to minimize answer loss after KV concatenation on HotpotQA and 2WikiMQA, then the paper reports 'preserving generation quality' on LongBench, which includes HotpotQA, 2WikiMQA, and the same-family MuSiQue. Thus the central composability/quality result is evaluated on the training distribution family, not as an out-of-sample consequence of the structural design. The compositionality is a fitted parameter, and the benchmark result partly measures the fit itself.

full rationale

The paper is an empirical training-and-evaluation paper, so there is no formal derivation chain that could be circular in the mathematical sense. The strongest circularity signal is that 'reuse by construction' is asserted in §3.3.3 but the actual mechanism in §3.4 is end-to-end supervision after concatenation: the extractor is trained to make C2KVs composable, and then the paper's multi-document QA evaluation (HotpotQA, 2WikiMQA, MuSiQue) is on the same task family used for that training. This makes the central quality claim partially fitted rather than independently predicted. The position-agnostic claim is also not forced by the SIF mask, because original-token hidden states read by C2Tokens carry causal context from earlier blocks and C2Tokens attend to earlier C2Tokens; the paper's own equations show cross-block information flow, so calling the result 'context-independent by construction' reduces to the trained objective. These issues are balanced by substantial independent evidence: the TTFT/decode speedup measurements are system-level and not trained, RULER and GSM8K provide out-of-family evaluation, the base model is frozen, and comparisons against external baselines are proper. There is no load-bearing self-citation: the authors' own prior works appear only as background citations. Overall, the circularity is partial—present in how the composability claim is established and evaluated—but the efficiency and transfer claims retain independent content, giving a score of 4 rather than higher.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The framework's load-bearing choices are architectural: a learned compression manifold, a sink block, re-applied RoPE, and a training distribution assumed to teach composability. The paper provides no formal guarantee that C2Token KV pairs remain valid after arbitrary concatenation, and the main accuracy evidence partially overlaps with the SFT training tasks.

free parameters (3)
  • compression ratio k = 4x, 8x, 16x (plus dynamic sampling {4x, 8x, 16x})
    Controls KV storage size and accuracy; chosen by the authors, not derived. Tables 1 and 2 show the resulting accuracy trade-off.
  • residual mean-pooling scale = 1.0
    Hand-fixed in Eq. (7); no sensitivity study is provided.
  • C2Token embedding and per-layer QKV projection heads = Learned via SFT on 120k QA samples
    The primary fitted component of the method; it is optimized end-to-end on the evaluation task distribution, so the central representation is trained rather than derived.
axioms (4)
  • domain assumption RoPE can be applied to extracted KV caches after storage, with only positional reassignment, while preserving compatibility.
    Used in §3.2.4 and Appendix C.2. The paper assumes positional re-rotation is lossless and the frozen base model accepts re-rotated KVs, but does not prove this across arbitrary concatenations.
  • domain assumption Original tokens' hidden states are unaffected by C2Tokens under the SIF mask (§3.3.2, Eq. 8), so base model semantics are fully preserved.
    Depends on exact behavior of masked attention and normalization; any interaction between appended C2Tokens and layer normalization or attention softmax could perturb original-token representations.
  • domain assumption SFT on HotpotQA/2WikiMultiHopQA/LongMagpie teaches a composable KV manifold that transfers to LongBench and RULER.
    Required for the evaluation claims; the overlap between training tasks and LongBench evaluation tasks weakens the transfer evidence.
  • ad hoc to paper A sink block B(0) provides necessary global context during extraction.
    Introduced in Eq. (9) without an ablation of its necessity; included to stabilize extraction but not independently motivated.
invented entities (2)
  • C2Tokens no independent evidence
    purpose: Learnable compression tokens, one per k document tokens, whose KV pairs form the stored C2KV cache.
    Internal mechanism with no external falsifiable signature; its effectiveness is measured only on the paper's chosen benchmarks.
  • C2Extractor sidecar module no independent evidence
    purpose: Trainable module attached to a frozen LLM to compute compressed, composable KV representations.
    The claimed benefits depend entirely on this trained module; no independent validation outside the paper's experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 19844 in / 11468 out tokens · 129851 ms · 2026-08-01T17:11:05.728060+00:00 · methodology

0 comments
read the original abstract

Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C$^2$KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C$^2$KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C$^2$KV significantly reduces KV cache storage and transfer costs, achieving up to 17$\times$ inference speedup under long contexts, while preserving generation quality.

Figures

Figures reproduced from arXiv: 2607.17715 by Chaoyue Niu, Chuheng Du, Fan Wu, Guihai Chen, Hanlin Tang, Junyi Chen, Kan Liu, Lin Qu, Shengzhong Liu, Tao Lan.

Figure 1
Figure 1. Figure 1: Breakdown of Time-to-First-Token (TTFT) in long [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: KV Deviation grows as recompute ratio drops. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Lantecy comparison of blending and loading. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pipeline overview of C2KV. compression to reused blocks leads to severe fidelity loss, as stan￾dard compressed representations are not designed to be composable or position-agnostic. 3 C2KV Framework 3.1 Framework Overview We propose a non-prefix KV cache reuse framework C 2KV that en￾ables long documents to be compressed, stored, and directly reused at arbitrary positions. C 2KV augments a frozen base LLM… view at source ↗
Figure 5
Figure 5. Figure 5: Architecture overview of C2KV. all C 2Tokens are physically grouped after the document tokens, and logical block boundaries are enforced solely through the attention mask. Each C 2Token is assigned the same position index as the last token in its block. This positional assignment is used only during extraction; the resulting key-value pairs are stored before positional embeddings are applied. 3.3.2 Attenti… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison between Structured Attention Flow [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trade-off between Time-to-First-Token (TTFT, x-axis) and task accuracy (Score, y-axis) under non-prefix document [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Impact of context length on decode latency across different methods. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: RULER evaluation across different context lengths [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV

    cs.AI 2026-07 conditional novelty 6.5

    Source-omitted event KV rows can carry compact upstream state (semantic materialization); deliberate answer-free carriers raise recovery from 6% to 51% on Qwen3-8B, while passive natural mentions do not.

Reference graph

Works this paper leans on

47 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Shubham Agarwal, Sai Sundaresan, Subrata Mitra, Debabrata Mahapatra, Archit Gupta, Rounak Sharma, Nirmal Joshua Kapu, Tong Yu, and Shiv Saini. 2025. Cache- Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation. Proc. ACM Manag. Data3, 3, Article 136 (June 2025), 28 pages. doi:10.1145/3725273

    doi:10.1145/3725273 2025

  2. [2]

    Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. 2025. RocketKV: Accelerating Long-Context LLM Inference via Two- Stage KV Cache Compression.arXiv preprint arXiv:2502.14051(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  4. [4]

    Junyi Chen, Shihao Bai, Zaijun Wang, Siyu Wu, Chuheng Du, Hailong Yang, Ruihao Gong, Shengzhong Liu, Fan Wu, and Guihai Chen. 2025. Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation.arXiv preprint arXiv:2506.03887(2025)

    arXiv 2025

  5. [5]

    Junyi Chen, Chuheng Du, Renyuan Liu, Shuochao Yao, Dingtian Yan, Jiang Liao, Shengzhong Liu, Fan Wu, and Guihai Chen. 2025. TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling.arXiv preprint arXiv:2510.02758(2025)

    arXiv 2025

  6. [6]

    Ruirui Chen, Weifeng Jiang, Chengwei Qin, Ishaan Singh Rawal, Cheston Tan, Dongkyu Choi, Bo Xiong, and Bo Ai. 2024. LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association...

    doi:10.18653/v1/2024 2024

  7. [7]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Barcelona, Spain) (KDD ’24). Association for Computing Machinery, New York, NY, U...

    arXiv 2024

  9. [9]

    Chaochen Gao, Xing W, Zijia Lin, Debing Zhang, and Songlin Hu. 2025. LongMag- pie: A Self-synthesis Method for Generating Large-scale Long-context Instruc- tions. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=tuA2R6gZEA

    2025

  10. [10]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems6 (2024), 325–338

    2024

  11. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2025. LightRAG: Simple and Fast Retrieval-Augmented Generation. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 10746–10761. doi...

    doi:10.18653/v1/2025.findings-emnlp 2025

  13. [13]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. InProceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, Barcelona, Sp...

    doi:10.18653/v1/2020.coling-main.580 2020

  14. [14]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems37 (2024), 1270–1303

    2024

  15. [15]

    Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, tiancheng hu, zhang qin, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. 2025. EPIC: Efficient Position-Independent Caching for Serving Large Language Models. InForty- second International Conference on Machine Learning. https://openreview.net/ forum?id=qjd3ZUiHRT

    2025

  16. [16]

    Yiqian Huang, Shiqi Zhang, and Xiaokui Xiao. 2025. KET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAG. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2(Toronto ON, Canada)(KDD ’25). Association for Computing Machinery, New York, NY, USA, 1003–1012. doi:10.1145/3711896.3737012

    arXiv 2025

  17. [17]

    Dongkyu Kim, Byoungwook Kim, Donggeon Han, and Matouš Eibich. 2024. Au- torag: Automated framework for optimization of retrieval augmented generation pipeline.arXiv preprint arXiv:2410.20878(2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  19. [19]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F...

    2020

  20. [20]

    Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Chong Wang, and Chuang Gan

  21. [21]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before generation. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, ...

    2024

  22. [22]

    Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, et al. 2024. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache.arXiv preprint arXiv:2401.02669(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. Minicache: Kv cache compression in depth dimension for large language models.Advances in Neural Information Processing Systems37 (2024), 139997–140031

    2024

  24. [24]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al . 2025. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. InFindings of the Association for Computational Linguistics: NAACL

    2025

  25. [25]

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. 2025. TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association ...

    doi:10.18653/v1/2025.emnlp-main.334 2025

  26. [26]

    Dongyang Ma, Yan Wang, and Tian Lan. 2025. Block-Attention for Efficient Prefilling. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=7zNYY1E2fq

    2025

  27. [27]

    Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. 2025. MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference. InFindings of the Association for Computational Linguistics: ACL 2025. 18506–18523

    2025

  28. [28]

    Yifan Tan, Haoze Wang, Chao Yan, and Yangdong Deng. 2024. AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization. arXiv preprint arXiv:2409.16546(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Rajan Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, and Tomas Pfister. 2025. In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents. InProceedings of the 63rd Annual Meeting of the Association...

    2025

  30. [30]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic

  31. [31]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024. Duoattention: Efficient long-context llm inference with retrieval and streaming heads.arXiv preprint arXiv:2410.10819 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Xianglong Yan, Zhiteng Li, Tianao Zhang, Haotong Qin, Linghe Kong, Yulun Zhang, and Xiaokang Yang. 2025. ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration.arXiv preprint arXiv:2505.24357 (2025)

    arXiv 2025

  33. [33]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025). C2KV : Compressed and Composable KV Cache Reuse for Efficient LLM Inference KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea

    Pith/arXiv arXiv 2025

  34. [34]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. 2025. Qwen2. 5-1m technical report.arXiv preprint arXiv:2501.15383(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Huan Yang, Renji Zhang, Mingzhe Huang, Weijun Wang, Yin Tang, Yuanchun Li, Yunxin Liu, and Deyu Zhang. 2025. KVShare: An LLM Service System with Effi- cient and Effective Multi-Tenant KV Cache Reuse.arXiv preprint arXiv:2503.16525 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems. https:// openreview.net/forum?id=oDcAGSXZZP

    2025

  37. [37]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (...

    doi:10.18653/v1/d18- 2018

  38. [38]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. InProceedings of the Twentieth European Conference on Computer Systems(Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New Y...

    arXiv 2025

  39. [39]

    Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Kristina Toutanova, Anna Rumshisky, L...

    2021

  40. [40]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2025. LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pileh...

    doi:10.18653/v1/2025 2025

  41. [41]

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. 2024. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization.Advances in Neural Information Processing Systems37 (2024), 3304– 3331

    2024

  42. [42]

    Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. 2024. CaM: cache merging for memory-efficient LLMs inference. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 2428, 11 pages

    2024

  43. [43]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. InThirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/for...

    2023

  44. [44]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet...

    doi:10.52202/079017-2000 2024

  45. [45]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: a dataset for LLM question answering with external tools. InProceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2180, 27 pages. A Appendix Outline In this ap...

    2023

  46. [2022]

    arXiv:2211.09085 [cs.CL] https://arxiv.org/abs/2211.09085

    Galactica: A Large Language Model for Science. arXiv:2211.09085 [cs.CL] https://arxiv.org/abs/2211.09085

    Pith/arXiv arXiv

  47. [2025]

    arXiv preprint arXiv:2506.18879(2025)

    CommVQ: Commutative Vector Quantization for KV Cache Compression. arXiv preprint arXiv:2506.18879(2025)

    Pith/arXiv arXiv 2025