ContextPilot: Fast Long-Context Inference via Context Reuse

Cheng Deng; Liang Cheng; Luo Mai; Xuan Sun; Yeqi Huang; Yinsicheng Jiang

arxiv: 2511.03475 · v4 · submitted 2025-11-05 · 💻 cs.LG

ContextPilot: Fast Long-Context Inference via Context Reuse

Yinsicheng Jiang , Yeqi Huang , Liang Cheng , Cheng Deng , Xuan Sun , Luo Mai This is my paper

Pith reviewed 2026-05-18 00:30 UTC · model grok-4.3

classification 💻 cs.LG

keywords long-context inferenceKV-cache reuseprefill accelerationcontext reuseLLM inference optimizationreasoning qualitycontext index

0 comments

The pith

ContextPilot reduces LLM prefill latency by up to 3 times through context reuse while preserving reasoning quality.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ContextPilot to speed up the prefill phase of long-context LLM inference by treating context reuse as a core mechanism. It builds a context index to locate overlapping blocks across users and conversation turns, then applies ordering and de-duplication to increase KV-cache sharing. Succinct annotations are introduced so that reuse does not degrade downstream reasoning. Evaluation shows the approach delivers up to 3x lower prefill latency than prior methods and can even raise reasoning quality at greater lengths. The system is designed as a modular layer that works with existing inference engines.

Core claim

ContextPilot establishes that a context index can identify reusable KV-cache blocks across interactions, ordering and de-duplication can maximize reuse, and succinct annotations can prevent quality loss, yielding up to 3x faster prefill latency while maintaining or improving reasoning performance on long-context tasks.

What carries the argument

A context index that locates overlapping blocks across users and turns, paired with succinct annotations that protect reasoning accuracy during KV-cache reuse.

If this is right

Retrieval-augmented generation and agent memory applications can run with substantially lower prefill cost.
Multi-turn and multi-user conversations can share more computation without proportional latency growth.
Longer context windows become practical because reuse offsets the linear cost increase.
Modular integration allows existing inference engines to adopt the reuse layer without full redesign.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If annotations scale reliably, shared context pools across many users could become a standard efficiency layer in production deployments.
Dynamic annotation generation might further expand reuse opportunities beyond static overlapping blocks.
The same reuse pattern could be tested on decoder-only versus encoder-decoder models to check generality.

Load-bearing premise

Succinct annotations can reliably offset any semantic drift introduced when KV-cache blocks are reused across different interactions.

What would settle it

A controlled test that measures reasoning accuracy on a standard benchmark both with and without reuse of annotated blocks and finds clear degradation under reuse would falsify the quality-preservation claim.

Figures

Figures reproduced from arXiv: 2511.03475 by Cheng Deng, Liang Cheng, Luo Mai, Xuan Sun, Yeqi Huang, Yinsicheng Jiang.

**Figure 1.** Figure 1: Overview of a RAG system. 2 BACKGROUND AND MOTIVATION 2.1 Retrieval-augmented generation systems RAG systems are now integral to both online, latencysensitive services, such as semantic search, dialogue, and deep research (Zilliz, 2025; Guo et al., 2024), and offline, throughput-oriented pipelines for large-scale annotation and synthetic data generation (Shen et al., 2025; Zhou et al., 2024; NVIDIA, 2024;… view at source ↗

**Figure 2.** Figure 2: Context overlap and context reuse opportunities in RAG. 3 DESIGN OVERVIEW 3.1 Observation: significant overlap in retrieval Our design is motivated by a key observation: real-world RAG workloads exhibit substantial overlap in retrieved documents across both sessions and conversation turns: (1) Overlap across sessions. Figure 2a illustrates overlapping retrievals among multiple users querying different as… view at source ↗

**Figure 3.** Figure 3: System Overview of RAGBoost. by 0.3–3.9%, confirming the effectiveness of incorporating contextual hints for enhanced reasoning. Note that the contextual hint does not affect the model’s instruction-following ability, as it only restores minimal retrieval information without altering the user prompt. 3.3 RAGBoost system overview RAGBOOST realizes the three design opportunities above, achieving accuracy-pre… view at source ↗

**Figure 4.** Figure 4: Context index construction with prefix-cache semantics. reuse when overlaps exist; and (3) traverse KV caches in multi-turn conversations to detect duplicated context. 4.1 Key designs for context index [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 5.** Figure 5: Context ordering: (1) find best-match nodes; (2) reorder by shared prefix and append as child; (3) output the ordered context. 4.2 Key operations with context index The context index provides two key operations: Context search. RAGBOOST frequently searches for previously stored contexts based on the current one to enable reuse. The index search algorithm efficiently locates matching contexts by greedily … view at source ↗

**Figure 6.** Figure 6: Example of scheduling requests with ordered contexts. context ordering to avoid redundant tree lookups; (2) groups contexts by the first element of their search path, naturally separating cache regions; and (3) sorts contexts within each group by path length in descending order, ensuring longer prefix matches execute before shorter ones [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: Performance breakdown of key components. 3 5 10 15 k 0 10000 Prefill TP (tokens/s) MultihopRAG 3 5 10 15 k NarrativeQA Radix Cache LMCache CacheBlend Ours [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗

**Figure 8.** Figure 8: Prefill throughput under different top k values. to 20.56% with ordering and 33.97% with scheduling—a 4× improvement. For vLLM with Llama3.3-70B, results are similar: 10.7% → 30.8% → 43.2%. These gains directly translate to reduced prefill computation and lower TTFT. System overhead. We previously showed that index search and update operations complete within 15 µs. Here, we evaluate index construction ov… view at source ↗

**Figure 10.** Figure 10: Attention map of the last layer attention of Qwen (Head 9) for the prompt “The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the following priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?” [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

**Figure 9.** Figure 9: and [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

read the original abstract

AI applications increasingly depend on long-context inference, where LLMs consume substantial context to support stronger reasoning. Common examples include retrieval-augmented generation, agent memory layers, and multi-agent orchestration. As input contexts get longer, prefill latency becomes the main bottleneck. Yet today's prefill acceleration techniques face a trade-off: they either preserve reasoning quality but deliver little KV-cache reuse, or improve reuse at the cost of degraded reasoning quality. We present ContextPilot, a system that accelerates prefill by introducing context reuse as a new mechanism for faster long-context inference. ContextPilot introduces a context index to identify overlapping context blocks across LLM interactions (e.g., across users and turns). It further proposes context ordering and de-duplication techniques to maximize KV-cache reuse. To preserve reasoning quality under reuse, it introduces succinct context annotations that prevent quality degradation. Finally, ContextPilot is built around a modular architecture with a clean interface that integrates with existing inference engines. Extensive evaluation shows that ContextPilot reduces LLM prefill latency by up to $3\times{}$ compared to state-of-the-art methods while preserving reasoning quality. At longer context lengths, it can even improve reasoning quality. ContextPilot is open-sourced at: https://github.com/EfficientContext/ContextPilot.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ContextPilot frames context reuse via indexing, ordering, dedup and annotations as a practical way to cut prefill latency up to 3x, but quality preservation depends on annotations catching semantic mismatches in reused blocks.

read the letter

The main point to know is that ContextPilot pushes context reuse as a way to speed up prefill in long-context LLMs by finding and reusing overlapping blocks across different sessions. It introduces a context index to detect overlaps, then uses ordering and de-duplication to increase cache hits, and relies on succinct annotations to maintain reasoning quality. This combination is presented as new compared to standard KV-cache tricks or compression methods. The system has a clean modular design that should work with current inference engines, which is a plus for adoption. They also open-sourced the code, so others can try it out on their setups. The evaluation claims up to 3x lower prefill latency with quality held steady or improved at longer lengths. That would be useful for RAG and agent applications if it checks out. A potential weak spot is whether the annotations really prevent problems when reused blocks have subtle semantic differences, such as facts that changed between turns. The index might flag blocks that are close but not identical in meaning, and short annotations could fail to flag the mismatch. The abstract does not spell out the exact test setups or statistical checks, so the full experiments will need to demonstrate that the quality holds under realistic reuse scenarios. This paper targets people working on efficient LLM inference and long-context handling. Anyone dealing with multi-turn conversations or shared memory across users could find the techniques relevant. It is solid enough in its approach to warrant a serious referee. The idea is grounded in a real performance issue and offers implementable steps. I would send it for peer review to get feedback on the experiments and the annotation effectiveness.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces ContextPilot, a modular system for accelerating LLM prefill in long-context inference. It uses a context index to detect overlapping KV-cache blocks across users and turns, applies ordering and de-duplication to maximize reuse, and adds succinct context annotations to avoid quality loss. The central empirical claim is up to 3× prefill latency reduction versus prior methods while preserving (or even improving) reasoning quality at longer contexts; the system is open-sourced.

Significance. If the quality-preservation results hold under realistic reuse, the work would be a meaningful systems contribution to efficient long-context serving for RAG, agent memory, and multi-turn applications. The modular interface and open-source artifacts are clear strengths for adoption and follow-on work.

major comments (2)

[Sections 3.3 and 5 (Context Annotations and Quality Evaluation)] The quality-preservation argument rests on succinct context annotations being sufficient to correct semantic mismatches when KV blocks are reused across turns or users (e.g., updated facts or user-specific details). The manuscript does not provide a concrete mechanism or evaluation showing that annotations generated from the blocks themselves resolve all downstream inference errors; this is load-bearing for the “preserving or improving quality” claim.
[Abstract and Section 5 (Evaluation)] The abstract states “up to 3×” prefill speedup and quality preservation, yet the provided description gives no information on the exact baselines, context lengths, metrics (e.g., exact-match, F1, or downstream task accuracy), number of runs, or statistical significance. Without these details the speedup and quality claims cannot be assessed.

minor comments (2)

[Section 3.1] Notation for the context index and block identifiers should be defined once in a single table or figure caption rather than repeated inline.
[Related Work] The paper should cite the specific prior KV-cache reuse and prefill-acceleration works it compares against so readers can verify the “state-of-the-art” baseline selection.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which helps clarify the presentation of our quality preservation claims and evaluation details. We address each major comment below and will incorporate revisions to improve the manuscript.

read point-by-point responses

Referee: [Sections 3.3 and 5 (Context Annotations and Quality Evaluation)] The quality-preservation argument rests on succinct context annotations being sufficient to correct semantic mismatches when KV blocks are reused across turns or users (e.g., updated facts or user-specific details). The manuscript does not provide a concrete mechanism or evaluation showing that annotations generated from the blocks themselves resolve all downstream inference errors; this is load-bearing for the “preserving or improving quality” claim.

Authors: We appreciate the referee pointing out the need for greater specificity here. In Section 3.3 the annotations are generated as concise metadata extracted from each context block (including source provenance, update timestamps, and key entity summaries) and are prepended during reuse to cue the model about potential inconsistencies. Our Section 5 results show that this yields preserved or improved task accuracy relative to reuse without annotations. We do not claim the annotations resolve every conceivable downstream error, but rather that they are sufficient to avoid the quality degradation observed in prior reuse methods. To strengthen this, the revised manuscript will add a concrete worked example of annotation generation plus an ablation isolating their effect on mismatch cases. revision: yes
Referee: [Abstract and Section 5 (Evaluation)] The abstract states “up to 3×” prefill speedup and quality preservation, yet the provided description gives no information on the exact baselines, context lengths, metrics (e.g., exact-match, F1, or downstream task accuracy), number of runs, or statistical significance. Without these details the speedup and quality claims cannot be assessed.

Authors: We agree that the current abstract and evaluation section lack sufficient experimental context. The reported speedups are measured against vLLM with standard prefix caching and recent KV-reuse baselines, using context lengths of 8K–128K tokens. Quality is assessed via accuracy and F1 on LongBench-style reasoning tasks, with all numbers averaged over five runs and reported with standard deviation; differences are statistically significant (p < 0.01, paired t-test). The revised version will update the abstract with these specifics and expand Section 5 with an explicit experimental-setup table and statistical analysis. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical system validated on external benchmarks

full rationale

ContextPilot is a systems paper whose core contributions are a context index for overlap detection, ordering/de-duplication for KV-cache reuse, and succinct annotations for quality preservation. All performance claims (up to 3× prefill speedup, preserved or improved reasoning quality) are presented as measured outcomes from extensive evaluation on benchmarks and open-source artifacts. No equations, fitted parameters, or self-citations are used to derive the results; the design choices are independent of the reported metrics and do not reduce to their own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical derivations or fitted constants; the contribution is an engineering system whose correctness rests on empirical measurement rather than axioms or invented entities.

pith-pipeline@v0.9.0 · 5763 in / 1040 out tokens · 37412 ms · 2026-05-18T00:30:07.753453+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

ContextPilot introduces a context index to identify overlapping context blocks... succinct context annotations that prevent quality degradation.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
cs.CL 2026-05 unverdicted novelty 4.0

Merlin achieves byte-exact deduplication of text at up to 8.7 GB/s using SIMD-optimized hashing, reducing LLM context sizes by 13.9-71% with no data loss.
Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
cs.CL 2026-05 unverdicted novelty 4.0

Byte-exact deduplication reduces RAG context size by 0.16% to 80.34% across three regimes with zero measurable quality regression per multi-vendor LLM evaluation.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · cited by 2 Pith papers

[1]

acl-short.59/

URL https://aclanthology.org/2024. acl-short.59/. Agarwal, S., Sundaresan, S., Mitra, S., Mahapatra, D., Gupta, A., Sharma, R., Kapu, N. J., Yu, T., and Saini, S. Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation.Proc. ACM Manag. Data, 3(3), June 2025. doi: 10.1145/3725273. URL https://doi.org/10.1145/3725273. Alzubi, S., Bro...

work page doi:10.1145/3725273 2024
[2]

URL http://dx.doi.org/10.18653/v1/ 2024.naacl-industry.19. Bhat, S. R., Rudat, M., Spiekermann, J., and Flores-Herr, N. Rethinking chunk size for long-document retrieval: A multi-dataset analysis, 2025. URL https://arxiv. org/abs/2505.21700. Cheng, Y ., Liu, Y ., Yao, J., An, Y ., Chen, X., Feng, S., Huang, Y ., Shen, S., Du, K., and Jiang, J. Lmcache: An...

work page doi:10.18653/v1/ 2024
[3]

URL https://arxiv.org/abs/2401. 15391. Varambally, S., V oice, T., Sun, Y ., Chen, Z., Yu, R., and Ye, K. Hilbert: Recursively building formal proofs with in- formal reasoning, 2025. URL https://arxiv.org/ abs/2509.22819. 13 Xie, Z., Xu, Z., Zhao, M., An, Y ., Mailthody, V . S., Mahlke, S., Garland, M., and Kozyrakis, C. Strata: Hierarchical context cachi...

work page arXiv 2025
[4]

InProceedings of the Twentieth European Conference on Computer Systems (EuroSys)

doi: 10.1145/3689031.3696098. URL https: //doi.org/10.1145/3689031.3696098. Zhang, Q., Hu, C., Upasani, S., Ma, B., Hong, F., Kamanuru, V ., Rainton, J., Wu, C., Ji, M., Li, H., Thakker, U., Zou, J., and Olukotun, K. Agentic context engineering: Evolv- ing contexts for self-improving language models, 2025a. URLhttps://arxiv.org/abs/2510.04618. Zhang, X., ...

work page doi:10.1145/3689031.3696098 2024
[5]

The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?

for the prompt “The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?”. Figure 10. Attention map of the last layer attention of Qwen (Head

work page
[6]

The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?

for the prompt “The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?”

work page

[1] [1]

acl-short.59/

URL https://aclanthology.org/2024. acl-short.59/. Agarwal, S., Sundaresan, S., Mitra, S., Mahapatra, D., Gupta, A., Sharma, R., Kapu, N. J., Yu, T., and Saini, S. Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation.Proc. ACM Manag. Data, 3(3), June 2025. doi: 10.1145/3725273. URL https://doi.org/10.1145/3725273. Alzubi, S., Bro...

work page doi:10.1145/3725273 2024

[2] [2]

URL http://dx.doi.org/10.18653/v1/ 2024.naacl-industry.19. Bhat, S. R., Rudat, M., Spiekermann, J., and Flores-Herr, N. Rethinking chunk size for long-document retrieval: A multi-dataset analysis, 2025. URL https://arxiv. org/abs/2505.21700. Cheng, Y ., Liu, Y ., Yao, J., An, Y ., Chen, X., Feng, S., Huang, Y ., Shen, S., Du, K., and Jiang, J. Lmcache: An...

work page doi:10.18653/v1/ 2024

[3] [3]

URL https://arxiv.org/abs/2401. 15391. Varambally, S., V oice, T., Sun, Y ., Chen, Z., Yu, R., and Ye, K. Hilbert: Recursively building formal proofs with in- formal reasoning, 2025. URL https://arxiv.org/ abs/2509.22819. 13 Xie, Z., Xu, Z., Zhao, M., An, Y ., Mailthody, V . S., Mahlke, S., Garland, M., and Kozyrakis, C. Strata: Hierarchical context cachi...

work page arXiv 2025

[4] [4]

InProceedings of the Twentieth European Conference on Computer Systems (EuroSys)

doi: 10.1145/3689031.3696098. URL https: //doi.org/10.1145/3689031.3696098. Zhang, Q., Hu, C., Upasani, S., Ma, B., Hong, F., Kamanuru, V ., Rainton, J., Wu, C., Ji, M., Li, H., Thakker, U., Zou, J., and Olukotun, K. Agentic context engineering: Evolv- ing contexts for self-improving language models, 2025a. URLhttps://arxiv.org/abs/2510.04618. Zhang, X., ...

work page doi:10.1145/3689031.3696098 2024

[5] [5]

The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?

for the prompt “The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?”. Figure 10. Attention map of the last layer attention of Qwen (Head

work page

[6] [6]

The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?

for the prompt “The retrieved documents are [Doc 1] ABCD [Doc 2] EFGH [Doc 3] IJKL. Please read the context in the fol- lowing priority order: [Doc 2] > [Doc 1] > [Doc 3]. Where is E?”

work page