Pith. sign in

REVIEW 2 major objections 6 minor 139 references

KV-cache systems for LLM serving concentrate into five architectural envelopes; among distributed designs, ownership is where the remaining variance lives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 09:49 UTC pith:LHEP5EMN

load-bearing objection Useful four-axis taxonomy and measurement agenda for KV-cache serving; ownership residual claim is qualitative but scoped honestly. the 2 major comments →

arxiv 2607.02574 v1 pith:LHEP5EMN submitted 2026-06-30 cs.DC cs.AIcs.LG

From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

Jie Li , Tongyang Wang , Yong Chen This is my paper
classification cs.DC cs.AIcs.LG
keywords KV cacheLLM servingdistributed systemsmemory hierarchyprefill/decode disaggregationprefix reuseownershipmeasurement gaps
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This survey argues that the KV cache has stopped being a temporary per-request tensor and become a first-order memory object that serving systems must place, retain, own, and move. Classifying more than thirty systems on four axes—locality, lifetime, ownership, and substrate—the authors show the designs cluster into five recurring envelopes: local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership is the axis that most often separates otherwise-matched distributed systems. The paper also audits current evaluations, finds seven KV-specific measurements that almost no one reports, and pairs those gaps with open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics. A sympathetic reader cares because the map turns a scatter of point solutions into a small design space and names what must be measured before the next systems can be fairly compared.

Core claim

When more than thirty KV-management systems and frameworks are sorted by locality, lifetime, ownership, and substrate, they concentrate around five architectural archetypes rather than filling the design space uniformly. Among the distributed archetypes, once workload and hardware fix locality, lifetime, and substrate, ownership accounts for much of the remaining design variance—and end-to-end metrics hide that choice.

What carries the argument

A four-axis taxonomy—(A) locality, (B) lifetime, (C) ownership, (D) substrate—whose repeated signatures induce five envelopes (Local-paged, Disagg-pipeline, Shared-store, Memory-pool, Hybrid-tier) and make ownership the residual free variable among distributed systems.

Load-bearing premise

That the surveyed systems and their qualitative axis signatures are representative enough for the five-envelope concentration and the ownership-dominance claim to generalize, rather than being forced by hardware, inclusion rules, or sample size.

What would settle it

A matched A/B/D comparison of a centralized-ownership (C1) system against a distributed-ownership (C2) system—e.g., DistServe versus KVDirect—showing whether P99 TTFT decompositions differ as predicted (C1 dominated by queueing and prefill; C2 also showing directory, lock, or consensus stalls), or are instead indistinguishable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can treat Local-paged as the default when KV does not cross workers, Disagg-pipeline for phase interference, Shared-store only when reuse pays for lookup and transfer, Memory-pool when CXL-like fabrics exist, and Hybrid-tier for multi-mechanism production stacks.
  • Evaluations that report only TTFT, TPOT, or goodput cannot identify which architectural choice paid off; the seven missing measurements become prerequisites for fair cross-archetype comparison.
  • Fault-tolerant, multi-tenant, and per-session KV designs are open because the taxonomy finds B4 (durable) empty and B1 (per-session) unoccupied by published systems.
  • A CXL-backed prefix cache with cross-session reuse and explicit ownership is a documented gap at the intersection of Memory-pool and Shared-store.
  • Closing completion semantics, reuse-distance distributions, and public KV-aware traces unblocks several design questions at once.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If ownership truly is residual free variance, future distributed KV papers should report control-plane cost and tail attribution as first-class results, not only data-path bandwidth.
  • The empty durable-KV region implies that production long-context or multi-turn services may be one store failure away from re-prefill storms that current goodput numbers never capture.
  • The taxonomy’s falsifiable C1-versus-C2 prediction is a ready A/B template for the next generation of disaggregated and shared-store systems.
  • As hybrid-tier systems compose more mechanisms, the field may need an explicitly compositional extension of the four axes rather than single-tuple labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This survey argues that KV cache has become a first-order distributed memory object in LLM serving and organizes more than thirty systems with a four-axis taxonomy (locality, lifetime, ownership, substrate). From those axes it induces five architectural archetypes—Local-paged, Disagg-pipeline, Shared-store, Memory-pool, and Hybrid-tier—and claims that, once workload and hardware are fixed, ownership explains much of the remaining design variance among distributed systems. The paper supplies operational classification rules, per-system tables with evidence tiers, boundary-case documentation, a measurement-gap audit (MG1–MG7), and a paired research agenda (DG1–DG7), including a falsifiable C1-versus-C2 tail-latency prediction.

Significance. The contribution is organizational rather than algorithmic, but it is timely and useful for a fast-moving systems area that has lacked a shared control-plane vocabulary. Strengths include an operational classification procedure (§3), explicit inclusion/exclusion rules, evidence-tier labeling in Tables 8–9, documented boundary systems and threats to validity (§4.5–§4.6), and a concrete falsifiable prediction (§5.4) that the authors correctly note cannot yet be checked because of MG6. The pairing of design gaps with required measurements (Table 12) is a practical community contribution beyond a pure literature map. If the taxonomy is adopted, it should improve cross-paper comparison and evaluation design for disaggregated and shared-store KV systems.

major comments (2)
  1. Abstract and §4.7 state that ownership “accounts for much of the remaining design variance” among distributed systems. The supporting evidence is qualitative pairwise comparison (DistServe vs. KVDirect; Mooncake vs. LMCache) plus Table 7’s approximate within-archetype counts, not a quantified variance decomposition. The claim is already hedged as empirical in §4.1 and §4.6, but the abstract wording is stronger than the evidence. Please either (i) soften the abstract/§4.7 phrasing to “ownership is the axis on which otherwise-matched distributed systems most often disagree in this population,” or (ii) add a short explicit statement of the comparison set size and that no formal variance attribution is claimed.
  2. §5.4’s C1-vs-C2 prediction is a genuine strength, but the proposed test pairs are not fully matched on unmodeled factors (implementation maturity, control-plane feature set, evaluation workloads, and publication venue). DistServe and KVDirect share (A1,B0,D-local+D1) by taxonomy, yet a negative result could be attributed to those confounds rather than ownership. Please state matching conditions and failure modes more carefully (what would count as a clean falsification versus an inconclusive comparison), so the prediction remains usable by follow-up work.
minor comments (6)
  1. Table 7 reports “approximate distinct values” for within-archetype variance. A one-sentence method note (how multi-valued systems were counted; whether boundary systems inflate Hybrid-tier variance) would make the concentration claim easier to re-check.
  2. Figure 3 omits B3 from the vertical axis while Table 6 includes B3 modifiers; the caption explains this, but a small legend entry for “B3 omitted (retention modifier)” on the figure itself would reduce reader confusion.
  3. §5.1 states “35 entries spanning 33 distinct systems” while the abstract says “more than thirty.” Align the abstract with the exact table count, or state the count once in §1.5 and reuse it.
  4. Several arXiv/framework systems carry quantitative performance language in surrounding prose even when the evidence-tier column is A/F. A consistent hedge (“author-reported”) at first mention of non-peer-reviewed numbers would match the methodology claim in §1.5.
  5. §3.2 notes B1 and B4 are empty; DG7 later motivates B1. Consider a single forward pointer from §3.2 to DG7 so readers do not treat empty levels as taxonomy defects.
  6. Minor copyediting: “interact” → “interacted” near the end of §2.4; check consistent hyphenation of “prefill/decode” vs. “prefill-decode” across sections.

Circularity Check

0 steps flagged

No significant circularity: survey taxonomy and archetype concentration are empirical classification, not a derivation that reduces to its inputs by construction.

full rationale

This is a systems survey. The load-bearing claims are (1) a four-axis taxonomy of locality/lifetime/ownership/substrate, (2) concentration of classified systems into five architectural envelopes, and (3) ownership as residual design variance among distributed systems once workload and hardware are fixed. Axes are defined operationally (§3), systems are assigned primary tuples from dominant contributions with documented boundary cases (Tables 8–9, §4.5), and archetypes are stated as empirical envelopes induced by repeated signatures rather than as a forced partition or fitted model (§4.1–4.2, Table 6). The ownership claim is a qualitative observation over the classified set (§4.7), not a prediction obtained by fitting a free parameter and re-labeling it. The §5.4 C1-vs-C2 tail-latency prediction is explicitly offered as currently untestable (MG6) and falsifiable, not as a result. Prior surveys are positioned as answering different questions (§7); there is no load-bearing uniqueness theorem, ansatz, or self-citation chain that forces the five envelopes or the ownership residual. Inclusion rules and threats to validity are stated openly (§1.5, §4.6). No step reduces a claimed prediction or first-principles result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 3 invented entities

As a systems survey, the paper does not fit physical constants or run regressions. Load-bearing commitments are definitional axes, inclusion/exclusion rules, and qualitative concentration/ownership claims over a hand-curated population. Free parameters are essentially none; invented entities are the taxonomy constructs themselves, which are analytical labels rather than physical postulates.

axioms (5)
  • ad hoc to paper A system’s dominant KV-management contribution can be assigned a primary (A,B,C,D) tuple, with secondary labels only for distinct modes.
    Classification rule in §3; enables the archetype map but is a methodological choice, not a theorem.
  • domain assumption Locality, lifetime, ownership, and substrate are sufficiently orthogonal that changing one while holding others fixed is meaningful.
    §3.6 orthogonality argument; authors note A3/C3 often co-occur with D3 but retain them separately.
  • ad hoc to paper Inclusion requires a material change to KV placement, lifetime, ownership, substrate, scheduling, reuse, or footprint; pure weight quantization and non-KV attention work are out of scope.
    §1.5 methodology; shapes the classified population and thus the five envelopes.
  • domain assumption Once workload and hardware fix locality/substrate/lifetime pressures, residual distributed design variance is largely ownership (C1 vs C2 vs C3).
    Central interpretive claim in abstract and §4.7; supported by examples, not a proved partition of variance.
  • domain assumption End-to-end metrics (TTFT, TPOT, goodput) are insufficient to attribute gains to specific KV mechanisms without MG1–MG7.
    §2.6 and §5.5 audit; standard systems-evaluation stance applied to KV.
invented entities (3)
  • Four-axis taxonomy (locality, lifetime, ownership, substrate) independent evidence
    purpose: Provide separable coordinates for classifying KV-management systems and frameworks.
    Primary analytical contribution; independent of any single system’s self-description.
  • Five architectural archetypes (Local-paged, Disagg-pipeline, Shared-store, Memory-pool, Hybrid-tier) independent evidence
    purpose: Name recurring envelopes induced by repeated axis signatures in the classified set.
    Empirical clustering labels; existence depends on the surveyed population and axis definitions.
  • Measurement gaps MG1–MG7 and design gaps DG1–DG7 independent evidence
    purpose: Enumerate missing KV-specific counters/traces and pair them with open architecture problems.
    Agenda constructs derived from the evaluation audit; falsifiable by future papers reporting those metrics.

pith-pipeline@v1.1.0-grok45 · 39776 in / 3497 out tokens · 36890 ms · 2026-07-12T09:49:30.446994+00:00 · methodology

0 comments
read the original abstract

The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes -- local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current evaluations and identifies seven missing KV-specific measurements, linking them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.

Figures

Figures reproduced from arXiv: 2607.02574 by Jie Li, Tongyang Wang, Yong Chen.

Figure 1
Figure 1. Figure 1: Prefill materializes key and value vectors for prompt tokens; decode then advances one token per step [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: KV cache footprint versus context length for representative LLM architectures in FP16/BF16. Solid [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Archetype map in the (𝐴, 𝐵) projection. Colored regions give a visual map of the five archetype envelopes from [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Timeline of LLM serving and KV-cache system designs. Systems are positioned by release date [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

139 extracted references · 54 linked inside Pith

  1. [1]

    J., Soloveychik, I., and Kamath, P.Keyformer: Kv cache reduction through key tokens selection for efficient generative inference.Proceedings of Machine Learning and Systems(2024)

    Adnan, M., Arunkumar, A., Jain, G., Nair, P. J., Soloveychik, I., and Kamath, P.Keyformer: Kv cache reduction through key tokens selection for efficient generative inference.Proceedings of Machine Learning and Systems(2024)

    2024

  2. [2]

    S., Tumanov, A., and Ramjee, R.Taming throughput-latency tradeoff in llm inference with sarathi-serve

    Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R.Taming throughput-latency tradeoff in llm inference with sarathi-serve. InUSENIX Symposium on Operating Systems Design and Implementation(2024)

    2024

  3. [3]

    S., and Ramjee, R.Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.arXiv preprint arXiv:2308.16369(2023)

    Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R.Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.arXiv preprint arXiv:2308.16369(2023)

    Pith/arXiv arXiv 2023

  4. [4]

    Ainslie, J., Lee-Thorp, J., De Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S.Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving 29 Methods in Natural Language Proce...

    2023

  5. [5]

    Y., Rajbhandari, S., Awan, A

    Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al.Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis(2022), IEEE, pp. 1–15

    2022

  6. [6]

    Barrett, C., Cai, R., Chen, B., Chen, T., Ré, C., Sheng, Y., Song, Z., Tian, Y., "atlas" W ang, Z., Zhang, Z., Zheng, L., and Zhou, T.H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems 36(2023), 34661–34710

    2023

  7. [7]

    E., and Cohan, A.Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150(2020)

    Beltagy, I., Peters, M. E., and Cohan, A.Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150(2020)

    Pith/arXiv arXiv 2004

  8. [8]

    Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J.Striped attention: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431(2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E. J., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radfor...

    2020

  10. [10]

    Cai, T., Chen, D., Huang, Y., Lewis, P., Li, Y., Locatelli, A., Venkitesh, B., Yang, B., and Ye, H.Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems 37(2024), 22947–22970

    2024

  11. [11]

    D., Chen, D., and Dao, T.Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774(2024)

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T.Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., and Xiao, W.Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Chen, C., Zhao, X., Cheng, G., Xu, Y., Deng, S., and Yin, J.Next-gen computing systems with compute express link: a comprehensive survey.arXiv preprint arXiv:2412.20249(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Chen, S., Jiang, R., Yu, D., Xu, J., Chao, M., Meng, F., Jiang, C., Xu, W., and Liu, H.Kvdirect: Distributed disaggregated llm inference.arXiv preprint arXiv:2501.14743(2024)

    Pith/arXiv arXiv 2024

  15. [15]

    Chen, S., Wong, S., Chen, L., and Tian, Y.Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595(2023)

    Pith/arXiv arXiv 2023

  16. [16]

    InInternational Conference on Learning Representations(2024), vol

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J.Longlora: Efficient fine-tuning of long-context large language models. InInternational Conference on Learning Representations(2024), vol. 2024, pp. 8220–8238

    2024

  17. [17]

    Child, R., Gray, S., Radford, A., and Sutskever, I.Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509(2019)

    Pith/arXiv arXiv 1904

  18. [18]

    [19]Corporation, A.Amd instinct mi300 series architecture.Technical whitepaper(2023)

    Consortium, C.Compute express link (cxl) specification 3.0.Technical specification, Revision 3.0, Version 1.0(2022). [19]Corporation, A.Amd instinct mi300 series architecture.Technical whitepaper(2023). [20]Corporation, N.Nvidia h100 tensor core gpu architecture.Technical whitepaper(2022). [21]Corporation, N.Nvidia grace hopper superchip architecture.Tech...

    2022

  19. [19]

    InInternational Conference on Learning Representations(2024), vol

    Dao, T.Flashattention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations(2024), vol. 2024, pp. 35549–35562

    2024

  20. [20]

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems 35(2022), 16344–16359

    2022

  21. [21]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Gpt3.int8(): 8-bit matrix multiplication for transformers at scale.Advances in neural information processing systems(2022)

    2022

  22. [22]

    Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D.Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078(2023)

    Pith/arXiv arXiv 2023

  23. [23]

    Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., W ang, W., Zheng, N., and Wei, F.Longnet: Scaling transformers to 1,000,000,000 tokens.arXiv preprint arXiv:2307.02486(2023)

    Pith/arXiv arXiv 2023

  24. [24]

    L., Zhang, C., Xu, Y., Shang, N., Xu, J., Y ang, F., and Y ang, M.Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753(2024)

    Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Y ang, F., and Y ang, M.Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753(2024)

    Pith/arXiv arXiv 2024

  25. [25]

    Dong, H., Y ang, X., Zhang, Z., W ang, Z., Chi, Y., and Chen, B.Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference.arXiv preprint arXiv:2402.09398(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Dragojević, A., Narayanan, D., Hodson, O., and Castro, M.Farm: fast remote memory.Networked Systems Design and Implementation(2014), 401–414

    2014

  27. [27]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., W ang, T., W ang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, 30 Jie Li, Tongyang Wang, and Yong Chen K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui, C.Glam: Effic...

    2022

  28. [28]

    Duan, J., Lu, R., Duanmu, H., Li, X., Zhang, X., Lin, D., Stoica, I., and Zhang, H.Muxserve: flexible spatial-temporal multiplexing for multiple llm serving.arXiv preprint arXiv:2404.02015(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Fedus, W., Zoph, B., and Shazeer, N.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research 23, 120 (2022), 1–39

    2022

  30. [30]

    K.Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.Advances in Neural Information Processing Systems(2024)

    Feng, Y., Lv, J., Cao, Y., Xie, X., and Zhou, S. K.Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.Advances in Neural Information Processing Systems(2024)

    2024

  31. [31]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323(2022)

    Pith/arXiv arXiv 2022

  32. [32]

    Y., Dao, T., Saab, K

    Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C.Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052(2022)

    Pith/arXiv arXiv 2022

  33. [33]

    Fu, Y., Bailis, P., Stoica, I., and Zhang, H.Break the sequential dependency of llm inference using lookahead decoding.arXiv preprint arXiv:2402.02057(2024)

    Pith/arXiv arXiv 2024

  34. [34]

    Proceedings of Machine Learning and Systems(2022)

    Gale, T., Narayanan, D., Young, C., and Zaharia, M.Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems(2022)

    2022

  35. [35]

    L., Khandelwal, A., and Zhong, L.Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems 6(2024), 325–338

    Gim, I., Chen, G., seob Lee, S., Sarda, N. L., Khandelwal, A., and Zhong, L.Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems 6(2024), 325–338

    2024

  36. [36]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., V aughan, A., et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Gu, A., and Dao, T.Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752 (2023)

    Pith/arXiv arXiv 2023

  38. [38]

    Gu, A., Goel, K., and Ré, C.Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396(2021)

    Pith/arXiv arXiv 2021

  39. [39]

    G.Efficient memory disaggregation with infiniswap

    Gu, J., Lee, Y., Zhang, Y., Chowdhury, M., and Shin, K. G.Efficient memory disaggregation with infiniswap. Networked Systems Design and Implementation(2017), 649–667

    2017

  40. [40]

    In International conference on machine learning(2020), PMLR, pp

    Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.Retrieval augmented language model pre-training. In International conference on machine learning(2020), PMLR, pp. 3929–3938

    2020

  41. [41]

    Han, C., W ang, Q., Xiong, W., Chen, Y., Ji, H., and W ang, S.Lm-infinite: Simple on-the-fly length generalization for large language models.arXiv preprint arXiv:2308.16137(2023)

    Pith/arXiv arXiv 2023

  42. [42]

    He, J., Qiu, J., Zeng, A., Y ang, Z., Zhai, J., and Tang, J.Fastmoe: A fast mixture-of-expert training system.arXiv preprint arXiv:2103.13262(2021)

    Pith/arXiv arXiv 2021

  43. [43]

    He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B.Zipcache: Accurate and efficient kv cache quantization with salient token identification.Advances in Neural Information Processing Systems 37(2024), 68287–68307

    2024

  44. [44]

    A., Welbl, J., Clark, A., et al.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556 10 (2022)

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556 10 (2022)

    Pith/arXiv arXiv 2022

  45. [45]

    B., Awan, A

    Holmes, C., Tanaka, M., Wyatt, M. B., Awan, A. A., Rasley, J., Rajbhandari, S., Aminabadi, R. Y., Qin, H., Bakhtiari, A., Kurilenko, L. V., and He, Y.Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference.arXiv preprint arXiv:2401.08671(2024)

    Pith/arXiv arXiv 2024

  46. [46]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A.Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems 37(2024), 1270–1303

    2024

  47. [47]

    Hu, C., Huang, H., Hu, J., Xu, J., Chen, X., Xie, T., W ang, C., W ang, S., Bao, Y., Sun, N., et al.Memserve: Context caching for disaggregated llm serving with elastic memory pool.arXiv preprint arXiv:2406.17565(2024)

    Pith/arXiv arXiv 2024

  48. [48]

    Hwang, C. H., Cui, W., Xiong, Y., Y ang, Z., Liu, Z., Hu, H., W ang, Z., Salas, R., Jose, J., Ram, P., Chau, J., Cheng, P., Y ang, F., Y ang, M., and Xiong, Y.Tutel: Adaptive mixture-of-experts at scale.Proceedings of Machine Learning and Systems 5(2023), 269–287

    2023

  49. [49]

    Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E.Atlas: Few-shot learning with retrieval augmented language models.Journal of Machine Learning Research 24, 251 (2023), 1–43

    2023

  50. [50]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y.Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509 (2023)

    Pith/arXiv arXiv 2023

  51. [51]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al.Mixtral of experts.arXiv preprint arXiv:2401.04088(2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al.Minference 1.0: From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving 31 Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems 37(2024)...

    2024

  53. [53]

    Jin, Y., Wang, T., Lin, H., Song, M., Li, P., Ma, Y., Shan, Y., Yuan, Z., Li, C., Sun, Y., et al.P/d-serve: Serving disaggregated large language model at scale.arXiv preprint arXiv:2408.08147(2024)

    Pith/arXiv arXiv 2024

  54. [54]

    Johnson, J., Douze, M., and Je´gou, H.Billion-scale similarity search with gpus.IEEE Transactions on Big Data 7, 3 (2019), 535–547

    2019

  55. [55]

    Y., Ré, C., and Mirhoseini, A.Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099(2024)

    Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Y., Ré, C., and Mirhoseini, A.Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099(2024)

    Pith/arXiv arXiv 2024

  56. [56]

    Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T.Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm.arXiv preprint arXiv:2403.05527(2024)

    Pith/arXiv arXiv 2024

  57. [57]

    InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)(2020), pp

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t.Dense passage retrieval for open-domain question answering. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)(2020), pp. 6769–6781

    2020

  58. [58]

    InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval(2020), pp

    Khattab, O., and Zaharia, M.Colbert: Efficient and effective passage search via contextualized late interaction over bert. InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval(2020), pp. 39–48

    2020

  59. [59]

    Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451(2020)

    Pith/arXiv arXiv 2001

  60. [60]

    H., Gonzalez, J., Zhang, H., and Stoica, I.Efficient memory management for large language model serving with pagedattention

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I.Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles(2023), pp. 611–626

    2023

  61. [61]

    InProceedings of the 57th annual meeting of the association for computational linguistics(2019), pp

    Lee, K., Chang, M.-W., and Toutanova, K.Latent retrieval for weakly supervised open domain question answering. InProceedings of the 57th annual meeting of the association for computational linguistics(2019), pp. 6086–6096

    2019

  62. [62]

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z.Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668(2020)

    Pith/arXiv arXiv 2006

  63. [63]

    T., and Rabusseau, G.Kq-svd: Compressing the kv cache with provable guarantees on attention fidelity.arXiv preprint arXiv:2512.05916(2025)

    Lesens, D., Rakhshan, B. T., and Rabusseau, G.Kq-svd: Compressing the kv cache with provable guarantees on attention fidelity.arXiv preprint arXiv:2512.05916(2025)

    arXiv 2025

  64. [64]

    In International Conference on Machine Learning(2023), PMLR, pp

    Leviathan, Y., Kalman, M., and Matias, Y.Fast inference from transformers via speculative decoding. In International Conference on Machine Learning(2023), PMLR, pp. 19274–19286

    2023

  65. [65]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D.Retrieval-augmented generation for knowledge-intensive nlp tasks.UCL Discovery (University College London) 33(2020), 9459–9474

    2020

  66. [66]

    S., Hsu, L., Ernst, D., Zardoshti, P., Novakovic, S., Shah, M., Rajadnya, S., Lee, S., Agarwal, I., et al.Pond: Cxl-based memory pooling systems for cloud platforms

    Li, H., Berger, D. S., Hsu, L., Ernst, D., Zardoshti, P., Novakovic, S., Shah, M., Rajadnya, S., Lee, S., Agarwal, I., et al.Pond: Cxl-based memory pooling systems for cloud platforms. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(2023), pp. 574–587

    2023

  67. [67]

    Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., Hu, N., Dong, W., Li, Q., and Chen, L.A survey on large language model acceleration based on kv cache management.arXiv preprint arXiv:2412.19442(2024)

    Pith/arXiv arXiv 2024

  68. [68]

    Li, Y., Wei, F., Zhang, C., and Zhang, H.Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077(2024)

    Pith/arXiv arXiv 2024

  69. [69]

    E., et al

    Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)(2023), pp. 663–679

    2023

  70. [70]

    Lin, B., Zhang, C., Peng, T., Zhao, H., Xiao, W., Sun, M., Liu, A., Zhang, Z., Li, L., Qiu, X., et al.Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache.arXiv preprint arXiv:2401.02669 (2024)

    Pith/arXiv arXiv 2024

  71. [71]

    Lin, J., Tang, J., Tang, H., Shang, Y., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S.Awq: Activation-aware weight quantization for llm compression and acceleration.Proceedings of machine learning and systems 6(2024), 87–100

    2024

  72. [72]

    Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B.Minicache: Kv cache compression in depth dimension for large language models.Advances in Neural Information Processing Systems 37(2024), 139997–140031

    2024

  73. [73]

    InProceedings of the 2026 ACM/SIGDA International Symposium on Field Programmable Gate Arrays(2026), pp

    Liu, D., and Yu, Y.Cxl-speckv: A disaggregated fpga speculative kv-cache for datacenter llm serving. InProceedings of the 2026 ACM/SIGDA International Symposium on Field Programmable Gate Arrays(2026), pp. 56–66

    2026

  74. [74]

    Liu, H., and Abbeel, P.Blockwise parallel transformers for large context models.Advances in neural information processing systems 36(2023), 8828–8844

    2023

  75. [75]

    In International Conference on Learning Representations(2024), vol

    Liu, H., Zaharia, M., and Abbeel, P.Ringattention with blockwise transformers for near-infinite context. In International Conference on Learning Representations(2024), vol. 2024, pp. 3992–4008

    2024

  76. [76]

    32 Jie Li, Tongyang Wang, and Yong Chen

    Liu, X., Tang, Z., Dong, P., Li, Z., Li, B., Hu, X., and Chu, X.Chunkkv: Semantic-preserving kv cache compression for efficient long-context llm inference.Advances in Neural Information Processing Systems 38(2026), 28728–28778. 32 Jie Li, Tongyang Wang, and Yong Chen

    2026

  77. [77]

    Liu, Y., Cheng, Y., Yao, J., An, Y., Chen, X., Feng, S., Huang, Y., Shen, S. S. P., Zhang, R., Du, K., and Jiang, J. Lmcache: An efficient kv cache layer for enterprise-scale llm inference.arXiv preprint arXiv:2510.09665(2025)

    arXiv 2025

  78. [78]

    InProceedings of the 4th International Conference on Artificial Intelligence and Intelligent Information Processing(2025), pp

    Liu, Y., Fu, J., Liu, S., Zou, Y., Zhang, S., and Zhou, J.Kv cache compression for inference efficiency in llms: A review. InProceedings of the 4th International Conference on Artificial Intelligence and Intelligent Information Processing(2025), pp. 207–212

    2025

  79. [79]

    InProceedings of the ACM SIGCOMM 2024 Conference(2024), pp

    Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Y ao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., and Jiang, J.Cachegen: Kv cache compression and streaming for fast large language model serving. InProceedings of the ACM SIGCOMM 2024 Conference(2024), pp. 38–56

    2024

  80. [80]

    Liu, Y., and Qian, C.Trinity: Disaggregating vector search from prefill-decode disaggregation in llm serving.arXiv preprint arXiv:2512.02281(2025)

    arXiv 2025

Showing first 80 references.