Pith. sign in

REVIEW 2 major objections 5 minor 21 references

A shared LLM prefix cache can be work-conserving and causally fair: charge each newly materialized KV block to the tenant that created it, and let that outstanding 'admission debt' decide whose state gets evicted.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 23:30 UTC pith:V3ELPYBA

load-bearing objection A real gap, a credible mechanism, and an honest scope restriction—worth refereeing if the artifact question gets addressed. the 2 major comments →

arxiv 2608.01657 v1 pith:V3ELPYBA submitted 2026-08-03 cs.DC

Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Caches

Zhiyu Wang , Rajkumar Buyya This is my paper
classification cs.DC
keywords LLM servingprefix cachingKV cachecache replacementmulti-tenant fairnessadmission responsibilitywork-conserving sharingcache pollution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Shared prefix caching lets one LLM tenant's traffic silently evict another tenant's reusable state from GPU memory: the tenant that keeps introducing new, one-shot prefixes creates the pressure, while the tenant that repeatedly uses a stable hot prefix loses the cached state it depends on. The paper calls this the admission-responsibility gap and argues that closing it requires tracking which tenant created each new block of cached state, not just which blocks look valuable. Its proposed mechanism, PrefixShield, charges each newly materialized KV block to the tenant that caused its creation, prevents that tenant's replay traffic from converting unpaid debt into high-retention status, and — only when the shared cache is actually full — selects eviction victims from the tenant with the largest outstanding debt. In paired experiments on a production LLM serving system under one-touch pollution, the victim's cache hit ratio rises from 5.54% with LRU to 14.93%, a 9.39-point gain over LRU and 8.64 points over the reuse-aware S3-FIFO policy, with the gap widening at scale: at 4096 blocks the victim is restored from 4.92% to 84.87%. The paper's point is that object-value signals rank what to retain, but persistent responsibility — not reuse history — determines which tenant pays.

Core claim

PrefixShield claims that persistent admission responsibility—carried by the group that materializes new full KV blocks—protects a victim tenant beyond reuse-aware replacement, without static partitions or admission denial. Two invariants do the work: complete pressure attribution (every new full block is charged exactly once to its creating group; pure reuse charges nothing) and no debt laundering (reuse evidence is recorded, but promotion to the REUSE class is gated while the owning group's debt is outstanding). When the cache is contested, eviction splits in two: projected normalized debt selects which group supplies victims, and a retention-class order (probationary, normal, reuse) select

What carries the argument

The load-bearing object is admission debt: a per-tenant, block-denominated token bucket that is debited for every newly materialized full KV block and refilled at a configured rate, where a negative balance is outstanding responsibility rather than a quota. Debt drives two decisions: it gates reuse promotion into the high-retention REUSE class (invariant I2, no debt laundering), and, normalized by contract entitlement, it selects which group must supply eviction victims when capacity is contested (argmax over projected debt); within the chosen group, retention classes PROBATIONARY < NORMAL < REUSE pick the specific block. The third piece is two-stage eviction itself: responsibility names the

Load-bearing premise

The protection assumes every cached token block can be attributed to exactly one tenant, the group that first materialized it, and that tenants never reuse each other's cached content — cross-group shared prompts would require new multi-owner accounting rules. It also protects a victim only while the polluting tenant creates blocks faster than its operator-configured refill entitlement; at the boundary the gain over LRU is zero.

What would settle it

Two concrete tests. (1) Run one-touch pollution with the attacker's creation rate at or below its configured refill entitlement (measured λ_new/ρ ≈ 0.54): the paper's own sweep shows the victim-hit gain over LRU falls to zero. (2) Let two tenants share a common system prompt so the same cached blocks are hittable by both: if namespace-scoped lookup refuses the cross-group hit, memory is wasted despite identical content; if it allows the hit, block ownership and debt attribution become ambiguous. Which outcome occurs settles the mechanism's scope.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A globally shared prefix cache can be both work-conserving and causally fair: while a peer is idle, an active group occupies 509–510 of 511 usable blocks, yet the group that creates sustained pressure is the group that bears reclamation.
  • Reuse-aware replacement and responsibility-aware reclamation solve complementary problems — object value ranks what to retain, group responsibility selects who pays — so they can be stacked rather than substituted, which is why PrefixShield beats S3-FIFO under one-touch churn and replay.
  • Responsibility is temporary by design: once a group stops creating new state, debt clears under quiescence at the configured refill rate, so no permanent penalty or post-attack reconfiguration is required.
  • The mechanism extends beyond two tenants: debt accrues independently per accounting group, so concurrent aggressors each accumulate their own responsibility and reclamation targets the group with the largest projected normalized debt.
  • The operator-chosen refill rate and burst capacity define the boundary between benign and harmful creation: measured creation below the refill entitlement accrues no responsibility (gain over LRU is zero at ratio 0.54), and the protection scales with the creation-to-refill ratio.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper explicitly defers cross-group content deduplication; extending PrefixShield to shared system prompts would require multi-principal ownership rules, since a block hittable by two tenants can no longer be charged to a single creator — a genuinely open design problem rather than a minor tweak.
  • Because the protection depends on the creation-to-refill ratio, the refill configuration itself is an attack surface: a tenant holding a high-entitlement contract can generate sustained pressure without accruing debt, so deployments should audit measured λ_new/ρ per contract rather than assume the policy alone is protective.
  • The two-stage logic — first decide who pays, then which object — should transfer to other shared-state systems where one client's writes displace another's expensive-to-reconstruct state, such as distributed CPU caches or multi-tenant key-value stores, though the paper does not claim this.
  • A testable corollary of the paper's own sweep: at creation rates within the configured entitlement, PrefixShield's victim-hit gain over LRU is exactly zero, implying the mechanism is best characterized as an entitlement-enforcement layer on top of any value-aware replacement policy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper addresses the admission-responsibility gap in multi-tenant LLM prefix caching, where one tenant's creation of new KV blocks can displace another tenant's reusable cached state while existing policies rank object value or partition capacity. The authors propose PrefixShield, which meters newly materialized full KV blocks per accounting group/contract, maintains a refill token bucket, gates promotion to the REUSE retention class until debt clears (Eq. 4), and performs two-stage eviction: first selecting the group with highest projected normalized debt (Eqs. 5–8), then selecting a block by PROBATIONARY/NORMAL/REUSE segment order. The mechanism is implemented in vLLM and evaluated against LRU, S3-FIFO, and StaticQuota on synthesized traces derived from production characterizations. Headline results are a 9.39 pp victim hit-ratio gain over LRU and 8.64 pp over S3-FIFO under one-touch pollution at 512-block scale, a gain of 2.00 pp over S3-FIFO under two-pass replay, restoration of victim hits from 4.92% to 84.87% at 4096 blocks, statistical equivalence to S3-FIFO on benign ShareGPT traffic, and work-conserving occupancy of 509–510/511 blocks by an active borrower. The paper also reports an operating-boundary sweep (zero gain at λ_new/ρ = 0.54), two ablations (reuse segmentation and debt-gated promotion), and overhead measurements.

Significance. If the central results hold, PrefixShield provides a previously missing control point: persistent attribution of cache-creation pressure to the group that caused it, without static partitioning or admission denial. The main strengths are the clear design principle, the implementation in a production-grade system (vLLM), paired 10-run comparisons with bootstrap CIs, a monotone operating-boundary sweep, and mechanism ablations isolating the two novel components. The delayed-replay ablation (Fig. 3b) is particularly persuasive: the debt gate prevents reuse laundering and the advantage disappears exactly at the designed clearance time. The central limitation is the namespace-scoped ownership assumption: cross-group content reuse is explicitly excluded (Section II.C), so the headline protection is demonstrated only for workloads with disjoint per-group prefixes; this restricts the practical scope of the contribution. The parameter ρ_c is also a free operator choice whose default operating point is not reported, making the headline gain's location on the operating boundary unclear.

major comments (2)
  1. [Section II.C, Section IV, Section V] The paper explicitly assumes native prefix lookup is namespace-scoped by accounting group and states cross-group content reuse is outside the prototype. This is a load-bearing restriction: the ownership and debt-accounting invariants (Eqs. 2–5, Algorithm 1) require a single immutable owner per block. In production multi-tenant serving, tenants commonly share system prompts, templates, and retrieved context; under the current design such content is materialized once per group, multiplying KV memory where a content-addressed cache would deduplicate. All evaluated traces use disjoint per-group prefixes, so the paper does not quantify memory amplification or hit-ratio behavior under shared content. The abstract and introduction should clearly state that the mechanism assumes no cross-group deduplication, or the evaluation should add a shared-prefix experiment (e.g., the same 1,536-token pref
  2. [Section III.B, Section V.D, Fig. 2b] The headline protection gain is zero at measured λ_new/ρ = 0.54 and increases to 59.07 pp at 4.23, so the benefit is contingent on an operator-configured refill rate ρ_c that is below the actual new-block creation rate. The paper does not report the default one-touch workload's λ_new/ρ, so the reader cannot locate the headline 9.39 pp gain on the operating-boundary curve. It also provides no practical guidance for setting ρ_c and β_c beyond 'operators can derive ρ_c from tier-level state-creation budgets.' The paper should report the default ratio, add a marker in Fig. 2b, and include a sensitivity experiment showing the effect of mis-setting ρ_c (e.g., 2× high or 2× low) on the victim-protection and benign-workload results.
minor comments (5)
  1. [Section V.A] The one-touch pressure rate uses min{0.75Cprefill, 1.15αCprefill}. The constants 0.75 and 1.15 are not motivated; please clarify their origin and how they relate to the trace characterization.
  2. [Fig. 2b] The x-axis label 'Measured λ_new/ρ' should define λ_new (new-block creation rate) and mark the default one-touch operating point; currently the reader cannot relate the sweep to the headline numbers.
  3. [Section III.D, Eq. (8)] The projected-debt decrement uses W_g(t), the maximum weight among a group's indebted contracts, independent of the contract that actually owns the evicted block. Please justify this conservative choice and note its effect in multi-contract groups.
  4. [Section V.D] The overhead reports 'Python metadata occupies 218.2 KiB at 512 blocks/8 groups' without a vLLM baseline; a relative overhead would be more informative. Also, 'Mean P95 TTFT' in Section V.B is awkward; use 'mean of per-run P95 TTFT'.
  5. [References] Several references (e.g., [2], [8], [16]) have stray spaces before periods in author initials (e.g., 'Y . Sheng'); please fix. Also, the abstract's 'restoring the victim from 4.92% to 84.87% at 4096-block scale' should point to Fig. 2a.

Circularity Check

0 steps flagged

No significant circularity: PrefixShield's protection claims are empirically tested against external baselines; the responsibility mechanism is an implemented policy, not an input restated as a prediction.

full rationale

The paper's derivation chain runs from a design principle (the accounting group that creates persistent cache pressure remains responsible until debt recovers) to three concrete mechanisms: metering newly materialized KV blocks, gating reuse promotion while debt is outstanding, and using projected debt to select the eviction-supplying group. The headline results are measured paired gains over external baselines (LRU, S3-FIFO, StaticQuota) under one-touch pollution, low-and-slow rotation, and two-pass replay (Section V.B, Table I). These are not fitted parameters renamed as predictions: the refill rates and burst capacities are fixed policy configuration, and the creation-pressure sweep in Section V.D explicitly reports that at a measured λ_new/ρ ratio of 0.54 the gain is zero, which is a stated operating boundary rather than a hidden fitted input. The proof of quiescent debt clearance (Proposition 1) is a direct consequence of the token-bucket definition, but it is not used to claim an empirical result. There are no load-bearing self-citations: all cited prior systems (vLLM, S3-FIFO, FairRide, NyxCache, etc.) are external, and no uniqueness theorem or design premise is justified by the authors' own prior work. The closest thing to a circularity concern is the scope restriction in Section II.C: 'Cross-group content reuse would require multi-principal rules for ownership and admission responsibility and is outside the current prototype.' This limits the evaluation to disjoint-prefix tenant workloads and is explicitly acknowledged, including as future work in Section VIII. A scope limitation that narrows external validity is not circularity. The mechanism's attribution of eviction to the debtor group is a design invariant, while the empirical content—whether that attribution protects victim hit ratios without hurting benign traffic or work conservation—is genuinely tested. The paper is self-contained against external benchmarks and does not reduce its central claim to its inputs.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The paper's central claim rests on operator-chosen accounting parameters (refill rate ρ_c, burst β_c), a clean group-separation model, and two design invariants (debt-gated promotion, responsibility-selected eviction). None of these are derived from external principles; they are the mechanism. The paper experimentally sweeps the most sensitive one (λ_new/ρ).

free parameters (6)
  • Refill rate ρ_c per contract = 64, 40, 16 blocks/s (critical, standard, best-effort)
    Operator-configured entitlement for new-block creation. The headline protection depends on measured creation exceeding ρ: at a measured λ_new/ρ of 0.54 the gain over LRU is zero (Fig. 2b).
  • Burst capacity β_c = 256, 192, 128 blocks
    Short-term burst credit in the token bucket; shapes when debt begins to accrue relative to traffic bursts.
  • One-touch attack pressure rate = min{0.75 C_prefill, 1.15 α C_prefill}
    Hand-picked constants that set attacker creation relative to configured refill; the default operating point falls in the debt-accruing regime.
  • Cache capacity and victim working set = 512 blocks (511 usable); 384-block victim hot set
    Deliberately extreme contention: the victim set is 75% of usable capacity; other scales are swept in Fig. 2(a).
  • Block size and shared prefix length = 16 tokens/block; 1,536-token prefixes
    Determines the 96-block-per-prefix hot set; an experimental choice, not a production-cache estimate.
  • Debt-clearance tolerance ε = ≥ 0, exact value not stated
    Implementation tolerance for treating debt as cleared (Eqs. 4, 6); its calibrated value is not given.
axioms (5)
  • standard math Token-bucket refill and debt arithmetic (Eqs. 1-3): stored balance refills lazily as min(β, B + ρ·Δt), and debt is the max of 0 and -balance.
    Simple accounting arithmetic used throughout; internally consistent and cross-checked against reported debt values in Section V.D.
  • domain assumption Clean tenant separability: every block has an immutable accounting owner and prefix lookup is namespace-scoped by group.
    Stated in Sections II.C and IV; excludes cross-group content deduplication, which is common in production shared prompts. The paper flags this as outside the prototype.
  • domain assumption Adversary model: a legitimate client controlling only its own token sequences and timing; unbounded group creation and network-level attacks are excluded.
    Section II.C bounds the problem to request-level pollution; the paper acknowledges this scope.
  • ad hoc to paper Debt-gated promotion rule (Eq. 4): promotion to REUSE requires proven reuse and D_g(t) ≤ ε.
    The anti-laundering design invariant; its benefit is measured in the Fig. 3(b) delayed-replay ablation rather than derived.
  • ad hoc to paper Eviction ordering: responsibility selects the group (Eqs. 5-7), then segment order PROBATIONARY < NORMAL < REUSE selects the block (Eq. 9), with projected debt decremented by 1/W_g (Eq. 8).
    Selector design; the segmentation part is ablated in Fig. 3(a), and the 1/W_g decrement is a conservative contract-agnostic choice.
invented entities (1)
  • Admission debt (raw debt D_g, normalized debt êD_g, and projected responsibility P(j)_g) independent evidence
    purpose: Carries creation responsibility of new KV blocks across requests and selects which group supplies eviction candidates under contention.
    A new ledger quantity, not a physical entity. It is operationalized end-to-end: debt levels are measured, the debt gate is ablated (Fig. 3b), and the predicted effect (gated groups promote nothing while debt is outstanding) is directly observable, so it has a falsifiable handle in the system.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Caches." pith.science (2026). https://pith.science/paper/V3ELPYBA

@misc{pith2026260801657,
  author       = {Pith},
  title        = {Pith review of: Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Caches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3ELPYBA}},
  note         = {Machine review of arXiv:2608.01657}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Shared prefix caching turns Graphics Processing Unit (GPU) memory into persistent state shared across Large Language Model (LLM) tenants. A group that materializes new Key-Value (KV) blocks can force another to lose reusable state, yet request-time schedulers account for transient service, replacement policies primarily rank object value, and static partitioning strands idle capacity. We call this mismatch the admission-responsibility gap. To close it, we propose PrefixShield, which meters newly materialized full KV blocks, carries responsibility across requests, gates reuse promotion while debt remains, and uses projected debt to select the group supplying eviction candidates. We implement PrefixShield in vLLM. In paired runs under one-touch pollution, PrefixShield improves victim cache hit ratio by 9.39 percentage points over the Least Recently Used (LRU) policy and 8.64 points over S3-FIFO, restoring the victim from 4.92% to 84.87% at 4096-block scale, and gains 2.00 points over S3-FIFO under two-pass replay. It preserves benign ShareGPT behavior and work-conserving access to idle capacity. Delayed replay yields a 35.16-point advantage while debt remains. These results show that object-value signals rank what to retain, while persistent responsibility determines which group bears reclamation pressure.

Figures

Figures reproduced from arXiv: 2608.01657 by Rajkumar Buyya, Zhiyu Wang.

Figure 1
Figure 1. Figure 1: PrefixShield’s request-to-reclamation lifecycle. New KV blocks accrue admission responsibility; cache hits provide reuse evidence. Debt gates promotion [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scaling, operating boundary, and deployment sensitivity. Panels report paired PrefixShield-minus-LRU victim cache-hit gains with 95% bootstrap CIs: [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mechanism isolation. (a) Paired cache-hit gain of full PrefixShield over no reuse segmentation, with 95% bootstrap CIs. (b) Post-pressure victim hit [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

  1. [1]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” inProceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2023, pp. 611–626

    work page 2023

  2. [2]

    SGLang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “SGLang: Efficient execution of structured language model programs,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024

    work page 2024

  3. [3]

    Stateful large language model serving with Pensieve,

    L. Yu, J. Lin, and J. Li, “Stateful large language model serving with Pensieve,” inProceedings of the European Conference on Computer Systems (EuroSys), 2025, pp. 144–158

    work page 2025

  4. [4]

    Online context caching for distributed large language models serving,

    B. Gao, Z. He, Y . Yao, Z. Lou, Z. Zhou, and W.-F. Wong, “Online context caching for distributed large language models serving,” in Proceedings of the IEEE Conference on Computer Communications (INFOCOM), 2025, pp. 1–10

    work page 2025

  5. [5]

    MELL: Memory-efficient large language model serving via multi-GPU KV cache management,

    Q. Liu, Z. Hong, F. Chen, P. Li, and S. Guo, “MELL: Memory-efficient large language model serving via multi-GPU KV cache management,” inProceedings of the IEEE Conference on Computer Communications (INFOCOM), 2025, pp. 1–10

    work page 2025

  6. [6]

    Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot,

    R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot,” inProceedings of the USENIX Conference on File and Storage Technologies (FAST), 2025, pp. 155–170

    work page 2025

  7. [7]

    KVCache cache in the wild: Characterizing and optimizing KVCache cache at a large cloud provider,

    J. Wang, J. Han, X. Wei, S. Shen, D. Zhang, C. Fang, R. Chen, W. Yu, and H. Chen, “KVCache cache in the wild: Characterizing and optimizing KVCache cache at a large cloud provider,” inProceedings of the USENIX Annual Technical Conference (USENIX ATC), 2025, pp. 465–482

    work page 2025

  8. [8]

    Fairness in serving large language models,

    Y . Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” inProceedings of the USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI), 2024, pp. 965–988

    work page 2024

  9. [9]

    FIFO queues are all you need for cache eviction,

    J. Yang, Y . Zhang, Z. Qiu, Y . Yue, and K. V . Rashmi, “FIFO queues are all you need for cache eviction,” inProceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2023, pp. 130–149

    work page 2023

  10. [10]

    FairRide: Near- optimal, fair cache sharing,

    Q. Pu, H. Li, M. Zaharia, A. Ghodsi, and I. Stoica, “FairRide: Near- optimal, fair cache sharing,” inProceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2016, pp. 393–406

    work page 2016

  11. [11]

    NyxCache: Flexible and efficient multi-tenant persistent memory caching,

    K. Wu, K. Tu, Y . Patel, R. Sen, K. Park, A. Arpaci-Dusseau, and R. Arpaci-Dusseau, “NyxCache: Flexible and efficient multi-tenant persistent memory caching,” inProceedings of the USENIX Conference on File and Storage Technologies (FAST), 2022, pp. 1–16

    work page 2022

  12. [12]

    TinyLFU: A highly efficient cache admission policy,

    G. Einziger, R. Friedman, and B. Manes, “TinyLFU: A highly efficient cache admission policy,”ACM Transactions on Storage, vol. 13, no. 4, pp. 1–31, 2017

    work page 2017

  13. [13]

    CacheBlend: Fast large language model serving for RAG with cached knowledge fusion,

    J. Yao, H. Li, Y . Liu, S. Ray, Y . Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang, “CacheBlend: Fast large language model serving for RAG with cached knowledge fusion,” inProceedings of the European Conference on Computer Systems (EuroSys), 2025, pp. 94–109

    work page 2025

  14. [14]

    Oneiros: KV cache optimization through parameter remapping for multi-tenant LLM serving,

    R. Li, S. Pal, V . N. Pullu, P. Sinha, J. Ryoo, L. K. John, and N. J. Yadwadkar, “Oneiros: KV cache optimization through parameter remapping for multi-tenant LLM serving,” inProceedings of the ACM Symposium on Cloud Computing (SoCC), 2025, pp. 88–101

    work page 2025

  15. [15]

    ServeGen: Workload characterization and generation of large language model serving in production,

    Y . Xiang, X. Li, K. Qian, Y . Zhang, W. Yu, E. Zhai, X. Jin, and J. Zhou, “ServeGen: Workload characterization and generation of large language model serving in production,” inProceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2026, pp. 1845–1859

    work page 2026

  16. [16]

    IC-Cache: Efficient large language model serving via in-context caching,

    Y . Yu, Y . Gan, N. Sarda, L. Tsai, J. Shen, Y . Zhou, A. Krishnamurthy, F. Lai, H. M. Levy, and D. E. Culler, “IC-Cache: Efficient large language model serving via in-context caching,” inProceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2025, pp. 375– 398

    work page 2025

  17. [17]

    CacheGen: KV cache compression and streaming for fast large language model serving,

    Y . Liu, H. Li, Y . Cheng, S. Ray, Y . Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang, “CacheGen: KV cache compression and streaming for fast large language model serving,” inProceedings of the ACM SIGCOMM Conference (SIGCOMM), 2024, pp. 38–56

    work page 2024

  18. [18]

    ThunderServe: High-performance and cost-efficient LLM serving in cloud environments,

    Y . Jiang, F. Fu, X. Yao, T. Wang, B. Cui, A. Klimovic, and E. Yoneki, “ThunderServe: High-performance and cost-efficient LLM serving in cloud environments,” inProceedings of Machine Learning and Systems (MLSys), vol. 7, 2025

    work page 2025

  19. [19]

    Jenga: Effective memory management for serving LLM with heterogeneity,

    C. Zhang, K. Du, S. Liu, W. Kwon, X. Mo, Y . Wang, X. Liu, K. You, Z. Li, M. Long, J. Zhai, J. E. Gonzalez, and I. Stoica, “Jenga: Effective memory management for serving LLM with heterogeneity,” in Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2025, pp. 446–461

    work page 2025

  20. [20]

    LServe: Efficient long-sequence LLM serving with unified sparse attention,

    S. Yang, J. Guo, H. Tang, Q. Hu, G. Xiao, J. Tang, Y . Lin, Z. Liu, Y . Lu, and S. Han, “LServe: Efficient long-sequence LLM serving with unified sparse attention,” inProceedings of Machine Learning and Systems (MLSys), vol. 7, 2025

    work page 2025

  21. [21]

    Memory- efficient KV cache optimization for large language model inference at the edge,

    C. Zhang, H. Tan, H. Pan, Y . Xu, H. Du, L. Zhang, and X. Fu, “Memory- efficient KV cache optimization for large language model inference at the edge,” inProceedings of the IEEE Conference on Computer Communications (INFOCOM), 2026, pp. 1–10

    work page 2026

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.