Pith. sign in

REVIEW 2 major objections 8 minor 22 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

KV cache optimization for LLM serving can be decomposed into three orthogonal system behaviors that reveal missed co-design opportunities

2026-07-10 00:47 UTC pith:GNCIWYER

load-bearing objection Solid survey with a useful taxonomy; the co-design affinity analysis is novel but rests on under-validated subjective inputs. the 2 major comments →

arxiv 2607.08057 v1 pith:GNCIWYER submitted 2026-07-09 cs.LG cs.AIcs.CL

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

Jiantong Jiang , Peiyu Yang , Rui Zhang , Feng Liu This is my paper
classification cs.LG cs.AIcs.CL
keywords servingcacheinfrastructurelanguagelargellmssurveysystem-aware
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the sprawling literature on optimizing the key-value cache — the memory structure that lets language models avoid recomputing past token states during generation — can be organized into a clean three-dimensional space: temporal behaviors (when KV data is scheduled or computed), spatial behaviors (where KV data is placed or moved across memory tiers and devices), and structural behaviors (how KV data is compressed or retained). The authors call this scope sKis, short for system-aware KV infrastructure for serving LLMs, and restrict it to techniques that operate at inference time without retraining the model or modifying its architecture. By mapping roughly 100 methods onto these three axes and then computing a co-occurrence network across the resulting categories, the survey identifies which behavior pairs are commonly co-designed and which remain isolated. The central finding is that most KV cache techniques optimize a single behavior in isolation, and the most productive co-design pattern — pairing hardware-aware execution with compute-device orchestration — is still narrowly applied, while compression methods (quantization, low-rank approximation, structural pruning) are almost never integrated with the scheduling, pipelining, or migration mechanisms that would let their memory savings translate into actual latency or throughput gains. The paper uses this gap analysis to motivate six open challenges, including SLO-driven tail-latency control, energy-aware optimization, and trustworthy serving under compression-induced quality degradation.

Core claim

The paper's central organizational claim is that KV cache optimization decomposes into temporal, spatial, and structural behaviors, and that the relationships between these behaviors — not the behaviors themselves — are where the field's biggest gaps lie. The co-design affinity network shows that hardware-aware execution and compute-device orchestration form the strongest observed pairing, while KV cache compression, despite being the most heavily studied individual behavior, is nearly disconnected from all other behaviors in the literature. This isolation means that compression algorithms reduce memory footprints but rarely produce end-to-end system gains because the (de)compression costs,I

What carries the argument

The survey constructs two analytical instruments: a behavior × objective matrix (Table 7) that maps each of the seven sub-behaviors to serving objectives like mean latency, tail latency, throughput, GPU memory, interconnect I/O, energy, and quality impact; and a behavior-behavior co-design affinity network (Figure 6) computed via Tanimoto-normalized co-occurrence of primary and secondary category assignments across all surveyed papers. Together these instruments let the authors identify which behaviors directly impact which objectives, which behavior pairs are commonly co-designed, and which pairs represent missed opportunities.

Load-bearing premise

The co-design affinity network is built from the authors' own assignment of each paper to a primary category and optionally a secondary category, with a fixed weight of 0.5 for secondary associations and a display threshold of 0.14. These choices are not subjected to sensitivity analysis, so the key structural findings — particularly that compression is isolated and that hardware-aware execution pairs most strongly with compute-device orchestration — depend on classification

What would settle it

If an independent group of researchers re-categorized the same set of papers using the same seven sub-behaviors but different primary/secondary assignments, and the co-design affinity network changed substantially — for instance, if compression no longer appeared isolated, or if a different behavior pair emerged as the strongest co-design pattern — then the paper's key observations O6 and O7, and the open challenges derived from them, would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the three-behavior taxonomy is adopted by the community, future KV cache papers would be expected to report which behaviors they touch and whether they co-design across behaviors, making cross-paper comparison more systematic than the current practice of reporting isolated speedup or memory numbers.
  • The finding that compression is isolated from system integration suggests that the next wave of impactful work may come not from better compression algorithms but from systems that fuse compression with migration, scheduling, and runtime control — for example, co-deciding eviction, offload, and prefetch under shared bandwidth budgets.
  • The identification of energy as an under-explored objective across all behaviors implies that power-aware KV cache management — profiling energy per token, integrating power constraints into scheduling decisions — is an open frontier with practical data-center relevance.
  • The observation that tail latency is rarely reported, despite temporal behaviors mapping cleanly to latency reduction, suggests that current benchmarks may be systematically hiding the failure modes that matter most for user experience.
  • The dual-use observation that KV eviction can serve as a defense mechanism against jailbreak attacks hints at a broader convergence between efficiency optimization and safety mechanisms in LLM serving.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Tanimoto normalization used to compute co-design affinity depends on the authors' assignment of primary versus secondary categories to each paper, which is a subjective editorial decision. If the co-occurrence scores are sensitive to reassignment of borderline cases, the strongest and weakest edges in the network could shift, potentially weakening the claim that compression is uniquely isolate
  • The taxonomy's three dimensions map loosely onto classical operating-system concerns (scheduling, memory management, I/O), which raises the question of whether a fourth dimension — security or fault tolerance — is implicitly absorbed into the structural dimension or genuinely missing, given that the paper identifies trustworthiness as an open challenge but does not assign it a behavioral axis.
  • The restriction to no-retraining, no-architecture-change methods means the survey excludes techniques like MLA (multi-head latent attention) that reshape the KV footprint at the model level. As such methods become standard in production models, the sKis scope may need to expand or explicitly address the boundary between serving-time and model-level KV optimization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. This survey reviews system-aware KV cache optimization for LLM serving (sKis), covering serving-time, KV-centric methods that improve system metrics without retraining. The authors organize ~100 methods into a three-dimensional taxonomy: temporal (execution and scheduling), spatial (placement and migration), and structural (representation and retention). The paper further contributes a behavior-objective matrix (Tab. 7) and a behavior-behavior co-design affinity network (Fig. 6, App. F) computed via Tanimoto-normalized co-occurrence of category assignments. From these analyses, the authors derive seven observations (O1-O7) and six open challenges (C1-C6). The scope is well-motivated and the taxonomy is a genuinely useful lens that is decoupled from model and kernel details, as claimed.

Significance. The survey's principal contribution is its behavior-oriented taxonomy, which is more stable and analytically tractable than prior lifecycle- or layer-based organizations (Tab. 1, Tab. 8). The co-design affinity computation (App. F) is transparently specified with explicit formulas, enabling reproducibility. The cross-behavior analysis (Fig. 6, Tab. 7) is a novel analytical device for a survey and successfully highlights under-explored regions such as energy-aware sKis (O4, C2) and KVCC isolation (O7, C5). The comprehensive method mapping (Tab. 9, App. D) and the curated GitHub repository add practical value. The open challenges (C1-C6) are well-grounded in the observations and are actionable.

major comments (2)
  1. §6.1, Fig. 6, App. F: The co-design affinity network is the paper's most novel analytical contribution, generating O6 (HAE-CDO strongest co-design) and O7 (KVCC isolated), which directly motivate C4 and C5. However, the computation depends on three layers of subjective or unjustified parameter choices: (1) primary vs. secondary category assignments in Tab. 9, made by author judgment; (2) the secondary-category weight α=0.5, chosen without justification; (3) the edge threshold θ=0.14, described as chosen to 'reduce clutter' (an aesthetic criterion). Examining Tab. 12, the normalized affinity for HAE-CDO is 0.53 and HAE-OVLP is 0.42 — a gap that could plausibly close if a few papers' primary/secondary labels shifted. The raw co-occurrence matrix (Tab. 11) shows HAE-OVLP at 9.25 vs. HAE-CDO at 10.0, a small margin. If the ranking flips under reasonable parameter perturbations, O6 is an arte
  2. Tab. 7 (behavior × objective matrix): The cells are filled subjectively (direct ● vs. indirect #, with ⋆ for ≥70% reporting gains), but no formal rubric for distinguishing 'direct' from 'indirect' is provided, and no inter-annotator agreement or validation against external ground truth is reported. For a survey, some subjectivity is expected, but the paper's claim that the taxonomy enables 'principled analysis' (§1, §6) is weakened when the analytical artifacts rest on unvalidated subjective judgments. The authors should at minimum provide a brief rubric for the direct/indirect distinction and acknowledge the subjectivity as a limitation. This issue is load-bearing for O2 (temporal behaviors act most directly on latency and throughput) and the derived challenges C1 and C2.
minor comments (8)
  1. §2, Fig. 1: The term 'sKis' is introduced as an abbreviation but the capitalization is unconventional. A brief note on pronunciation or rationale for the styling would improve readability.
  2. Tab. 3: The 'Avg. bits' column is described as 'based on the reported main results' but the specific model, context length, and configuration for each method are not standardized. A footnote noting that these are not directly comparable across methods would help.
  3. Tab. 6: The 'Budget policy' column uses abbreviations (L, H) that are defined in the footnote, but the distinction between 'Preset' and 'Adaptive' could be clarified with a one-line explanation in the caption.
  4. §5.1.1: The statement 'lower bitwidth does not always translate into end-to-end system gains' is important but is stated briefly. A concrete example or reference would strengthen this practical insight.
  5. App. F, Tab. 11: The raw co-occurrence matrix includes self-co-occurrence (diagonal) as '—' but it is unclear whether the diagonal is excluded by design or simply not computed. Clarifying this would improve reproducibility.
  6. §6.2, C3: The dual-use observation (Jiang et al., 2025b, turning KV eviction into defense against jailbreak) is interesting but introduced very briefly. A sentence explaining the mechanism would help readers unfamiliar with that work.
  7. References: Several arXiv preprints lack clear venue information. Where possible, updating to published versions would strengthen the bibliography before camera-ready.
  8. Fig. 2: The taxonomy figure is compact but the sub-bullets (e.g., 'KV-centric scheduling', 'Pipelining') are small. Enlarging or restructuring for legibility would help.

Circularity Check

0 steps flagged

No circularity found: survey paper's analytical contributions are descriptive, not derivational

full rationale

This is a survey paper that organizes existing literature rather than deriving or predicting results from fitted parameters. The paper's two analytical contributions—the behavior × objective matrix (Tab. 7) and the behavior-behavior co-design affinity network (Fig. 6, App. F)—are computed from the authors' own category assignments of surveyed papers, not from fitting parameters to data and then claiming those fits as predictions. The co-design affinity scores use a Tanimoto normalization of primary/secondary category co-occurrences (App. F, Eqs. for C_ij and S_ij), which is a descriptive aggregation of subjective labels, not a derivation chain where outputs reduce to inputs by construction. The observations O1–O7 and challenges C1–C6 are interpretive conclusions drawn from these aggregations. While the subjectivity of category assignments and the lack of sensitivity analysis for α=0.5 and θ=0.14 are legitimate methodological concerns (better suited for a correctness or robustness critique), they do not constitute circularity: the paper does not claim to predict something that is definitional or fitted by construction. Self-citations (Jiang et al. 2025a/b/c) appear as data points within the taxonomy, not as load-bearing premises for the taxonomy's validity. No step in the paper's reasoning reduces to its own inputs by definition, ansatz, or self-citation chain. The derivation chain is self-contained as a descriptive survey with no circular logic.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The paper introduces two hand-set parameters (α, θ) for its co-design analysis and relies on the assumption that its taxonomy cleanly partitions methods. The 'sKis' label is a scoping device. No new physical entities or forces are postulated.

free parameters (2)
  • Secondary category weight (α) = 0.5
    Used in the co-design affinity computation (App. F) to weight secondary category associations. Chosen by hand without sensitivity analysis.
  • Affinity edge threshold (θ) = 0.14
    Edges in the co-design network (Fig. 6) are drawn only if S_ij > 0.14. This threshold is set to reduce clutter but affects which co-design patterns are visible.
axioms (3)
  • domain assumption KV cache optimization techniques can be cleanly partitioned into temporal, spatial, and structural behaviors.
    This is the foundational taxonomy assumption stated in §2. While reasonable, some methods span multiple dimensions, and the authors acknowledge a single paper may touch several categories.
  • domain assumption The sKis scope (serving-time, KV-centric, no retraining) is a meaningful and distinct research boundary.
    The paper excludes model architecture changes and training-time compression (§2, Fig. 1). This is a practical but somewhat artificial boundary, as some methods blur the line between serving-time adaptation and lightweight fine-tuning.
  • ad hoc to paper Co-occurrence of behaviors in papers reflects meaningful co-design affinity rather than mere topical trends.
    The co-design affinity network (Fig. 6) and derived observations (O6, O7) assume that co-occurrence implies co-design. This is an interpretive assumption; papers may co-occur behaviors for convenience rather than deep co-design.
invented entities (1)
  • sKis (system-aware KV infrastructure for serving LLMs) no independent evidence
    purpose: Defines the paper's scope as serving-time, KV-centric, no-retraining optimization.
    A scoping label rather than a physical entity. It is a naming convention for the survey's boundary, not a measurable phenomenon.

pith-pipeline@v1.1.0-glm · 40361 in / 2224 out tokens · 280237 ms · 2026-07-10T00:47:28.186846+00:00 · methodology

0 comments
read the original abstract

Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.

Figures

Figures reproduced from arXiv: 2607.08057 by Feng Liu, Jiantong Jiang, Peiyu Yang, Rui Zhang.

Figure 1
Figure 1. Figure 1: Positioning of the survey scope (“sKis”). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prefill and decode phases of LLM inference. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Taxonomy of sKis and associated methods. Each method is annotated with its primary contributions for [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of KV cache placement and migra [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (behavior-behavior co-design affinity net￾work) visualizes cross-behavior co-occurrence in the literature, with edge thickness proportional to normalized weights. Low-score edges are omitted, and computation details are provided in App. F). This affinity reflects observed co-design patterns rather than validated performance gains. We con￾clude key observations (O1-O7) as follows. O1. Structural works are m… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 10 internal anchors

  1. [1]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He

    KVCompose: Efficient structured KV cache compression with composite tokens.arXiv preprint arXiv:2509.05165. Reza Yazdani Aminabadi, Samyam Rajbhandari, Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed- Inference: enabling efficient inference of transformer models at...

    work page arXiv 2022

  2. [2]

    QAQ: Quality Adaptive Quantization for LLM KV Cache

    QAQ: Quality adaptive quantization for LLM KV cache.arXiv preprint arXiv:2403.04643. Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, and Feng Lyu. 2025. Accelerating LLM inference throughput via asynchronous KV cache prefetching.arXiv preprint arXiv:2504.06319. Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion S...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference

    LazyLLM: Dynamic token pruning for effi- cient long context LLM inference.arXiv preprint arXiv:2407.14057. Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-efficient large lan- guage model serving for multi-turn conversations with CachedAttention. InUSENIX Annual Technical Con...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    InConference on Empirical Methods in Natural Language Processing, pages 21158–21166

    Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters. InConference on Empirical Methods in Natural Language Processing, pages 21158–21166. Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, and Feng Liu. 2025. Trustworthy machine learning: From data to models.Foundations and Trends® in Privac...

    work page arXiv 2025

  5. [5]

    GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM

    GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serv- ing with PagedAttention. InProceedings ...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    A Survey on Large Language Model Acceleration based on KV Cache Management

    Hong Kong world: Leveraging structural regularity for line-based SLAM.IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 45(11):13035–13053. Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. 2024b. A survey on large language model acceleration based on KV cache man- agemen...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  7. [7]

    Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

    Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234. NVIDIA. 2023. TensorRT-LLM. https://github. com/NVIDIA/TensorRT-LLM. NVIDIA. 2025a. GenAI-Perf.NVIDIA Docs. NVIDIA. 2025b. NVIDIA NIM LLMs benchmarking. NVIDIA Docs. OpenAI. 2023. GPT-4 technical report.arXiv preprint arXiv:2303...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Mooncake: A KVCache-centric disaggre- gated architecture for LLM serving.arXiv preprint arXiv:2407.00079. Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. CAKE: Cascading and adaptive KV cache eviction with layer preferences. InInternational Conference on Learning Representations. Haoran Qiu, Weichao Mao...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

    Power-aware deep learning model serving with µ-serve. InUSENIX Annual Technical Conference, pages 75–93. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsu...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.21203/rs.3.rs- 2018

  10. [10]

    Idris O Sunmola, Zhenjun Zhao, Samuel Schmidgall, Yumeng Wang, Paul Maria Scheikl, and Axel Krieger

    PMLR. Idris O Sunmola, Zhenjun Zhao, Samuel Schmidgall, Yumeng Wang, Paul Maria Scheikl, and Axel Krieger

    work page

  11. [11]

    Surgical Gaussian Surfels: Highly Accurate Real-time Surgical Scene Rendering using Gaussian Surfels

    Surgical gaussian surfels: Highly accurate real-time surgical scene rendering.arXiv preprint arXiv:2503.04079. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: query- aware sparsity for efficient long-context LLM infer- ence. InInternational Conference on Machine Learn- ing, pages 47901–47911. Hongduan Tian, Fe...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    A Survey of Large Language Models

    A survey of large language models.arXiv preprint arXiv:2303.18223. Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024b. Atom: Low- bit quantization for efficient and accurate LLM serv- ing.Proceedings of Machine Learning and Systems, 6:196–209. Youpeng Zhao, Di Wu, an...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    A Survey on Efficient Inference for Large Language Models

    DistServe: Disaggregating prefill and decod- ing for goodput-optimized large language model serv- ing. InUSENIX Symposium on Operating Systems Design and Implementation, pages 193–210. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Gra- ham Neubig. 2024a. WebAre...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    retain” vs. “evict

    where errors can have severe consequences. One representative example (also related to our discussion inC3) is that KV cache eviction and compression can compromisequality robustness. They may drop rare but critical tokens with low ac- cumulated attention (e.g., an exception clause in a contract, or a high value in a financial limit), which can lead to ca...

    work page 2025

  15. [15]

    BALI (Jurkschat et al., 2025) measures LLM inference across six frameworks or acceler- ation approaches

    evaluates the inference performance of the LLaMA model family across a variety of hardware platforms. BALI (Jurkschat et al., 2025) measures LLM inference across six frameworks or acceler- ation approaches. It divides inference into three measured stages: setup, tokenize, and generate, and supports two settings: a technical setting with a fixed number of ...

    work page 2025

  16. [16]

    Workloads.A future sKis benchmark should cover the following three workload types to stress temporal, spatial, and structural KV behaviors:

    KV-related resource metricsthat measure the utilization of resources, such as KV cache mem- ory footprint (as % of total GPU memory), av- erage effective KV bitwidth (for compression methods), KV-related interconnect I/O (the vol- ume of KV transferred across memory tiers), KV hit rate in memory tiers, KV-related stalls (% of time spent waiting for KV tra...

    work page

  17. [17]

    Multi-tenant or bursty online serving workloads to test the stability of temporal scheduling under high concurrency

    work page

  18. [18]

    Long-context task workloads to test KV cache placement and migration when memory and I/O become bottlenecks

    work page

  19. [19]

    Reporting standards.In addition to the basic in- formation like model, hardware, and configuration, we recommend the following reporting standards for sKis benchmarks:

    Heterogeneous workloads, such as agent work- loads (Zeng et al., 2025c; Zhou et al., 2024a), structured editing (Zeng et al., 2025d), or domain-shifted workloads (Tian et al., 2024, 2026; Li et al., 2023, 2025b; Yan et al., 2025a; Zeng et al., 2025a), to test the robustness of structural KV cache optimizations against dis- tribution shift. Reporting stand...

    work page 2024

  20. [20]

    Performance under graduated context lengths to validate scalability

    work page

  21. [21]

    memory curves for structural methods to reveal the trade-offs

    Accuracy vs. memory curves for structural methods to reveal the trade-offs

    work page

  22. [22]

    Detailed hardware and topology setups, espe- cially for temporal and spatial methods

    work page