Pith. sign in

REVIEW 2 major objections 5 minor 46 references

HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HiSparse claims that long-context sparse-attention decoding can be scaled by keeping each request's full KV history in host memory and giving it a fixed-size GPU cache, yielding up to 4.7x peak generation throughput without changing model…

desk verdict HiSparse is a genuine, well-engineered systems contribution that turns the KV capacity wall into a bandwidth tradeoff, but its headline gains rest on a single trace of selection locality and a no-IO oracle claim that needs error bars. read the letter →

arxiv 2608.07009 v1 pith:FZYMMHDA submitted 2026-08-07 cs.DC

Zhiqiang Xie , Zhangheng Huang , Tingwei Huang , Ziyi Xu , Ruiyang Ma , Christos Kozyrakis This is my paper · ORCID
classification cs.DC
keywords KVcachesparseattentionLLMservinghierarchicalmemoryGPUprefetchinglong-contextinferencedecodethroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HiSparse argues that long-context sparse-attention serving hits a capacity wall because systems keep the entire KV cache in GPU HBM even though each decode step reads only a few thousand selected entries. The paper's claim is that a request's full KV history can live in host memory while its decode footprint on the GPU is bounded by a small, fixed-size cache, because sparse selections have strong temporal locality. A fused kernel resolves each layer's selections—hit detection, LRU replacement, and host fetches—inside the steady-state decode graph, and exact layer-wise prefetching hides miss latency for models that share selections across layers. Because only KV placement changes, model outputs are identical to the full-HBM baseline. The payoff is that decode throughput scales with cache size rather than context length, up to 4.7x on long-context workloads.

What carries the argument

The central mechanism is a two-level KV hierarchy: an authoritative pinned host-memory pool holds every request's full KV history, while a fixed-size GPU cache of B KV-record slots per request and layer holds only recently selected records, managed by LRU with a per-step refinement that promotes hits above newly fetched misses. The load-bearing identity is the footprint bound: per-request decode HBM becomes the product of the number of layers, B, the KV elements per token, and bytes per element, instead of growing with context length, which converts a capacity bottleneck into a tunable latency/bandwidth tradeoff. Three pieces carry the argument: the fused RESOLVE CUDA kernel (a shared-memory hash table over the selected positions, a parallel probe and scan to mark hits and choose victims, vectorized non-coherent loads from pinned host memory, and page-table publication to the attention backend), the GPU-assisted IO path that keeps scattered host fetches near link bandwidth, and exact layer-wise prefetch that replays an anchor layer's miss plan for models sharing selections across layers. The trace study in the evaluation justifies LRU: at B=2k it misses 13.4% of selections on the long-context trace, versus 30% when only the current top-k set is staged.

What would settle it

Feed a synthetic or real indexer that selects k uniformly random positions at each decode step on the H200 testbed: if the LRU cache's miss rate stays near 100%, host-to-device traffic saturates the PCIe link, and peak generation throughput falls well below the reported 4.7x gain (or TPOT degrades sharply), the temporal-locality premise is refuted; similarly, running the same benchmark on a Grace-based system with host memory comparable to HBM would test the second-tier-capacity assumption.

Watch

Extended reading notes

Core claim

HiSparse demonstrates that top-k sparse attention's per-step demand signal—the set of logical positions each layer selects—can be used to decouple logical KV availability from physical GPU residency. The system keeps the complete KV cache in pinned host memory and serves each request's decode from a small per-layer GPU cache of B slots, with B at least k and typically 2k to 4k; because B is independent of context length, the per-request decode HBM footprint stays fixed even as the context grows. A fused CUDA kernel resolves each layer's selections—building a hash table over the selected positions, probing the cache to mark hits, choosing LRU victims, fetching misses from host memory with vectorized non-coherent loads, and publishing physical slots to the attention backend—entirely inside the steady-state decode graph. For models that share indexer selections across layers, an exact prefetch scheme replays an anchor layer's miss plan into the shared layers, overlapping host transfers with computation and hiding roughly half the remaining IO. Since only KV placement changes, model outputs are identical to the full-HBM baseline; measured peak generation throughput improves by up to 4.7x on long-context workloads, and a no-IO oracle shows the resolution mechanism adds no measurable per-token cost.

Load-bearing premise

HiSparse's throughput gains depend on sparse selections having strong temporal locality, so that a small per-layer LRU cache (B=2k) holds most selections on the GPU and host-to-device miss traffic stays off the decode critical path; the paper additionally assumes host DRAM is much larger than GPU HBM, a premise it acknowledges fails on Grace-based systems.

Editorial extensions

If this is right

  • A request's decode-time HBM footprint becomes proportional to the GPU-cache size B, not the context length; at 128K tokens the per-request KV footprint drops from 13.09 GB to about 0.4 GB at B=4096.
  • Peak generation throughput on long contexts improves by up to 4.7x because the same HBM admits a much larger decode batch; decode-only (disaggregated) throughput gains reach about 2.9x in the measured configuration.
  • TTFT at high load drops because bounded decode residency leaves HBM headroom for prefill work in PD-colocated serving, shifting the queueing wall to higher concurrency.
  • Contexts whose KV cache exceeds HBM become servable, with the maximum context set by host-tier capacity rather than device memory.
  • Exact layer-wise prefetching for shared-index models hides roughly half of the remaining IO exposure, cutting per-token latency by 13–15% at matched concurrency without changing outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If sparse selections in other workloads are as locally clustered as the measured trace, the same hierarchy should transfer to document-grounded agents and code repositories, but the design's sensitivity to miss rate means operators should profile their own selection streams before adopting the default B=2k.
  • The exact-prefetch result suggests a testable model-design principle: architectures that share indexer selections across layers convert KV placement into a scheduling problem, so one could measure how much additional anchor-layer density (e.g., sharing every two layers instead of every four) improves throughput before host-link bandwidth saturates.
  • A three-tier extension (GPU cache, host DRAM, and storage) is the natural next step once host memory becomes the binding constraint, but the prefetch and IO paths would need to adapt to the much higher latency of the third tier, which the current design does not address.
  • Because the miss-resolution kernel consumes only logical positions, any future sparse-attention selector emits the same interface; the system's value is independent of a selector's accuracy, so it should compose with algorithmic improvements to indexers without serving-side changes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents HiSparse, a hierarchical KV-cache management system for serving top-k sparse-attention LLMs. HiSparse keeps the complete KV history of each request in pinned host DRAM and bounds the per-request GPU-side decode footprint with a fixed-size LRU-managed cache of B KV-record slots per request and layer. A fused CUDA kernel (RESOLVE) performs hit detection, eviction, metadata update, and batched host-to-device fetches inside the decode CUDA graph, and for models that share indexer selections across layers an exact layer-wise prefetch overlaps remaining transfers with intervening computation. The paper claims exact sparse-attention outputs (only KV placement changes), indexer-agnostic operation across DSA, NSA, and Quest, and up to 4.7x peak generation throughput on long-context workloads on H200, B200, and GH200, with the implementation merged into upstream SGLang.

Significance. If the results hold, HiSparse addresses a genuine and increasingly important bottleneck: top-k sparse attention reduces attention compute but not the HBM capacity cost of residency. The design is exact by construction, which is a strong correctness property, and the paper is unusually honest in separating the capacity benefit from the IO overhead. The upstream SGLang integration is a substantial artifact, and the evaluation spans three model families and three hardware platforms. The central exactness mechanism is valid, and no step in the paper reduces to a fitted quantity: the capacity-wall arithmetic is an accounting identity, the hit-rate numbers come from traces, and the end-to-end gains are measured against an unmodified baseline. The main risk is the narrowness of the locality evidence and the corresponding uncertainty about how the quantitative gains generalize.

major comments (2)
  1. [§4.3] The locality evidence that carries the design's IO argument rests on a single trace: one GLM-5.1 request on a 100,384-token LongBenchV2 prompt, replayed over 1,000 decode steps, from which the 13.4% miss rate at B=4096 and the associated 87% hit rate are derived. This is the only direct evidence that a small LRU cache keeps host-to-device fetch traffic small enough at high batch; §4.4 shows that IO dominates resolve time at batch 64 on H200, so a workload with weaker locality would directly erode or eliminate the 4.7x peak-throughput claim. The end-to-end benchmarks prove the system works on the tested workloads, but they do not measure in situ hit rates, and the other evaluated selectors (Quest on Qwen, NSA on DeepSeek-V4) may have different selection locality than DSA on GLM-5.1. Please add per-model selection-trace studies (or in-situ hit-rate collection during the end-to-end runs) across DSA, NSA, and Quest on multiple workload types, plus a sensitivity sweep of end-to-end throughput and TPOT versus miss rate or host-link bandwidth.
  2. [§4.6] The no-IO oracle skips host-memory IO entirely, so it provides a valid upper bound on any IO-hiding scheme, but it does not measure the achievable overlap under real link contention: exact prefetching repositions transfers rather than eliminating them, and Figure 8 reports 11.2 ms of exposed IO per token at concurrency 256 even with prefetch. To support the claim that host-device IO is the only price of bounded residency, the paper should report measured host-link utilization or an equivalent contention metric during the end-to-end runs, and compare the oracle against a more realistic bound that charges prefetch traffic for link bandwidth. As written, the claim that the resolve mechanism itself adds no measurable per-token cost is established only at low concurrency where link contention is absent.
minor comments (5)
  1. [§1, §3.1] The abstract and §1 state that a request's decode-time HBM consumption scales with the GPU-cache size rather than with context length; this is exact only for attention KV records, since §3.2 notes that page tables, LRU metadata, and indexer state remain HBM-resident and grow with Lctx. The paper should either consistently phrase the bound as applying to KV records (as §3.1 does with 'up to metadata') or quantify the metadata term in the 1M-token feasibility discussion.
  2. [§5, §7] The conclusion's statement that the maximum servable context is set by host-tier capacity should be explicitly re-scoped to platforms where host DRAM is much larger than HBM, since §5 correctly acknowledges that this fails on Grace-based GB200/GB300 systems.
  3. [Figure 7] The legend and caption describe H200 as 'wide light lines' and GH200 as 'dark dashed lines,' but these styles are hard to distinguish in grayscale; please use distinct marker shapes and label the curves directly in each panel.
  4. [Figure 8] The no-IO oracle serves stale KV records and therefore has invalid outputs; the text is explicit about this, but the caption should state it more prominently so that readers do not mistake the oracle for a real configuration.
  5. [§4.3] The name 'Belady' is spelled with diacritics ('Bélády') in several places; standard usage in the literature omits the accents.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's load-bearing claims rest on accounting identities, disclosed design invariants, trace measurements, and comparisons against an unmodified baseline, not on fits or self-citations that reduce to their own inputs.

full rationale

The derivation chain contains no step that reduces to its own input by construction. The capacity-wall statement is an arithmetic identity: a decode batch of Nbatch requests at length Lctx requires Nbatch times Lctx tokens of resident KV while attention reads only Nbatch times k tokens per step, so the claimed saturation behavior is not a prediction extracted from fitted parameters. The bounded-footprint claim, 'A request's decode-time HBM consumption therefore scales with the GPU-cache size rather than with its context length,' follows directly from the definition of B as a per-request, per-layer fixed cache in Section 3.2 and is presented as a design invariant rather than as an empirical payoff. Similarly, 'Because only KV placement changes, model outputs are unchanged' is a construction property of the miss-resolution interface: the selected positions and attention kernels are unmodified, and the paper does not dress this invariant up as a measured result. The locality numbers that underwrite the IO tradeoff are measured by replaying an external LongBenchV2 selection trace in Section 4.3, and the 4.7x peak-throughput figure is an end-to-end measurement against unmodified SGLang v0.5.11 in Section 4.1. Reuse of the authors' own HiCache host-tier infrastructure and Strata's GPU-assisted IO is explicitly disclosed as implementation provenance in Appendix A and is not load-bearing in the argument; no invoked theorem or prior result is used to forbid alternatives. The paper also states its main vulnerability directly in Section 5, noting that the host-DRAM-much-larger-than-HBM assumption fails on Grace-based GB200/GB300 systems, which is an honest limitation rather than a hidden circular premise. The single-trace basis for the 87% hit rate is a legitimate generalization concern, but it is a matter of external validity, not circularity: the hit rate is measured, not derived from the throughput claims it is used to explain. No equation or claim in the paper is equivalent to its own inputs by definition, no fitted parameter is renamed as a prediction, and no self-citation carries the weight of the central result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the domain assumptions listed above; there are no hidden fitted constants in the derivation sense. B is an explicit, documented serving knob, but the performance numbers depend on its value, so it is a free parameter in the ledger. No unobserved physical entities are postulated: the GPU cache, page tables, and RESOLVE kernel are implemented software components with observable behavior.

free parameters (3)
  • GPU cache size B (slots per request-layer) = B = 2k (e.g., 4096 for k=2048; range 2k-4k)
    Serving configuration parameter chosen by profiling sweeps (§4.4). It directly sets the HBM footprint per request and the hit rate, so the reported throughput gains depend on the chosen value.
  • Host-to-device ratio (host-pool capacity) = not fixed; set by --hisparse-config
    Configuration parameter in --hisparse-config; sets how much host DRAM is reserved relative to device KV budget, determining how many long-context requests can be staged.
  • Swap-in transfer block size = tuned per platform; not numerically specified
    Per-thread transfer block size in the GPU-assisted IO path (§3.4); tuned for link bandwidth, affecting miss-fetch latency and thus TPOT.
assumptions (5)
  • domain assumption Top-k sparse attention reads exactly the selected k KV entries per query; unselected entries do not affect the output.
    Definition of the sparse-attention interface assumed by HiSparse (§2.1); true for DSA, NSA, and Quest as described.
  • domain assumption The sparse selector emits selected positions before the attention kernel touches KV records, providing an interposition point.
    Required for miss resolution to happen before attention; stated as a shared property of the three selector families (§2.1).
  • domain assumption Sparse selections exhibit strong temporal locality, so LRU with a small cache (B≈2k) captures most reuse.
    Load-bearing for the whole design; supported by one LongBenchV2 trace (§4.3) and cited observations [6], but assumed to generalize across workloads.
  • domain assumption Host DRAM is much larger than GPU HBM and pinned host memory can hold full KV histories.
    Explicitly acknowledged as the fundamental limitation (§5); holds on the PCIe H200 testbeds but not on Grace-based GB200/GB300.
  • domain assumption GPU-assisted IO (ld.global.nc.v2.b64 against pinned host memory) achieves near-link bandwidth for scattered fetches.
    Borrowed from the authors' Strata work [36] and relied on for the miss-fetch path (§3.4); not independently benchmarked in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management." pith.science (2026). https://pith.science/paper/FZYMMHDA

@misc{pith2026260807009,
  author       = {Pith},
  title        = {Pith review of: HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZYMMHDA}},
  note         = {Machine review of arXiv:2608.07009}
}
read the original abstract

Top-k sparse attention makes long-context LLM decoding cheap to compute: each step reads only a few thousand selected KV entries rather than the full context. Serving systems, however, typically keep the entire KV cache in GPU HBM so that every position stays selectable, so a request's memory bill still grows with its full context length--decoding hits a capacity wall long before it runs out of compute, and a context whose KV cache exceeds HBM cannot be served at all. We present HiSparse, an exact, indexer-agnostic hierarchical KV cache for sparse-attention serving. HiSparse keeps each request's full KV history in host memory and bounds its decode footprint with a small, fixed-size GPU cache; a fused CUDA kernel resolves each layer's selections--hit detection, LRU replacement, and host-to-device fetches--inside the decode CUDA graph; and, for models that share selections across layers, exact layer-wise prefetching hides roughly half of the remaining miss overhead. Because only KV placement changes, model outputs are unchanged. HiSparse is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, and Quest) on H200, B200, and GH200 platforms: it improves peak generation throughput by up to 4.7x on long-context workloads while preserving comparable per-token latency and reducing time-to-first-token at high load--and a no-IO oracle shows the resolution mechanism itself adds no measurable per-token cost, leaving host-device IO as the only price of bounded residency.

Figures

Figures reproduced from arXiv: 2608.07009 by the authors.

Figure 1
Figure 1. HiSparse decouples decoding throughput from GPU memory capacity on long-context [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. HiSparse overview. Host memory keeps the authoritative full KV cache of every active [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The fused miss-resolution kernel. For one request and layer, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: End-to-end serving of DeepSeek-V4-Flash (NSA) on [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Peak generation throughput across input lengths for two additional sparse-attention families, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Per-step top-k miss rate (averaged across layers, smoothed) when replaying the same LongBenchV2 sparse-selection trace of GLM-5.1 (k=2048) under seven cache configurations; B counts KV-record slots per request and layer. Top-k-only staging (Swap-vanilla, B=2048) misses…
Figure 7
Figure 7. Figure 7: Miss-resolution breakdown across models, GPU-cache sizes, and platforms: H200 with [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Layer-wise exact prefetching with IndexCache-shared selections: GLM-5.2-FP8 (DSA) [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 27 canonical work pages

  1. [1]

    LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Ji- azheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    doi:10.18653/v1/2025.acl-long.183 2025

  2. [2]

    IndexCache: Accelerating sparse attention via cross-layer index reuse, 2026

    Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. IndexCache: Accelerating sparse attention via cross-layer index reuse, 2026. URL https: //arxiv.org/abs/2603.12201

    arXiv 2026

  3. [3]

    Laszlo A. Belady. A study of replacement algorithms for a virtual-storage computer.IBM Systems Journal, 5(2):78–101, 1966. doi: 10.1147/sj.52.0078

    doi:10.1147/sj.52.0078 1966

  4. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer,

  5. [5]

    ArkVale: Efficient generative LLM inference with recallable key-value eviction

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. ArkVale: Efficient generative LLM inference with recallable key-value eviction. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  6. [6]

    ESS: An offload- centric latent-cache management architecture for DeepSeek-V3.2-Exp, 2025

    Xinhang Chen, Chao Zhang, Jiahuan He, Wei Liu, Jianming Zhang, Wenlong Zhou, Xiao Li, Pai Zeng, Shiyong Li, Yuanpan Qian, Dong Li, and Zhaogeng Li. ESS: An offload- centric latent-cache management architecture for DeepSeek-V3.2-Exp, 2025. URL https: //arxiv.org/abs/2512.10576

    work page arXiv 2025

  7. [7]

    MagicPIG: LSH sampling for efficient LLM generation

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuan- dong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. MagicPIG: LSH sampling for efficient LLM generation. InInternational Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=mZn2Xyh9Ec

    work page 2024

Show all 46 references
  1. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems, 2022. URLhttps://openreview.net/forum?id=H4DqfPSibmx

    2022

  2. [10]

    DeepSeek-V3.2: Efficient reasoning & agentic AI

    DeepSeek-AI. DeepSeek-V3.2: Efficient reasoning & agentic AI. Hugging Face model card, 2025. URL https://huggingface.co/deepseek-ai/DeepSeek-V3.2 . Accessed 2026-05-04

    2025

  3. [11]

    DeepSeek-V3.2: Pushing the frontier of open large language models, 2025

    DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models, 2025. URL https://arxiv.org/abs/2512.02556

    2025 arXiv

  4. [12]

    DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URLhttps://arxiv.org/abs/2606.19348

    2026

  5. [13]

    Cost-efficient large language model serving for multi-turn conversations with CachedAttention

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with CachedAttention. InUSENIX Annual Technical Conference (ATC), 2024. URLhttps://www...

    2024

  6. [14]

    GLM-5: from vibe coding to agentic engineering, 2026

    GLM-5 Team. GLM-5: from vibe coding to agentic engineering, 2026. URL https://arxiv. org/abs/2602.15763

    2026 arXiv

  7. [15]

    FastDecode: High-throughput GPU-efficient LLM serving using heterogeneous pipelines, 2024

    Jiaao He and Jidong Zhai. FastDecode: High-throughput GPU-efficient LLM serving using heterogeneous pipelines, 2024. URLhttps://arxiv.org/abs/2403.11421

    2024 arXiv

  8. [16]

    NEO: Saving GPU memory crisis with CPU offloading for online LLM inference

    Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. NEO: Saving GPU memory crisis with CPU offloading for online LLM inference. InProceedings of Machine Learning and Systems (MLSys), 2025. 17

    2025

  9. [17]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020

    2020

  10. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Princip...

    2023

  11. [19]

    InfiniGen: Efficient generative inference of large language models with dynamic KV cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024

  12. [20]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InAdvances in Neural Information Processing Systems, 2024

    2024

  13. [21]

    ECHO: Efficient KV cache offloading with lossless prefetching for serving native sparse attention LLMs

    Guangda Liu, Wenhao Chen, Chengwei Li, Zhenyu Ning, Jing Lin, Yiwu Yao, Quan Chen, Shixuan Sun, Jieru Zhao, and Minyi Guo. ECHO: Efficient KV cache offloading with lossless prefetching for serving native sparse attention LLMs. In20th USENIX Symposium on Operating Systems Desig...

    2026

  14. [22]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning, pages 32332–32344, 2024

    2024

  15. [23]

    NVIDIA GH200 Grace Hopper superchip

    NVIDIA. NVIDIA GH200 Grace Hopper superchip. Product page, 2026. URL https: //www.nvidia.com/en-us/data-center/grace-hopper-superchip/ . Accessed 2026- 05-04

    2026

  16. [24]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA),

  17. [25]

    Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In23rd USENIX Conference on File and Storage Technologie...

    2025

  18. [26]

    Qwen3-30B-A3B-Thinking-2507

    Qwen Team. Qwen3-30B-A3B-Thinking-2507. Hugging Face model card, 2025. URL https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507 . Accessed 2026-06- 18

    2025

  19. [27]

    Bench serving guide

    SGLang Project. Bench serving guide. SGLang Documentation, 2026. URL https://docs. sglang.io/docs/developer_guide/bench_serving. Accessed 2026-06-18

    2026

  20. [28]

    HiSparse: Hierarchical sparse attention

    SGLang Project. HiSparse: Hierarchical sparse attention. SGLang Documentation,

  21. [29]

    FlexGen: High-throughput generative inference of large language models with a single GPU

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. InInternational Conference on Ma- chine Learning, pag...

    2023

  22. [30]

    ShadowKV: KV cache in shadows for high-throughput long-context LLM inference

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. InInternational Conference on Machine Learning, pages 57355–57373, 2025. 18

    2025

  23. [31]

    Quest: Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context LLM inference. InInternational Conference on Machine Learning, pages 47901–47911, 2024. URL https://proceedings.mlr.press/ v235/tang24l.html

    2024

  24. [32]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020. URLhttps://arxiv.org/abs/2006.04768

    2020 arXiv

  25. [33]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations, 2024

    2024

  26. [34]

    SGLang HiCache: Fast hierarchical KV caching with your fa- vorite storage backends

    Zhiqiang Xie. SGLang HiCache: Fast hierarchical KV caching with your fa- vorite storage backends. LMSYS Blog, 2025. URL https://lmsys.org/blog/ 2025-09-10-sglang-hicache/. Accessed 2026-05-04

    2025

  27. [35]

    HiSparse: Turbocharging sparse attention with hierarchical memory

    Zhiqiang Xie, Zhangheng Huang, and Tingwei Huang. HiSparse: Turbocharging sparse attention with hierarchical memory. LMSYS Blog, April 2026. URL https://www.lmsys.org/blog/ 2026-04-10-sglang-hisparse/. Accessed 2026-05-04

    2026

  28. [36]

    Strata: Hierarchical context caching for long context language model serving

    Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis. Strata: Hierarchical context caching for long context language model serving. In20th USENIX Symposium on Operating Systems Design and Implementation (OSD...

    2026 arXiv

  29. [37]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, et al. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    2025 arXiv

  30. [38]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attentio...

    2025 doi

  31. [39]

    GLM-5.2: Built for long-horizon tasks

    Z.ai. GLM-5.2: Built for long-horizon tasks. Hugging Face Blog, June 2026. URL https: //huggingface.co/blog/zai-org/glm-52-blog. Accessed 2026-06-18

    2026

  32. [40]

    PQCache: Product quantization-based KVCache for long context LLM inference.Proceedings of the ACM on Management of Data, 3(3):201:1–201:30, 2025

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. PQCache: Product quantization-based KVCache for long context LLM inference.Proceedings of the ACM on Management of Data, 3(3):201:1–201:30, 2025. doi: 10.1145/3725338

    2025 doi

  33. [41]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InAdvances in Neural I...

    2023

  34. [42]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Informatio...

    2024

  35. [43]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 20...

    2024

  36. [2020]

    URLhttps://arxiv.org/abs/2004.05150

    2004 arXiv

  37. [2024]

    doi: 10.1109/ISCA59077.2024.00019

    2024

  38. [2026]

    Accessed 2026-05-04

    URL https://docs.sglang.io/docs/advanced_features/hisparse_guide. Accessed 2026-05-04

    2026

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.