Pith. sign in

REVIEW 5 major objections 6 minor 55 references

HiKV claims that pruning the KV cache at two granularities—dropping unimportant tokens and then unimportant vector elements within kept tokens—cuts external memory traffic 7.17x on average with under 1% accuracy loss, using a single reconfi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A two-stage token-then-element KV cache pruning scheme with a reconfigurable sorting accelerator is reported to reduce decoding memory traffic by ~7x and energy by up to 90% while keeping accuracy within ~1%.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Real algorithm-hardware co-design with a credible two-stage KV cache compression story, but the accuracy guarantee is a per-task calibration product and the element-level drop lacks an error bound; worth refereeing, but trust the numbers only after more evidence. the 5 major comments →

arxiv 2607.22389 v1 pith:4JM2K66O submitted 2026-07-24 cs.AR cs.AIcs.LG

HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

Chao Fang , Jun Yin , Man Shi , Marian Verhelst This is my paper
classification cs.AR cs.AIcs.LG
keywords KV cache compressionLLM decodingalgorithm-hardware co-designimportance-aware attentionreconfigurable sortermemory bandwidthlong-context LLMssparse attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that KV-cache redundancy in LLM decoding exists at two levels at once: many tokens are unimportant, and within each retained token many vector elements are unimportant. It argues that exploiting both granularities in sequence—first evicting tokens, then loading only the high-magnitude elements of the query and attention vectors—reaches compression ratios that no single-granularity method can. To make this practical, it proposes a hardware accelerator whose one reconfigurable sorter handles both token-level heap maintenance and element-level chunked sorting, so the memory savings are not offset by sorting overhead. If correct, this would make long-context LLM serving faster and far less energy-hungry for only about 8% added silicon area.

Core claim

On the paper's own terms, the discovery is that token-level and element-level importance are orthogonal and multiplicative: Stage I keeps only important tokens in a fixed dual-bank budget using a localized attention-weight accumulation and a min-heap, while Stage II selects the top-B elements of the query vector to guide K-cache loading and the top-B elements of each chunk of the softmax vector to guide V-cache loading. The paper reports that this preserves task scores within 1% of the uncompressed baseline at total compression ratios of 2x to 16x, reduces external memory access to 13.94% of baseline on average (a 7.17x reduction), speeds attention computation up to 7.95x, cuts energy by 80-

What carries the argument

Reconfigurable importance sorter (RIS): a circuit built from 2-input bitonic comparators that reconfigures into four modes—heapify-up and heapify-down for Stage I token eviction, global sort for the query vector, and chunked parallel sort for the softmax vector in Stage II. It lets one small block (1.51% of accelerator area) serve both granularities, keeping the sorting overhead to about 8% of the critical path while the memory traffic falls by a factor of about seven.

Load-bearing premise

The load-bearing premise is that the dropped low-magnitude elements of Q and P contribute so little to the attention output that they can be discarded without correction; the paper supports this with empirical curves rather than an error bound.

What would settle it

Instrument a decoding run to track, for every head and layer, the total attention weight carried by elements that Stage II drops (the bottom magnitudes of Q and of each P chunk). If any head reliably concentrates more than a few percent of its logit or output mass in dropped positions, or if chunked-sort recall against a global sort falls below about 90% on a few hundred sequences, the paper's 1%-accuracy and 7x-traffic claims would not hold together.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At fixed hardware, doubling both compression ratios multiplies the reduction in KV-cache traffic, so 4x token compression plus 2x element compression yields 8x lower memory traffic.
  • Because DRAM energy dominates decoding, cutting traffic by about 7x translates almost directly into 80-90% energy reduction, not a smaller constant.
  • The dual-bank design keeps importance maintenance at O(log B) per step instead of re-sorting all tokens, so dynamic pruning becomes affordable at long context lengths.
  • The claimed 1% accuracy tolerance is per task, and HiKV is configurable at runtime, so it can adapt to tasks that compress easily (16x) or only mildly (2x).
  • The two-stage scheme is orthogonal to adaptive per-task budget allocation, so those methods can be applied on top without conflict.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A fair stress test would measure, per attention head, what fraction of dot-product mass sits in the dropped low-magnitude elements for a much larger and more diverse prompt set than the 8-sample recall study; the 1% accuracy claim rests on that fraction staying small.
  • The V-cache side loads token rows scattered across DRAM chunks, and the paper itself reports a memory penalty from such fine-grained access; a layout that co-locates frequently selected elements across tokens could push the speedup closer to the 7.17x traffic reduction.
  • The hierarchical principle could be pushed further—e.g., quantizing or compressing the retained elements, or applying element selection during prefill—so the claimed mechanism is a template rather than an endpoint.
  • A formal error bound for chunked top-k selection under softmax would convert the empirical recall plot into a guarantee; absent that, the method's accuracy is a strong empirical regularity, not a proven invariant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes HiKV, an algorithm-hardware co-design for LLM decoding that reduces KV-cache memory traffic at two granularities: Stage I evicts low-importance tokens within a fixed budget, and Stage II selectively loads only the largest-magnitude elements of Q (for K) and of each P chunk (for V). A reconfigurable importance sorter (RIS) unifies heap-based token maintenance and chunk-based element selection in one circuit. The authors report up to 7.95x speedup and 90% energy reduction over a vanilla KV-cache baseline, a 7.17x external-memory-access reduction, and an additional 1.82-4.87x reduction versus token-level state-of-the-art methods, all within a claimed 1% accuracy loss, with only about 8% area overhead. The evaluation uses four LLMs on a selected subset of ten LongBench tasks, with rI and rII calibrated offline per task to meet the accuracy constraint.

Significance. If the accuracy and traffic-reduction claims hold, HiKV is a meaningful advance: it is, to my knowledge, one of the first KV-cache designs to combine token-level and element-level importance pruning in a unified hardware accelerator, and the detailed post-layout implementation in TSMC 16nm with DRAMSim3-based system simulation is a concrete strength. The authors also implement S-LLM, H2O, and Token-Picker on the same hardware platform, which improves comparability. The hardware description of the RIS is thorough, and the reported overheads are plausible. The main risk is not the hardware but the algorithmic guarantee: the near-losslessness of Stage II element dropping is supported only by an 8-sample recall plot and per-task accuracy curves, not by an error bound. Because the headline speedup and energy numbers are measured at per-task calibrated operating points, the claim of 'within 1% accuracy loss' is currently an empirical statement about a small, undisclosed benchmark subset rather than a robust, general property.

major comments (5)
  1. [Sec. III-C, Algorithm 1 lines 3 and 13-15, Fig. 7] The central claim of a 7.17x traffic reduction within 1% accuracy loss rests on the assumption that selecting top-B_II entries by magnitude of Q (for K) and of each P chunk (for V) is near-lossless for the softmax output. The support provided is recall against a global sort on 8 samples (Fig. 7) plus accuracy curves. Recall against a global sort does not bound the error in the actual attention output: a few dropped dimensions can carry substantial dot-product mass, and the chunked selection by construction caps how many elements per P chunk are kept, so high recall can coexist with concentrated discarded mass. The paper itself notes non-monotonic accuracy in some configurations (Sec. V-B), which further shows the approximation is not governed by a provable monotone or bounded error mechanism. Please provide either an error bound on the softmax/output perturbation or a substantially broad
  2. [Sec. V-A and Sec. V-B] The 1% accuracy constraint is defined inconsistently. Sec. V-A says a method must keep the "average score over selected tasks within 1%" of the vanilla baseline; Sec. V-B says "This 1% threshold is applied per task, and the attainable ratio thus varies with each task's compression sensitivity." These definitions lead to different operating points: per-task calibration allows each task to use a different rI/rII, while an average-score constraint can permit individual tasks to degrade by more than 1% as long as the mean stays within 1%. The paper should state which definition is used for the headline numbers in Figs. 17-19 and Table II, and report per-task accuracy and per-task ratios so the reader can verify the 1% claim.
  3. [Sec. V-A and Sec. V-D] The evaluation uses an unspecified "selected subset of ten tasks from LongBench," and no artifacts are released. Since rI and rII are calibrated offline per task, the reported 7.17x traffic reduction and 5.70x speedup are operating-point measurements on an undisclosed benchmark selection, not a generalizable result. Please list the ten tasks, state the selection criterion, and either release the calibration code and per-task results or provide the full accuracy table for all LongBench tasks. Without this, the reproducibility and external validity of the headline numbers cannot be assessed.
  4. [Sec. V-D, Figs. 17-18] The comparison to S-LLM, H2O, and Token-Picker may not be apples-to-apples. The paper says each baseline is configured at its most aggressive compression that keeps the average score within 1%, while HiKV uses per-task calibrated ratios. If HiKV is allowed per-task rI/rII while baselines use a single global setting, the geomean reductions are not directly comparable under an iso-accuracy constraint. Please report results under a common protocol: either per-task calibration for all methods, or a fixed global compression ratio for all methods, and show the sensitivity of the headline numbers to the calibration protocol.
  5. [Sec. IV-E, Fig. 14] The V-cache chunk-indexed layout is acknowledged to incur DRAM row misses when selected token rows within a chunk span different rows. This overhead is modeled in DRAMSim3, but the paper does not quantify how much of the theoretical 7.17x traffic reduction is lost to row-activation and partial-burst penalties. Since the fine-grained element-level access is the main source of the memory penalty noted in Sec. V-D, please report the achieved DRAM efficiency (useful bytes transferred versus total DRAM activation bytes) for the Stage II V-side access pattern.
minor comments (6)
  1. [Author affiliations] The affiliation line contains a typo: "UC Berkley" should be "UC Berkeley."
  2. [Sec. V-A] The phrase "systemic energy analysis" should be "system-level energy analysis."
  3. [Algorithm 1, line 4] The comment "where each element in elements Q is a head dimension" is awkward; consider rewording to "where each selected element corresponds to one head-dimension index."
  4. [Fig. 15] The legend entries "Ours (rI-1x)" through "Ours (rI-4x)" are hard to parse because the x-axis is total compression ratio rtotal. Please clarify in the caption how rII varies and how the different curves relate to the marker shapes.
  5. [Table II] The area/power values for prior works are scaled to 16nm using DeepScaleTool with no stated uncertainty. Please add a sentence on the scaling assumptions and on whether the scaled power includes SRAM/DRAM energy, to avoid over-precise cross-node comparisons.
  6. [Sec. IV-E] PagedAttention manages KV cache in fixed-size pages; the claim that HiKV's access pattern is "naturally compatible with block-wise memory management schemes such as PagedAttention" should be phrased more carefully, since the element-first and chunk-indexed layouts differ from the page layout in PagedAttention.

Circularity Check

1 steps flagged

One fitted-parameter circularity: the 1%-accuracy claim is enforced by per-task ratio calibration, not derived; the core speedup/memory figures remain measured operating points.

specific steps
  1. fitted input called prediction [Sec. V-A (Experimental Setup), Sec. V-B (Algorithm-Level Evaluation), Sec. V-D (System-Level Evaluation)]
    "We define a performance degradation of less than 1% compared to this vanilla baseline as the requirement for maintaining algorithmic integrity. For all evaluated methods, each is configured at the most aggressive compression setting that keeps the average score over selected tasks within 1% of the vanilla baseline. In our evaluation, rI and rII are determined by an offline per-task calibration..."

    The headline 'within negligible 1% accuracy loss' is not an independent result of the hierarchical importance mechanism: the compression ratios rI and rII are selected per task so that the average score stays within 1% of the vanilla baseline on the same selected LongBench tasks. Therefore the 1% condition holds by construction and cannot fail by design. The downstream external-memory-reduction (7.17x) and speedup (7.95x) numbers are then measured at these calibration-selected operating points, making them fitted operating points rather than predictions that validate the near-losslessness of Stage II.

full rationale

HiKV's central derivation is not circular in the strong sense: the two-stage compression is an algorithmic proposal whose claimed benefits (external memory traffic reduction, speedup, energy) are measured with a synthesized 16nm implementation and DRAM simulation, not derived from the accuracy target. Stage I's IS accumulation is a defined maintenance rule, Stage II's top-magnitude selection is an empirical approximation, and the combined rI x rII traffic reduction follows from the element/token layout and the two selection dimensions, not from the 1% constraint. No load-bearing self-citation or imported uniqueness theorem is present; the authors' own prior works cited for HBM2 energy parameters and energy modeling are cited alongside independent sources and are not load-bearing. The one genuine circularity is the framing of the 1% accuracy guarantee: the paper declares 1% loss as the requirement and then calibrates rI/rII per task to meet it on the same evaluation subset, so the 'within 1%' claim is a tautology of the calibration protocol. This makes the iso-accuracy comparisons fair as measured operating points, but they are not independent predictions. The lack of an error bound for dropped K/V elements and the reliance on an 8-sample recall-vs-global-sort plot are correctness risks, not circularity, because recall against a global sort only measures index overlap, not preservation of the attention output.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are introduced. The RIS, TISG, and scoreboard are engineering components whose behavior is specified by circuit descriptions and post-layout numbers, not postulated natural entities.

free parameters (4)
  • r_I (Stage I token compression ratio) = e.g., 3.0x for mistral-7b; geomean 2.86x
    Offline per-task calibration to keep accuracy within 1% of vanilla; directly determines the token budget B_I = L_p/r_I.
  • r_II (Stage II element compression ratio) = e.g., 3.5x for mistral-7b; geomean 2.50x
    Offline calibration; sets B_II = d_h/r_II and determines how many K/V elements are loaded.
  • Recent/important bank split (B_I/2 each) = 50/50
    Chosen by design; no sensitivity analysis is provided for this split, yet it determines how many tokens are kept in the fixed budget.
  • P-vector chunk size (set to d_h) = head dimension (64 or 128)
    The chunk size equals the head dimension; Fig. 7 measures recall at this choice but the final system does not explore or justify other chunk sizes.
axioms (4)
  • domain assumption Attention scores P = softmax(QK^T) reflect token importance
    Used to define importance scores for token eviction; introduced in Sec. III-A and used throughout Stage I.
  • ad hoc to paper Large-magnitude Q and P entries identify K/V elements whose removal keeps accuracy within 1%, with no error correction
    Stage II selection rule; supported only by empirical recall and accuracy curves, not by a derived error bound (Sec. III-C, Algorithm 1).
  • domain assumption Token importance persists after a token is promoted to the important bank (frozen IS)
    Stage I freezes important-bank importance scores to enable O(log B_I) heap maintenance; relies on the persistence-of-importance hypothesis from [21] (Sec. III-B).
  • domain assumption Token-level and element-level redundancy compress multiplicatively
    The claim that the two stages are orthogonal is based on the Fig. 5 visualization of a small set of sequences (8 input sequences, qwen2.5-0.5b), not on a general proof (Sec. III-A).

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding." pith.science (2026). https://pith.science/paper/4JM2K66O

@misc{pith2026260722389,
  author       = {Pith},
  title        = {Pith review of: HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4JM2K66O}},
  note         = {Machine review of arXiv:2607.22389}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.

Figures

Figures reproduced from arXiv: 2607.22389 by Chao Fang, Jun Yin, Man Shi, Marian Verhelst.

Figure 1
Figure 1. Figure 1: The dominated memory access of KV cache becomes the bottleneck of LLM decoding, motivating HiKV to shrink its footprint with high accuracy. growth. First, the cache size grows linearly with sequence length [8]–[10]. As LLMs generate longer outputs or process extended contexts for applications like document analysis and multi-turn conversations, each decoding step must access progressively larger cache volu… view at source ↗
Figure 3
Figure 3. Figure 3: During LLM autoregressive decoding, the KV cache grows linearly with each generation step, continuously amplifying external memory access volume and imposing an escalating memory bandwidth bottleneck. cache optimization approach. Sec. IV elaborates on the HiKV hardware architecture that enables efficient hierarchical impor￾tance maintenance. Sec. V evaluates our approach in terms of algorithm-, hardware-, … view at source ↗
Figure 4
Figure 4. Figure 4: Model memory transfer breakdown with different batch sizes when decoding tokens at long-context scenarios. KV cache increasingly dominates as batch size grows, while activation remains negligible. generating sequences in long-context scenarios under various batch size requests. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: HiKV applies orthogonal compression at two independent granularities at each decoding step. Stage I maintains a dual-bank structure to evict unimportant tokens, reducing the number of tokens loaded. Stage II selects significant elements via Q and P vector sorting, reducing the volume loaded per token. The two stages act on different dimensions of the KV cache and are jointly formalized in Algorithm 1. toke… view at source ↗
Figure 7
Figure 7. Figure 7: Chunk-based sorting recall on Qwen2.5-0.5B-Instruct, evaluated on 8 samples from the 2wikimqa task in LongBench. Each point is the mean over all query heads, layers, and samples. Error bars show the standard deviation. matrix access the dominant memory bottleneck. Hence, we leverage the element magnitudes of Q and P to identify which corresponding positions in K and V matrices are significant, enabling sel… view at source ↗
Figure 8
Figure 8. Figure 8: Operation flow of one decoding step for vanilla, H2O [20], and HiKV. of the elements that a global sort would select. The overlap remains stable across all tested rI values, indicating that the chunking error is independent of the Stage I compression ratio. D. Hierarchical Decoding Integration Having established the individual mechanisms of token￾level and element-level optimization, we elaborate how these… view at source ↗
Figure 9
Figure 9. Figure 9: HiKV accelerator architecture, where the highlighted components, the RIS, scoreboard buffer, and TISG, constitute the HiKV-specific overhead that enables hierarchical two-stage KV cache optimization. units in VU, responsible for GeMV and non-linear operations, respectively. The RIS serves as the core innovation of HiKV design and is elaborated in the following subsections. The on-chip operand memory is org… view at source ↗
Figure 10
Figure 10. Figure 10: (a): the parallel sorter, the partial merger, and the max￾min picker. These modules share the 2-input bitonic sorter as a common basic building block, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Stage I token importance tracking via RIS-enabled heap operations: (a) HEAPIFY_UP constructs the vectorized min-heap node by node; (b) HEAPIFY_DOWN maintains the heap upon token eviction, traversing only a single root-to-leaf path rather than scanning all tokens. associated circuit overhead. Through differential control signal configurations across the modules, the RIS efficiently supports four operationa… view at source ↗
Figure 12
Figure 12. Figure 12: Stage II element importance selection via RIS-enabled chunk-based sorting: GLOBAL_SORT and CHUNK_SORT proceed over multiple rounds of RIS invocations, skipping intervals confirmed outside the top-k range early. vector using the head dimension as the chunk size, extracting the top-BII most important elements from each chunk to drive selective loading from external memory. To this end, the RIS supports two … view at source ↗
Figure 14
Figure 14. Figure 14: Off-chip KV cache layout for one attention head. description of the experimental setup, we first analyze the algorithmic accuracy-compression trade-off. We then quantify the hardware-level area and power efficiency of the implemen￾tation. To demonstrate the system-level benefits, we measure the reductions in external memory traffic and improvements in end-to-end inference performance. Finally, we provide … view at source ↗
Figure 15
Figure 15. Figure 15: Accuracy comparison between our HiKV, S-LLM [14], and H2O [20] under various compression settings for various tasks in LongBench [7]. System Simulation. We use HBM2 as the external memory interface with a 1024-bit bus width, an average access energy of 3.9 pJ/bit, and a 256 GB/s peak bandwidth following [25], [31], [44], [50], [51]. The DRAM access is modeled with DRAMSim3 [52]. We leverage the LLM-Viewer… view at source ↗
Figure 17
Figure 17. Figure 17: External memory access normalized to the vanilla baseline. The HiKV reduction is decomposed into the Stage I (rI ) and Stage II (rII ) factors. mistral-7b llama3-8b longchat-7b qwen2.5-0.5b Geo. Mean 0 2 4 6 8 10 12 Speedup 7.95 × 2.55 × rI rII 2.7 × 3.0 × 5.26 × 1.24 × rI rII 2.0 × 2.6 × 6.20 × 1.55 × rI rII 3.1 × 2.0 × 3.74 × 1.26 × rI rII 2.5 × 1.5 × 5.70 × 1.47 × rI rII 2.6 × 2.2 × Baseline S-LLM H2O … view at source ↗
Figure 18
Figure 18. Figure 18: Inference speedup normalized to the vanilla baseline. For HiKV, the green value is the achieved speedup, split into the Stage I (rI ) and Stage II (rII ) factors, and the red value is the overhead, which comes from the memory penalty ∆mem and the sorting cost ∆I , ∆II . S-LLM is attributed to its sliding window policy. On llama3- 8b model, it fails to satisfy the accuracy constraint at any compression lev… view at source ↗
Figure 19
Figure 19. Figure 19: Energy consumption normalized to the vanilla baseline, split into DRAM, computation, and SRAM. The HiKV reduction is decomposed into the Stage I (rI ) and Stage II (rII ) factors. the RIS hardware to more effectively convert memory savings into inference acceleration. By effectively balancing the trade￾off between importance maintenance and memory bandwidth, HiKV realizes substantial inference speedup com… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 5 linked inside Pith

  1. [1]

    Mistral 7b,

    A. Q. Jianget al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Qwen2 technical report,

    A. Yanget al., “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL CARD.md

    2024

  4. [4]

    How long can context length of open-source llms truly promise?

    D. Liet al., “How long can context length of open-source llms truly promise?” inNeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    2023

  5. [5]

    Language models are few-shot learners,

    Z. Zhanget al., “Language models are few-shot learners,”Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  6. [6]

    Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning,

    K. Wanget al., “Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning,” inThe Twelfth International Con- ference on Learning Representations (ICLR), 2024

    2024

  7. [7]

    Longbench: A bilingual, multitask benchmark for long context understanding,

    Y . Baiet al., “Longbench: A bilingual, multitask benchmark for long context understanding,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), 2024, pp. 3119–3137

    2024

  8. [8]

    P3-llm: An integrated npu-pim accelerator for llm inference using hybrid numerical formats,

    Y . Chenet al., “P3-llm: An integrated npu-pim accelerator for llm inference using hybrid numerical formats,” inProceedings of the 53rd Annual International Symposium on Computer Architecture (ISCA), 2026

    2026

  9. [9]

    A survey on large language model acceleration based on kv cache management,

    L. Haoyanget al., “A survey on large language model acceleration based on kv cache management,”Transactions on Machine Learning Research (TMLR), 2025

    2025

  10. [10]

    Codec: Prefix-shared decoding kernel for llms,

    Z. Wanget al., “Codec: Prefix-shared decoding kernel for llms,” Proceedings of the ACM on Management of Data (SIGMOD), vol. 4, no. 3, pp. 1–27, 2026

    2026

  11. [11]

    Orca: A distributed serving system for transformer- based generative models,

    G.-I. Yuet al., “Orca: A distributed serving system for transformer- based generative models,” inUSENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022, pp. 521–538

    2022

  12. [12]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patelet al., “Splitwise: Efficient generative llm inference using phase splitting,” inACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132

    2024

  13. [13]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache,

    Z. Liuet al., “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” inInternational Conference on Machine Learning (ICML), 2024

    2024

  14. [14]

    Efficient streaming language models with attention sinks,

    G. Xiaoet al., “Efficient streaming language models with attention sinks,” inThe Twelfth International Conference on Learning Represen- tations (ICLR), 2024

    2024

  15. [15]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads,

    G. Xiaoet al., “Duoattention: Efficient long-context llm inference with retrieval and streaming heads,” inThe Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  16. [16]

    Snapkv: Llm knows what you are looking for before gener- ation,

    Y . Liet al., “Snapkv: Llm knows what you are looking for before gener- ation,”Advances in Neural Information Processing Systems (NeurIPS), vol. 37, pp. 22 947–22 970, 2024

    2024

  17. [17]

    Sepllm: Accelerate large language models by com- pressing one segment into one separator,

    G. Chenet al., “Sepllm: Accelerate large language models by com- pressing one segment into one separator,” inForty-second International Conference on Machine Learning (ICML), 2025

    2025

  18. [18]

    Lm-infinite: Zero-shot extreme length generalization for large language models,

    C. Hanet al., “Lm-infinite: Zero-shot extreme length generalization for large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2024, pp. 3991– 4008

    2024

  19. [19]

    Longformer: The long-document transformer,

    I. Beltagyet al., “Longformer: The long-document transformer,”arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  20. [20]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhanget al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, pp. 34 661–34 710, 2023

    2023

  21. [21]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,

    Z. Liuet al., “Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, pp. 52 342– 52 364, 2023

    2023

  22. [22]

    Kvo-llm: Boosting long-context generation throughput for batched llm inference,

    Z. Liet al., “Kvo-llm: Boosting long-context generation throughput for batched llm inference,” inACM/IEEE Design Automation Conference (DAC), 2025, pp. 1–7

    2025

  23. [23]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,

    M. Adnanet al., “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,”Proceedings of Machine Learning and Systems (MLSys), vol. 6, pp. 114–127, 2024

    2024

  24. [24]

    Alisa: Accelerating large language model inference via sparsity-aware kv caching,

    Y . Zhaoet al., “Alisa: Accelerating large language model inference via sparsity-aware kv caching,” inACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 1005–1017

    2024

  25. [25]

    Mata: A memory-efficient attention accelerator for llms exploiting look-back kv cache pruning,

    Z. Zhuet al., “Mata: A memory-efficient attention accelerator for llms exploiting look-back kv cache pruning,”IEEE Transactions on Computers (TC), 2025

    2025

  26. [26]

    Unicaim: A unified cam/cim architecture with static- dynamic kv cache pruning for efficient long-context llm inference,

    W. Xuet al., “Unicaim: A unified cam/cim architecture with static- dynamic kv cache pruning for efficient long-context llm inference,” in ACM/IEEE Design Automation Conference (DAC), 2025, pp. 1–7

    2025

  27. [27]

    Token-picker: Accelerating attention in text generation with minimized memory transfer via probability estimation,

    J. Parket al., “Token-picker: Accelerating attention in text generation with minimized memory transfer via probability estimation,” inPro- ceedings of the 61st ACM/IEEE Design Automation Conference (DAC), 2024. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 14

    2024

  28. [28]

    Dias: Distance-based attention sparsity for ultra-long- sequence transformer with tree-like processing-in-memory architecture,

    Z. Chenet al., “Dias: Distance-based attention sparsity for ultra-long- sequence transformer with tree-like processing-in-memory architecture,” inACM/IEEE Design Automation Conference (DAC), 2025, pp. 1–7

    2025

  29. [29]

    Veda: Efficient llm generation through voting-based kv cache eviction and dataflow-flexible accelerator,

    Z. Wanget al., “Veda: Efficient llm generation through voting-based kv cache eviction and dataflow-flexible accelerator,” inACM/IEEE Design Automation Conference (DAC), 2025, pp. 1–7

    2025

  30. [30]

    Kv-cache oriented query-aware sparse attention accelerator with cross-stage precision-configurable digital cim,

    Y . Zhanget al., “Kv-cache oriented query-aware sparse attention accelerator with cross-stage precision-configurable digital cim,”IEEE Transactions on Circuits and Systems II: Express Briefs (TCAS-II), 2025

    2025

  31. [31]

    End-to-end acceleration of generative models with runtime regularized kv cache management,

    A. Moradifirouzabadiet al., “End-to-end acceleration of generative models with runtime regularized kv cache management,”IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS), 2025

    2025

  32. [32]

    Edgellm: A highly efficient cpu-fpga heterogeneous edge accelerator for large language models,

    M. Huanget al., “Edgellm: A highly efficient cpu-fpga heterogeneous edge accelerator for large language models,”IEEE Transactions on Circuits and Systems I: Regular Papers (TCAS-I), 2025

    2025

  33. [33]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhonget al., “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” inUSENIX Sympo- sium on Operating Systems Design and Implementation (OSDI), 2024, pp. 193–210

    2024

  34. [34]

    Flightllm: Efficient large language model inference with a complete mapping flow on fpgas,

    S. Zenget al., “Flightllm: Efficient large language model inference with a complete mapping flow on fpgas,” inProceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays (FPGA), 2024, pp. 223–234

    2024

  35. [35]

    Ofq-llm: Outlier-flexing quantization for efficient low- bit large language model acceleration,

    G. Wanget al., “Ofq-llm: Outlier-flexing quantization for efficient low- bit large language model acceleration,”IEEE Transactions on Circuits and Systems I: Regular Papers (TCAS-I), 2025

    2025

  36. [36]

    Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches,

    J. Yuanet al., “Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches,” inThe 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  37. [37]

    Apt-llm: Exploiting arbitrary-precision tensor core comput- ing for llm acceleration,

    S. Maet al., “Apt-llm: Exploiting arbitrary-precision tensor core comput- ing for llm acceleration,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), 2025

    2025

  38. [38]

    A survey on efficient inference for large language models,

    Z. Zhouet al., “A survey on efficient inference for large language models,”arXiv preprint arXiv:2404.14294, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    When to stop? towards efficient code generation in llms with excess token prevention,

    L. Guoet al., “When to stop? towards efficient code generation in llms with excess token prevention,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024, pp. 1073–1085

    2024

  40. [40]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhanget al., “Llmcompass: Enabling efficient hardware design for large language model inference,” inACM/IEEE 51st Annual Interna- tional Symposium on Computer Architecture (ISCA), 2024, pp. 1080– 1096

    2024

  41. [41]

    Energy cost modelling for optimizing large language model inference on hardware accelerators,

    R. Geenset al., “Energy cost modelling for optimizing large language model inference on hardware accelerators,” inIEEE 37th International System-on-Chip Conference (SOCC), 2024, pp. 1–6

    2024

  42. [42]

    Llm inference unveiled: Survey and roofline model insights,

    Z. Yuanet al., “Llm inference unveiled: Survey and roofline model insights,”arXiv preprint arXiv:2402.16363, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Skipkv: Selective skipping of kv generation and storage for efficient inference with large reasoning models,

    J. Tianet al., “Skipkv: Selective skipping of kv generation and storage for efficient inference with large reasoning models,” inProceedings of Machine Learning and Systems (MLSys), 2026

    2026

  44. [44]

    Titanus: Enabling kv cache pruning and quantization on-the-fly for llm acceleration,

    P. Chenet al., “Titanus: Enabling kv cache pruning and quantization on-the-fly for llm acceleration,” inProceedings of the Great Lakes Symposium on VLSI (GLSVLSI), 2025, pp. 71–77

    2025

  45. [45]

    Infinigen: Efficient generative inference of large language models with dynamic kv cache management,

    W. Leeet al., “Infinigen: Efficient generative inference of large language models with dynamic kv cache management,” inUSENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024, pp. 155–172

    2024

  46. [46]

    Sparq attention: Bandwidth-efficient llm inference,

    L. Ribaret al., “Sparq attention: Bandwidth-efficient llm inference,” in International Conference on Machine Learning (ICML), 2024

    2024

  47. [47]

    Algorithm 232: Heapsort,

    J. Williams, “Algorithm 232: Heapsort,”Communications of the ACM, vol. 7, no. 6, pp. 347–348, 1964

    1964

  48. [48]

    Sorting networks and their applications,

    K. E. Batcher, “Sorting networks and their applications,” inProceedings of the April 30–May 2, 1968, spring joint computer conference, 1968, pp. 307–314

    1968

  49. [49]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwonet al., “Efficient memory management for large language model serving with PagedAttention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023

    2023

  50. [50]

    Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,

    C. Fanget al., “Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,” inIEEE International Sympo- sium on High Performance Computer Architecture (HPCA), 2025, pp. 1467–1481

    2025

  51. [51]

    Ten lessons from three generations shaped google’s tpuv4i: Industrial product,

    N. P. Jouppiet al., “Ten lessons from three generations shaped google’s tpuv4i: Industrial product,” inACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 2021, pp. 1–14

    2021

  52. [52]

    Dramsim3: A cycle-accurate, thermal-capable dram sim- ulator,

    S. Liet al., “Dramsim3: A cycle-accurate, thermal-capable dram sim- ulator,”IEEE Computer Architecture Letters (CAL), vol. 19, no. 2, pp. 106–109, 2020

    2020

  53. [53]

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference,

    Y . Fenget al., “Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference,” inAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  54. [54]

    Dynamickv: Task-aware adaptive kv cache compression for long context llms,

    X. Zhouet al., “Dynamickv: Task-aware adaptive kv cache compression for long context llms,” inConference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  55. [55]

    DeepScaleTool: A tool for the accurate estimation of technology scaling in the deep-submicron era,

    S. Sarangiet al., “DeepScaleTool: A tool for the accurate estimation of technology scaling in the deep-submicron era,” inIEEE International Symposium on Circuits and Systems (ISCAS), 2021, pp. 1–5

    2021

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.