Pith. sign in

REVIEW 5 major objections 7 minor 56 references

HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference

T0 review · 5 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HACK lets LLM attention run directly on 2-bit compressed keys and values, skipping dequantization and cutting job completion time by up to 70.9%.

desk verdict Solid systems paper with a plausible core result, but the 'no dequantization' sell is strongest for long sequences; deserves a serious referee. read the letter →

arxiv 2502.03589 v1 pith:H4TLQIRO submitted 2025-02-05 cs.DC cs.LG

Zeyu Zhang , Haiying Shen , Shay Vargaftik , Ran Ben Basat , Michael Mitzenmacher , Minlan Yu This is my paper · ORCID
classification cs.DCcs.LG
keywords disaggregatedLLMinferenceKVcachequantizationhomomorphicdequantizationoverheadattentionmatrixmultiplicationINT8executionjobcompletiontimelong-context
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in disaggregated LLM inference, the main hidden cost of quantizing the key-value cache—dequantizing the whole cache back to FP16 at every decode step—can be eliminated. HACK (Homomorphic Acceleration via Compression of the KV cache) performs both attention matrix multiplications directly on quantized data, using a cheap affine correction formula to approximate the true product, so keys and values stay compressed through prefill, network transfer, storage, and decode. If the claim holds, one mechanism attacks four bottlenecks at once: KV transmission over slow prefill networks, compute time in both stages, GPU memory pressure on decode instances, and memory-access latency for cached KV data. On Llama-3.1 70B with long-context workloads, HACK reports job-completion-time reductions of up to 70.9% versus a disaggregated baseline and up to 52.3% versus CacheGen and KVQuant, with slightly higher accuracy than those two methods.

What carries the argument

The load-bearing object is the affine homomorphic identity in Eq. (4): with partition-wise asymmetric quantization $a\approx s_a a' + m_a$ and $b\approx s_b b' + m_b$, the product expands into a quantized INT8 GEMM plus three correction terms built from row and column sums. The identity works because dequantization is linear, so the product of affine reconstructions decomposes into a small-integer product and cheap aggregates. Two design choices make the identity practical per decode iteration: the sums $\sum_z b'_{zj}$ are precomputed once when each partition fills and stored in $b+\lceil\log_2\Pi\rceil$ bits (about 2.2-2.7% of GPU memory), and the last partial block of $V$ is kept in FP16 (0.24-0.51% of GPU memory) so that appending a token never triggers a range update that would require requantizing all previous values. These devices convert an algorithm that would otherwise recompute aggregates or requantize the frontier of the cache into an incremental scheme whose overhead does not grow with the cached sequence length.

What would settle it

Run the HACK decode kernel on an A100 with long sequences, timing the 2-bit-to-INT8 expansion and correction-term path separately from the INT8 matrix multiplication, and compare that total with the time for a standard FP16 dequantize-plus-attention kernel on the same data; if the HACK path is not faster at sequence lengths of a few hundred tokens or more, the claimed dequantization elimination is not the source of the speedup.

Watch

Extended reading notes

Core claim

HACK's central discovery is that asymmetric quantization has an affine structure that survives matrix multiplication: if each element is quantized as $x \approx s x' + m$ partition-wise, then each element $c_{ij}$ of a product $C=AB$ is approximately $s_a s_b \sum_z a'_{iz} b'_{zj} + m_b s_a \sum_z a'_{iz} + m_a s_b \sum_z b'_{zj} + Z m_a m_b$. The first term is a fast INT8 product of the quantized matrices; the three remaining terms are row/column corrections that cost far less than reconstructing both full FP16 matrices. HACK applies this identity to the two matmuls in self-attention, $QK^\top$ and $PV$, storing keys and values as 2-bit integers together with per-partition minima, scales, and precomputed row sums. With the sums stored once per partition, the per-iteration correction cost is about $10(d_h + L_{KV})$ operations instead of the $4d_h L_{KV}$ operations needed to dequantize the cache, so the benefit grows with sequence length. In trace-driven evaluations on models from 7B to 180B parameters, HACK reports JCT reductions of up to 70.9% over the disaggregated baseline and up to 52.3% over prior KV-quantization methods.

Load-bearing premise

The method's speed claim rests on the assumption that expanding the stored 2-bit values to 8-bit integers inside the kernel and adding the correction terms costs much less than dequantizing the entire KV cache to 16-bit floats on every decode step; if that conversion is slow on a particular GPU, the headline JCT gains shrink.

Editorial extensions

If this is right

  • The decode loop no longer reconstructs an FP16 KV cache each step: keys and values stay 2-bit in the cache, the kernel expands them to INT8 locally, and the correction terms run on cached sums, so dequantization time falls from up to 37.9% of JCT to roughly 1.5-3.2%.
  • Long-prompt workloads gain the most: HACK improves average JCT by 36.8-45.1% over CacheGen/KVQuant and 55.3-61.6% over the baseline on arXiv and Cocktail, versus about 19-26% and 38-40% on short IMDb and HumanEval prompts, because the last-block-of-V FP16 fraction shrinks as sequences grow.
  • Cheap prefill GPUs with 10-50 Gbps networking become practical: in HACK, KV transmission drops to about 1.3-5.4% of JCT, and peak decode-GPU memory falls by 25.0-33.6% on long sequences.
  • Accuracy stays on par with or ahead of existing compressors: with the default partition size $\Pi=64$, HACK reports 0.16-0.78% higher accuracy than CacheGen and KVQuant, while finer partitions raise accuracy at the cost of up to 28% more JCT, giving an explicit accuracy-latency knob.
  • HACK composes with token-eviction methods, which remove tokens rather than lower precision; the paper states the two approaches are complementary and can be combined to shrink the KV footprint along both dimensions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same affine expansion applies to any GEMM whose operands are stored quantized and reused, so the technique is not inherently tied to attention or to disaggregated serving; HACK's compute-side benefit should also appear in monolithic decode servers.
  • The paper's cost comparison implies a hardware-dependent boundary: on GPUs without fast INT8 tensor cores, the expansion and correction terms can erode the advantage, so a kernel-level breakdown of the 2-bit-to-INT8 conversion time would show exactly where HACK stops being cheaper than FP16 dequantization.
  • A natural next step is to combine HACK with KV eviction: eviction shortens the token dimension while HACK lowers the precision of what remains, and the stored per-partition sums add only a few bits of metadata, so the two mechanisms attack orthogonal dimensions of the KV footprint.
  • Since the correction overhead scales like $O(d_h + L_{KV})$ per decode step rather than $O(d_h L_{KV})$, the relative benefit of HACK should keep growing on longer sequences, and the same identity should remain valid at 4-bit operands once INT4 tensor-core paths are available; the paper lists CUDA/INT4 support as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes HACK, a homomorphic quantization scheme for the KV cache in disaggregated LLM inference. HACK quantizes K, V, and Q to low precision, performs the two attention matrix multiplications directly on quantized values using INT8 arithmetic, and applies a correction formula (Eq. 4) to approximate the FP16 result without full dequantization. For the decode stage, HACK stores sums of quantized values to avoid recomputation and keeps the last block of V in FP16 to avoid requantization. The method is integrated into FlashAttention-2 and vLLM, and evaluated with trace-driven experiments across multiple models, datasets, and GPU instances, reporting JCT reductions up to 70.9% versus the disaggregated baseline and up to 52.3% versus KV quantization methods.

Significance. If the reported speedups hold, HACK would be a practical contribution to disaggregated LLM serving: it simultaneously reduces KV transmission, KV memory access, and dequantization overhead, and the open-sourced implementation supports reproducibility. The work includes extensive sensitivity tests and ablations (summation elimination and last-block requantization elimination) that clarify the design trade-offs. However, the load-bearing 'eliminates dequantization' cost model omits the 2-bit-to-INT8 conversion required by the implementation, and the empirical evaluation lacks variance reporting, so the magnitude of the claimed gains is not yet fully secure; the short-sequence limitations are partly acknowledged but not separated in the headline claims.

major comments (5)
  1. [§5.3 and §6] The cost comparison in §5.3—dequantization at 4·d_h·L_KV FLOPs versus HACK's approximation at 10·(d_h+L_KV) after summation elimination—omits the 2-bit-to-INT8 format conversion described in §6, which is applied to the full K and V cache in every decode iteration. This conversion has the same per-element memory-traffic and instruction pattern that makes naive dequantization expensive, so it is not negligible for the central claim that HACK eliminates heavy dequantization. Please report the measured conversion overhead separately in the JCT decomposition of Fig. 10, or extend the cost model to include packing/unpacking operations.
  2. [§7.2] For the short-sequence datasets IMDb and HumanEval, the decode-time improvement over CacheGen/KVQuant is only 11.5%–14.5%, which the paper attributes to the high proportion of tokens in the FP16 last block of V. The JCT reductions for these datasets (38.6% and 40.1% versus baseline) are nevertheless attributed to 'accelerating prefill and decode stages.' Please provide a component-wise attribution of the JCT reduction (prefill, communication, decode, memory-access) to separate the compute-acceleration effect from the transmission effect, particularly for short sequences; without this, the reader cannot judge how much of the headline gain is due to homomorphic multiplication rather than KV compression.
  3. [§7] All end-to-end results are reported as averages without standard deviations, confidence intervals, or the number of runs. Since the central quantitative claims are percentage reductions (e.g., up to 70.9% in Fig. 12), the lack of variance measures leaves open the possibility that some differences are within run-to-run noise. Please add variance statistics (or at least multiple-run summaries) for the main JCT figures.
  4. [§9] The related-work section cites TurboAttention as a method that also performs attention directly on quantized KV data, but no experimental comparison is provided. Given the paper's claim of improvements over 'state-of-the-art KV quantization methods,' the absence of a decode-time comparison with TurboAttention leaves the relative compute-acceleration advantage unquantified. Please add a comparison or explicitly justify its exclusion on the grounds of differing scope (e.g., no disaggregated communication component).
  5. [§2.2] CacheGen and KVQuant are described as 'strawman methods' implemented by the authors. If these implementations do not use the official, optimized kernels, the reported dequantization overhead (up to 37.9% of JCT) and the resulting JCT comparisons may be overestimated in HACK's favor. Please state whether official code was used and describe the dequantization kernel in enough detail to assess fairness.
minor comments (7)
  1. [§2] There is a typo at the start of Section 2: 'investiage' should be 'investigate'; the figure caption 'Varing GPU' should be 'Varying GPU'.
  2. [§7] In the first paragraph of Section 7, 'KVQaunt' should be 'KVQuant'.
  3. [§8] In Section 8, 'out future work' should be 'our future work'.
  4. [Eq. (3)] Equation (3) is typeset ambiguously: the softmax denominator should be written as the sum over the row indices, and the expression 'exp(x_{i,j})' appears without proper limits. Please reformat.
  5. [§7.3 and §7.5] The default partition size Π=64 is selected after inspecting the accuracy and JCT results for Π=32, 64, and 128 on the same benchmark datasets. This is a form of test-set model selection and may optimistically bias the reported numbers; please discuss the implications or use a validation split.
  6. [Table 6] The column headers in Table 6 (model abbreviations M, P, Y, L, F repeated for each dataset) are hard to parse; consider separating dataset blocks or using full model names.
  7. [Reference [2]] Reference [2] refers to 'The code of HKVQ', but the system in the paper is called HACK; please correct this discrepancy.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Eq. (4) is a self-contained algebraic decomposition and the headline JCT/accuracy results are measured, not derived from the method's own outputs.

full rationale

The load-bearing derivation is Section 5.2's Eq. (4), which expands (AB)_ij after substituting the affine quantization approximations a_iz ~= s_a q_a + m_a and b_zj ~= s_b q_b + m_b. This is an exact algebraic identity for the dequantized (approximate) matrices; it does not define the target output in terms of itself, and no fitted parameter is renamed as a prediction. The subsequent FLOP comparison in Section 5.3 compares dequantization cost 4*d_h*L_KV with the correction overhead 10*(d_h+L_KV) and is an analytic estimate, not a circular re-statement; any concern about omitted INT8 conversion cost is a correctness/performance issue, not circularity. Accuracy and JCT claims are external measurements against baseline, CacheGen, and KVQuant. The citation to THC [34] involves overlapping authors but is only related work and is not load-bearing. The default partition size Pi=64 is selected after inspecting the accuracy/JCT trade-off in Sections 7.3 and 7.5; this is hyperparameter tuning on the evaluation benchmarks, which is a mild overfitting concern but not a circular derivation, since HACK does not predict its own JCT from Pi. Overall, no step reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entity; homomorphic quantization is a method name for an affine-quantized matmul with correction terms. The main tuned quantity is the partition size Pi, with bit widths as additional design choices. The load-bearing assumptions are about the accuracy of per-partition affine reconstruction, the speed of INT8 correction arithmetic, the cheapness of 2-bit to INT8 conversion, and the stability of softmax under approximation error.

free parameters (3)
  • Quantization partition size Pi = 64 (default; 32 and 128 also evaluated)
    Selected after observing accuracy and JCT on the same benchmark datasets; smaller Pi improves accuracy but raises JCT (Table 8). This is a tuned hyperparameter, not derived from first principles.
  • KV quantization bit width = 2 bits
    Chosen to match the roughly 86% compression rate of CacheGen and KVQuant; lower precision also drives the need for small partitions and affects accuracy.
  • Q quantization bit width = 8 bits
    Q is not transmitted and is quantized at 8 bits to preserve accuracy; a design choice rather than a fitted value.
assumptions (4)
  • domain assumption Within each quantization partition, every element is represented well enough by s*q + m, so the affine expansion in Eq. (4) is a faithful approximation of the real matmul.
    Invoked in Section 5.2 when substituting a_iz approx s_a q_a + m_a; accuracy of the approximation depends on data distributions inside partitions, not on algebra.
  • domain assumption INT8 matrix multiplication plus the correction terms in Eq. (4) is faster than FP16 matmul on dequantized KV data on the target GPUs.
    Used throughout Sections 5.3 and 6; V100 lacks INT8 tensor cores, so the benefit there must come from communication and memory savings, and the paper does not profile the correction cost separately.
  • domain assumption The 2-bit to INT8 format conversion in the kernel is cheap enough to be ignored in the cost comparison against dequantization.
    Section 6 says the conversion is done in local GPU memory, but no separate time breakdown is provided; this assumption is load-bearing for the no-dequantization-overhead claim.
  • domain assumption The softmax of the approximated attention score S is close enough to the true softmax for downstream quality on the tested tasks.
    Required for accuracy claims in Section 7.3; validated only with ROUGE-1 and edit similarity, not with open-ended or reasoning benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference." pith.science (2026). https://pith.science/paper/H4TLQIRO

@misc{pith2026250203589,
  author       = {Pith},
  title        = {Pith review of: HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4TLQIRO}},
  note         = {Machine review of arXiv:2502.03589}
}
read the original abstract

Disaggregated Large Language Model (LLM) inference has gained popularity as it separates the computation-intensive prefill stage from the memory-intensive decode stage, avoiding the prefill-decode interference and improving resource utilization. However, transmitting Key-Value (KV) data between the two stages can be a bottleneck, especially for long prompts. Additionally, the computation time overhead for prefill and decode is key for optimizing Job Completion Time (JCT), and KV data size can become prohibitive for long prompts and sequences. Existing KV quantization methods can alleviate the transmission bottleneck and reduce memory requirements, but they introduce significant dequantization overhead, exacerbating the computation time. We propose Homomorphic Acceleration via Compression of the KV cache (HACK) for disaggregated LLM inference. HACK eliminates the heavy KV dequantization step, and directly performs computations on quantized KV data to approximate and reduce the cost of the expensive matrix-multiplication step. Extensive trace-driven experiments show that HACK reduces JCT by up to 70.9% compared to disaggregated LLM inference baseline and by up to 52.3% compared to state-of-the-art KV quantization methods.

Figures

Figures reproduced from arXiv: 2502.03589 by the authors.

Figure 1
Figure 1. Bottlenecks in disaggregated LLM inference. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Employing KV quantization across prefill [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Employing KV quantization across models. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Employing KV quantization across datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Overview of HACK in disaggregated LLM inference. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Illustration of partitioning for quantization. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Illustration of partitioning in self-attention. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: An example of requantization in the last block of 𝑉 . To do this, we could first dequantize old quan￾tized values using the old 𝑚𝑖𝑛𝑗 and 𝑠𝑗 , quantize them and the new value by the updated 𝑚𝑖𝑛𝑗 and 𝑠𝑗 , and then perform the quan￾tized matrix multiplication for the last…
Figure 9
Figure 9. Figure 9: Average JCT across requests for Llama-3.1 [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 11
Figure 11. Figure 11: Average JCT across requests for different [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 10
Figure 10. Figure 10: Average JCT decomposition for Llama-3.1 70B with varying datasets. We also tested different models with the Cocktail using A10G prefill instances [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 13
Figure 13. Figure 13: Average JCT across requests for individual [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Average JCT across requests with varying [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 40 canonical work pages

  1. [1]

    01-ai Model Yi

    2025. 01-ai Model Yi. https://huggingface .co/01-ai/Yi-34B-200K. (2025)

    work page 2025

  2. [2]

    The code of HKVQ

    2025. The code of HKVQ. https://anonymous.4open.science/r/HKVQ. (2025)

    work page 2025

  3. [3]

    Falcon-180B

    2025. Falcon-180B. https://huggingface.co/tiiuae/falcon-180B. (2025)

    work page 2025

  4. [4]

    GPT-4 explaining Self-Attention Mechanism

    2025. GPT-4 explaining Self-Attention Mechanism. https: //www.linkedin.com/pulse/gpt-4-explaining-self-attention- mechanism-fatos-ismali/. (2025)

    work page 2025

  5. [5]

    Meta Llama-3.1

    2025. Meta Llama-3.1. https://llama .meta.com/. (2025)

    work page 2025

  6. [6]

    Microsoft Phi-3

    2025. Microsoft Phi-3. https://huggingface .co/microsoft/Phi-3- medium-128k-instruct. (2025)

    work page 2025

  7. [7]

    Mistral-v0.3

    2025. Mistral-v0.3. https://huggingface .co/mistralai/Mistral-7B- Instruct-v0.3. (2025)

    work page 2025

  8. [8]

    Open Compute Project

    2025. Open Compute Project. https://www .opencompute.org/ documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf. (2025)

    work page 2025

Show all 56 references
  1. [9]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference. In Proceedings of Machine Learning and Sys- tems, P. Gibbons, G. Pekhimen...

    2024

  2. [10]

    Amazon Web Services

    Inc. Amazon Web Services. 2024. Recommended AWS GPU Instances. https://docs .aws.amazon.com/dlami/latest/devguide/ gpu.html. (2024)

    2024

  3. [11]

    Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. 2023. Dynamic Con- text Pruning for Efficient and Interpretable Autoregressive Transformers. In Advances in Neural Information Processing Sys- tems, A. Oh, T. Naumann, A. Glober...

    2023

  4. [12]

    arXiv. 2025. arXiv. https://arxiv .org. (2025). Accessed: 2025-01-30

    2025

  5. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  6. [14]

    Tencent Cloud. 2024. Tencent Cloud - A100 Instances. https:// www.tencentcloud.com/document/product/560/19701#GT4. (2024)

    2024

  7. [15]

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A Discourse- Aware Attention Model for Abstractive Summarization of Long Docu- ments. (2018). arXiv:cs.CL/1804.05685 https://arxiv.org/abs/1804.05685

    2018 arXiv

  8. [16]

    NVIDIA Corporation. 2025. NVIDIA NCCL Documentation. https: //docs.nvidia.com/deeplearning/nccl/index.html. (2025)

    2025

  9. [17]

    Sunhao Dai, Weihao Liu, Yuqi Zhou, Liang Pang, Rongju Ruan, Gang Wang, Zhenhua Dong, Jun Xu, and Ji-Rong Wen. 2024. Cock- tail: A Comprehensive Information Retrieval Benchmark with LLM- Generated Documents Integration. (2024). arXiv:cs.IR/2405.16546 https://arxiv.org/abs/2405.16546

    2024 arXiv

  10. [18]

    Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Par- allelism and Work Partitioning. (2023). arXiv:cs.LG/2307.08691 https://arxiv.org/abs/2307.08691

    2023 arXiv

  11. [19]

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini

  12. [20]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801 (2023)

    2023 arXiv

  13. [21]

    Google. 2024. Gemini 1.5. https://gemini .google.com/app. (2024)

    2024

  14. [22]

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. ZipCache: Accurate and Efficient KV Cache Quantiza- tion with Salient Token Identification. (2024). arXiv:cs.LG/2405.14256 https://arxiv.org/abs/2405.14256

    2024 arXiv

  15. [23]

    Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Richard Charles Hooper, Sehoon Kim, Hiva Moham- madzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In The Thirty-eighth Annual Conference on Neural Inform...

    2024

  16. [24]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. (2024). arXiv:cs.DC/2406.17565 https://arxiv.org/abs/2...

    2024 arXiv

  17. [25]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: 13 arXiv, 2025 Disaggregate LLM Inference for Mixed Downstream Workloads. (2024). arXi...

    2024 arXiv

  18. [26]

    IMDb. 2020. Genre Classification Dataset IMDb. https:// www.kaggle.com/datasets/hijest/genre-classification-dataset-imdb. (2020)

    2020

  19. [27]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse At- tention. In The Thir...

    2024

  20. [28]

    Hao Kang, Srikant Bharadwaj, James Hensman, Tushar Krishna, Victor Ruhle, and Saravan Rajmohan. 2024. TurboAttention: Effi- cient Attention Approximation For High Throughputs LLMs. (2024). arXiv:cs.LG/2412.08585 https://arxiv.org/abs/2412.08585

    2024 arXiv

  21. [29]

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. arXiv preprint arXiv:2403.05527 (2024). arXiv:cs.LG/2403.05527

    2024 arXiv

  22. [30]

    Philip Kiely. 2024. NVIDIA A10 vs A10G for ML model in- ference. https://www .baseten.co/blog/nvidia-a10-vs-a10g-for-ml- model-inference/. (2024)

    2024

  23. [31]

    John R. Klauder. 1983. Stochastic Quantization. In Recent Develop- ments in High-Energy Physics, H. Mitter and C. B. Lang (Eds.). Springer Vienna, Vienna, 251–281

    1983

  24. [32]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  25. [33]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. In- finiGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In 18th USENIX Symposium on Op- erating Systems Design and Implementation (OSDI 24) . USENIX Associ- ation, Sant...

    2024

  26. [34]

    Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. 2024. THC: Accelerating Dis- tributed Deep Learning Using Tensor Homomorphic Compression. In 21st USENIX Symposium on Networked Systems Design and Implemen- tation (NSDI 24) ...

    2024

  27. [35]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  28. [36]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Stream- ing for Fast Large Language...

    2024

  29. [37]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. In Advances in Neural Informatio...

    2023

  30. [38]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning- Free Asymmetric 2bit Quantization for KV Cache. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Lea...

    2024

  31. [39]

    OpenAI. 2021. Introducing Triton: Open-source GPU programming for neural networks. https://openai.com/index/triton/. (2021)

    2021

  32. [40]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) . 118–13...

    2024

  33. [41]

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. Mooncake: A KVCache- centric Disaggregated Architecture for LLM Serving. (2024). arXiv:cs.DC/2407.00079 https://arxiv.org/abs/2407.00079

    2024 arXiv

  34. [42]

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. 2024. DéjàVu: KV-cache Streaming for Fast, Fault- tolerant Generative LLM Serving. In Proceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning R...

    2024

  35. [43]

    Philippe Tillet. 2020. Fused Attention in Triton. https://triton-lang.org/ main/getting-started/tutorials/06-fused-attention.html. (2020)

    2020

  36. [44]

    vLLM Team. 2024. vLLM FP8 for KV Cache. https://docs.vllm.ai/en/ v0.5.4/quantization/fp8_e4m3_kvcache.html. (2024)

    2024

  37. [45]

    Jie Wang, Huanxi Liu, Dawei Feng, Jie Ding, and Bo Ding. 2024. FP4- Quantization: Lossless 4bit Quantization for Large Language Models. In 2024 IEEE International Conference on Joint Cloud Computing (JCC) . 61–67. https://doi.org/10.1109/JCC62314.2024.00017

    2024

  38. [46]

    Gang Wu. 2024. String Similarity Metrics – Edit Distance. https: //www.baeldung.com/cs/string-similarity-edit-distance. (2024)

    2024

  39. [47]

    Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song. 2024. FP6-LLM: Efficiently Serving Large Language Mod- els Through FP6-Centric Algori...

    2024 arXiv

  40. [48]

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. (2024). arXiv:cs.CL/2405.12532 https: //arxiv.org/abs/2405.12532

    2024 arXiv

  41. [49]

    Chaoran Zhang, Lixin Zou, Dan Luo, Min Tang, Xiangyang Luo, Zihao Li, and Chenliang Li. 2024. Efficient Sparse Attention needs Adaptive Token Release. (2024). arXiv:cs.CL/2407.02328 https://arxiv.org/abs/ 2407.02328

    2024 arXiv

  42. [50]

    Lei Zhang, Yunshui Li, Jiaming Li, Xiaobo Xia, Jiaxi Yang, Run Luo, Minzheng Wang, Longze Chen, Junhao Liu, and Min Yang

  43. [51]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lian- min Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Adv...

    2023

  44. [52]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (...

    2024

  45. [53]

    Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs. (2024). arXiv:cs.CL/2406.18294 https://arxiv.org/abs/2406.18294

    2024 arXiv

  46. [210]

    https://www.usenix.org/conference/osdi24/presentation/zhong- yinmin 15

  47. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)

    Efficient Memory Management for Large Language Model Serv- ing with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23) . Association for Computing Machinery, New York, NY, USA, 611–626. https://doi .org/10.1145/ 3600006.3613165

  48. [2024]

    In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.)

    A Simple and Effective 𝐿_2 Norm-Based Strategy for KV Cache Compression. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguis- tics, Miami, ...

    2024 doi

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.