Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TailorKV claims per-layer compression of the KV cache—1-bit quantization for dense-attention layers, dominant-token retrieval for sparse ones—keeps long-context accuracy near the full cache while roughly halving GPU memory.

desk verdict A genuinely useful hybrid KV-cache design whose 'nearly lossless' abstract claim is contradicted by its own RULER tables. read the letter →

arxiv 2505.19586 v2 pith:CKU6PBOF submitted 2025-05-26 cs.CL

Dingyu Yao , Bowen Shen , Zheng Lin , Wei Liu , Jian Luan , Bin Wang , Weiping Wang This is my paper
classification cs.CL
keywords KVcachecompressionlong-contextinference1-bitquantizationCPU-GPUoffloadingsparseattentiondynamictokenretrievallayer-wiseGPUmemoryreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard tricks for shrinking the attention cache during long-context inference—compressing every cached key and value, or selectively loading only the most important tokens—succeed or fail layer by layer, so the two should be combined rather than chosen between. TailorKV's central claim is that shallow layers with dense attention need global information across all tokens but can be aggressively 1-bit quantized without quality loss, while deeper layers with sparse attention hinge on a few dominant tokens and can be served by fetching only those from CPU memory. Across Llama-3.1-8B, Llama-2-7B, Yi-6B, and Yi-9B, the method keeps LongBench within 1.2 points of the full cache (52.6 versus 53.8) and InfiniteBench loss under 1.5 points while cutting peak GPU memory by more than half, and it beats every sparse or quantized baseline on RULER. The headline result is serving Llama-3.1-8B at 128k context on a single 24 GB RTX 3090 at 82 ms per token.

What carries the argument

The load-bearing machinery is the offline layer classifier plus the two-stage approximate attention it enables. The dense preference score $P$ is computed during prefilling by scoring the most recent $n_q$ queries against the full cached keys and summing the top-$k$ attention mass; a high $P$ means attention is spread across tokens (quantization-friendly), a low $P$ means attention is concentrated on few tokens (sparsity-friendly), with threshold $\tau = 0.2$ fixed across models. For sparsity-friendly layers, critical channels are identified from the element-wise product of the estimated query's channel magnitudes and the key cache's channel magnitudes, $s_i = |\hat{q}_i| \cdot \max(|K_i|)$, with the layer-$l$ query estimated as $\hat{q}^{(l)} = W_q^{(l)} h^{(l-1)}$ using the residual similarity of adjacent hidden states; this estimate drives a double-buffered prefetch of the critical key rows from CPU to GPU, and then the Top-K tokens are fetched for attention. The 1-bit quantization kernel with FP16×INT1 GEMV completes the design by carrying the global information in dense-attention layers at 1/16 of the memory per number.

What would settle it

Compute the cosine similarity between hidden states $h^{(l-1)}$ and $h^{(l)}$ on a 1M-token context or on a model with a different residual architecture, and compare the Top-K recall of the approximated attention scores against exact attention. The paper plots similarity only for the tested models and gives no failure threshold, so any regime where the similarity drops below the plotted range predicts a collapse in downstream accuracy.

Watch

Extended reading notes

Core claim

The paper's central claim is that quantization and sparse token retrieval complement each other because their error modes live in different layers. During prefilling, a dense preference score $P$ computed from attention weights classifies each layer as quantization-friendly—dense attention, global information—or sparsity-friendly—concentrated attention on a few dominant tokens. Quantization-friendly layers get static 1-bit quantization of the full KV cache, and sparsity-friendly layers keep a full-precision cache in CPU memory while fetching only the Top-K tokens each decoding step. The token selection itself is channel-driven: approximate attention scores are formed from the element-wise product of query-channel magnitudes and key-channel magnitudes, with the query of layer $l$ estimated from the hidden state of layer $l-1$ so the critical key channels can be prefetched during the previous layer's compute. With only layer 0 quantized to 1-bit and 64+128 tokens fetched per remaining layer in Llama-3.1-8B, LongBench average is 52.6 versus 53.8 for the full cache, InfiniteBench loss is under 1.5 points, and decode latency at 128k is 82 ms per token on a single RTX 3090.

Load-bearing premise

The dynamic retrieval works only if adjacent layers in the network have nearly identical internal representations, so that the query of one layer can be predicted from the layer before it; if that similarity breaks down at extreme context lengths, unusual attention heads, or in model families beyond the Llama and Yi models tested, the framework will fetch the wrong tokens and accuracy will silently drop.

Editorial extensions

If this is right

  • A 128k-context Llama-3.1-8B fits on one 24 GB RTX 3090 at 82 ms per token, with LongBench 52.6 versus 53.8 for the full cache.
  • At a 34.2× compression ratio, TailorKV-1 matches the LongBench average of SimLayerKV at its 1.53× ratio (both 52.6), suggesting the per-layer tailoring, not the raw budget, is what preserves accuracy.
  • The hybrid approach beats pure eviction (StreamingLLM, SnapKV) and pure selection (Quest, PQCache) on LongBench, InfiniteBench, and RULER at the same or smaller token budgets, because each individual strategy's failure mode is covered by the other.
  • Quantizing only the 0th layer yields the best accuracy in ablation, while quantizing sparsity-friendly layers degrades performance, so the layer classification itself is the accuracy-critical choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The inter-layer similarity heuristic could be made self-monitoring: measure the actual similarity between $h^{(l-1)}$ and $h^{(l)}$ during decoding and fall back to computing the true query when it dips, trading a small amount of latency for robustness on unfamiliar models.
  • If the layer dichotomy generalizes, the same dense-preference score could steer prefill acceleration: dense-attention layers should resist sparsified or kernel-skipped prefill, while sparse-attention layers can tolerate it—directly testable with the $P$ score already defined.
  • The paper's finding that query and key outliers appear at dynamic channel positions suggests a quantization refinement the authors did not build: reserve a small full-precision buffer for whichever channels spike at decode time, instead of quantizing all channels uniformly, which could push 1-bit layers beyond the single quantized layer used here.
  • Head-wise classification, named as future work in the limitations, would let the framework quantize the dense heads and sparsify the sparse heads within a single layer, potentially compressing the mixed layers that currently force a whole-layer choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes TailorKV, a hybrid KV cache compression framework for long-context LLM inference. The central observation is that some layers have dense attention and tolerate aggressive quantization well, while other layers have sparse attention and tolerate Top-K token selection well. TailorKV classifies layers offline using a dense-preference score P, statically quantizes the quantization-friendly layers (typically layers 0 or {0,1}) to 1-bit or 2-bit precision, offloads the remaining layers' KV cache to CPU memory, and dynamically retrieves Top-K tokens during decoding using an approximated attention score. The paper reports accuracy on LongBench, InfiniteBench, and RULER for Llama-3.1-8B, Yi-6B, and Yi-9B, together with memory and latency measurements, and claims that the method achieves nearly lossless performance under aggressive compression while outperforming state-of-the-art baselines.

Significance. The paper makes a useful structural suggestion: instead of applying one compression strategy to all layers, one can exploit the complementary strengths of quantization and sparse retrieval at different layers. The system implementation is substantial, the experiments cover three models and three long-context benchmarks, and the ablation studies in Section 5.4 and Appendix F give useful evidence that the layer-specific strategy matters and that dynamic channel selection helps. The release of code is a further strength. However, the headline claim of 'nearly lossless performance under aggressive compression settings' is not supported by the paper's own RULER results, and one of the baseline comparisons is weakened in a way that affects the state-of-the-art claim. The core hybrid idea remains plausible and worth pursuing, but the manuscript needs substantial revision in its claims and in the baseline fairness analysis before it can be considered for publication.

major comments (5)
  1. [Abstract; Section 7; Table 16] The abstract and conclusion describe TailorKV as achieving 'nearly lossless performance under aggressive compression settings,' but this is contradicted by the paper's own RULER results. For Llama-3.1-8B at 128k, Table 16 reports a full-cache average of 77.6, while TailorKV-2 scores 68.5 (a drop of 9.1 points) and TailorKV-1 scores 63.3 (a drop of 14.3 points). At 64k the gaps are 5.2 and 9.9 points, respectively. InfiniteBench also shows a 3.6-point drop for TailorKV-1 (40.4 vs 44.0). Only LongBench is close (52.6 vs 53.8 for TailorKV-1). The 'nearly lossless' claim must be qualified to the specific benchmarks and configurations where it holds, or removed and replaced with a quantitative statement of the actual trade-off.
  2. [Section 5.1, Table 6; Section 5.2] The PQCache baseline is weakened for contexts above 64k by restricting K-Means to one iteration, while LongBench uses 'adaptive' iterations (Table 5). The text in Section 5.2 explicitly says that this restriction 'compromises accuracy and exposes PQCache's limitations.' Since the lengths above 64k are exactly where TailorKV claims its largest advantage, this setup makes the 'outperforming the state-of-the-art' claim difficult to evaluate. Please run PQCache with its intended adaptive iteration setting, or present the accuracy-latency trade-off that justifies the one-iteration restriction, and adjust the comparative claim accordingly.
  3. [Section 4.1; Section 5.1; Table 9] The offline identification procedure is not parameter-free: the threshold tau = 0.2 is tuned on the synthetic LongBench task, the quantization-friendly layer set Q is chosen per model (Table 9), the number of critical channels differs by benchmark (8 for LongBench, 12 for InfiniteBench and RULER), and the token budgets also differ. Because these choices are made with knowledge of the evaluation benchmarks, the claim in Appendix C that the metric P is consistent across datasets is not by itself sufficient. The paper should provide a sensitivity analysis over tau and Q, and should clearly separate benchmark-specific tuning from fixed hyperparameters in the experimental protocol.
  4. [Section 4.2; Appendix B] Dynamic retrieval relies on approximating the query of layer l by q_hat^(l) = W_q^(l) h^(l-1), justified by cosine similarity between adjacent hidden states. Appendix B and Figure 11 show positive similarity, but there is no bound or failure analysis. If this approximation degrades for long contexts, for particular attention heads, or for other model families, the prefetched critical channels and the resulting Top-K selections will be silently wrong. Please report Top-K retrieval recall or end-task accuracy against an oracle that uses the true query, across sequence lengths and heads, and discuss the conditions under which the approximation breaks.
  5. [Abstract; Table 4; Table 13] The abstract states that Llama-3.1-8B with 128k context can be served on a single RTX 3090 at 82 ms per token, but the reported latency tables do not clearly support this. Table 13 gives 0.082 s for the full cache on an A100 at 128k, while Table 4 reports TailorKV on an RTX 3090 only up to 64k (0.074 s). Please clarify which configuration produces the 82 ms/token figure; if it is the full-cache A100 number, the sentence should be corrected to report TailorKV's actual 128k decoding latency.
minor comments (5)
  1. [Section 4.4, Table 2] The memory formulas for 'Ours (Q)' and 'Ours (S)' do not define all symbols (e.g., d_s, l_q) and do not state whether the expressions are per layer or per head; please clarify the notation.
  2. [Section 3, Equation (5)] The sparse error E is defined for a single attention row; it would be helpful to state explicitly whether Figure 1 reports per-head or averaged values, since attention sparsity varies by head.
  3. [Section 5.2, Table 15] The sentence claiming an 'average performance loss under 1.5% compared to the full cache' is only accurate for TailorKV-2 on InfiniteBench; TailorKV-1 loses 3.6 points on that benchmark. Please state which configuration and metric the claim refers to.
  4. [Table 1] The column label 'RB-P' should be expanded to RepoBench-P, and the caption should explain how the 1-bit (L={0}) row relates to the KIVI comparison.
  5. [Figure 6] The caption says TailorKV uses 128+(896) tokens while other methods use 1024 tokens; if the 128 and 896 are local and retrieved tokens, then the total is also 1024, and the caption should say this explicitly to avoid implying an unequal budget.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TailorKV's compression-preference metric is an attention-mass descriptor, and the reported gains are empirical; the RULER shortfall is a correctness concern, not a circular derivation.

full rationale

No circular step meets the bar of quoting the paper and showing that an equation reduces to its own input. Section 4.1 defines P = n_q − sum of top-k attention weights, which is an attention-sparsity summary rather than a quantity fitted to downstream accuracy. The claim that dense layers are quantization-friendly is independently supported by Table 1, which shows that 1-bit quantization of layer 0 is nearly lossless while quantizing other layers degrades accuracy. Appendix B's q̂^(l) = W_q^(l) h^(l−1) is a stated approximation validated by cosine-similarity plots in Figure 11, not imported from a self-citation or a uniqueness theorem. The dynamic retrieval score is a heuristic approximation of attention, and the reported LongBench, InfiniteBench, and RULER numbers are measured outcomes, not quantities derived from that approximation. The disclosed tuning of τ on a synthetic LongBench task and the benchmark-specific critical-channel counts (8 for LongBench, 12 for InfiniteBench/RULER) are statistical-validity caveats about hyperparameter selection, not circular derivations; they do not make the accuracy figures equal to the fitted parameters by construction. The large RULER degradation relative to full cache is a correctness or overclaim issue, which is outside the circularity definition. No load-bearing self-citation, ansatz-smuggling via citation, or renaming of a known result as a prediction is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two empirical observations (attention density varies by layer; outlier channels are dynamic) and one approximation (inter-layer hidden-state similarity). None of these is derived from first principles, and the layer split threshold tau plus per-model Q sets are calibrated on the evaluation benchmarks. No new physical or architectural entities are introduced.

free parameters (5)
  • tau (dense preference threshold) = 0.2
    Used in Eq. 9 to split layers into quantization-friendly versus sparsity-friendly; value chosen by experimentation on LongBench (Section 4.1).
  • quantization-friendly layer set Q = Llama-3.1-8B: {0}; Llama-2-7B, Yi-6B, Yi-9B: {0,1}
    Selected per model from observations and LongBench ablations; defines which layers receive 1-bit or 2-bit quantization instead of Top-K retrieval.
  • number of critical channels = 8 (LongBench), 12 (InfiniteBench and RULER)
    Controls retrieval quality and latency; tuned per benchmark (Section 5.4 and implementation details).
  • token budget (nlocal, ntopk) = 64+128 LongBench; 128+896 InfiniteBench/RULER
    Number of tokens kept local on GPU and fetched from CPU for sparsity-friendly layers.
  • quantization bit size and group size = 1 or 2 bits; group size 64
    TailorKV-1 and TailorKV-2 variants; group size 64 with FP16 zero point and scale.
assumptions (5)
  • standard math Softmax attention and uniform quantization definitions (Eqs. 3, 4).
    Background used without proof; standard definitions.
  • domain assumption Attention scores correlate with outlier channels in query and key, and outlier positions are dynamic (Section 3, Figure 2).
    Load-bearing for the dynamic retrieval algorithm; not derived from first principles.
  • domain assumption Adjacent hidden states are similar enough that q_hat^(l) = W_q^(l) h^(l-1) approximates q^(l) (Appendix B, Eq. 13).
    Needed for prefetching critical channels one layer ahead; validated only by cosine-similarity plots.
  • domain assumption Dense preference score P computed on prefill is stable across datasets (Appendix C, Figure 12).
    Justifies offline layer classification without per-task recalibration.
  • ad hoc to paper Layers with dense attention tolerate aggressive quantization better than sparsity, while sparse layers tolerate Top-K selection better (Section 3, Table 1).
    Empirical design premise of the hybrid framework; not a theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization." pith.science (2026). https://pith.science/paper/CKU6PBOF

@misc{pith2026250519586,
  author       = {Pith},
  title        = {Pith review of: TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKU6PBOF}},
  note         = {Machine review of arXiv:2505.19586}
}
read the original abstract

The Key-Value (KV) cache in generative large language models (LLMs) introduces substantial memory overhead. Existing works mitigate this burden by offloading or compressing the KV cache. However, loading the entire cache incurs significant latency due to PCIe bandwidth bottlenecks in CPU-GPU communication, while aggressive compression causes notable performance degradation. We identify that certain layers in the LLM need to maintain global information and are unsuitable for selective loading. In contrast, other layers primarily focus on a few tokens with dominant activations that potentially incur substantial quantization error. This observation leads to a key insight that loading dominant tokens and quantizing all tokens can complement each other. Building on this insight, we propose a hybrid compression method, TailorKV, which seamlessly integrates quantization and offloading. TailorKV develops an inference framework along with a hardware-friendly implementation that leverages these complementary characteristics. Extensive long-context evaluations exhibit that TailorKV achieves nearly lossless performance under aggressive compression settings, outperforming the state-of-the-art. Particularly, the Llama-3.1-8B with 128k context can be served within a single RTX 3090 GPU, reaching 82 ms per token during decoding.

Figures

Figures reproduced from arXiv: 2505.19586 by the authors.

Figure 1
Figure 1. Observations on attention. (a) Attention weights on Llama-2-7B-32K-Instruct. Detailed visualizations are in Appendix I. (b) Sparse error of different models on the 2WikiMQA dataset, with only the top 5% of attention scores retained. (c) Sparse error on different datasets, with only the top 5% of attention scores retained. Strategy RB-P LCC GovReport TriviaQA 16-bit 56.7 63.4 34.9 91.6 1-bit (KIVI) 24.4 26.2 8.3 18.6… view at source ↗
Figure 2
Figure 2. (Top) Query and key in Llama-3.1-8B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. System overview of TailorKV. Offline identification categorizes the layers into quantization-friendly and sparsity-friendly. For quantization-friendly layers, we employ aggressive static quantization. For sparsity-friendly layers, we dynamically retrieve Top-K tokens. Critical current query and critical key cache represent the outliers in the query and key cache, respectively. pressed by the formula qK⊤ [PITH_FULL_… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Timeline of dynamic retrieval. Blue signifies computation and pink signifies communication. 4.3 Static Quantization Unlike traditional quantization methods (Liu et al., 2024d; Yang et al., 2024a; He et al., 2024), Tai￾lorKV focuses on ensuring that each layer "plays it…
Figure 6
Figure 6. Figure 6: The average accuracy of different methods on [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: Latency breakdown (ms) under different meth [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Ablation studies. (a) Performance comparison with different layers quantized to 1-bit. (b) Performance of TailorKV with dynamic or static channels. (c) Performance comparison with different numbers of critical channels. 8.1× and 2.0× faster than the GQA model. Tai￾lorK…
Figure 11
Figure 11. Figure 11: Cosine similarity between adjacent layers. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 10
Figure 10. Figure 10: Comparison of TailorKV with other methods [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 12
Figure 12. Figure 12: Dense preference score P for layers across different offline datasets. Methods Configurations StreamingLLM num local: 128, num initial: 64 SnapKV window size: 64, max capacity prompt: 128, kernel size: 7, pooling: max pooling Quest page size: 16, token budget: 196 PQC…
Figure 13
Figure 13. Figure 13: Visualization of attention weights across the 2WikiMQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Magnitude of query, key and value for Llama-3.1-8B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  2. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

Reference graph

Works this paper leans on

42 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    01-ai . 2024 a . Yi-6b-200k. https://huggingface.co/01-ai/Yi-6B-200K. Accessed: 2024-07-01

    work page 2024

  2. [2]

    01-ai . 2024 b . Yi-9b-200k. https://huggingface.co/01-ai/Yi-9B-200K. Accessed: 2024-07-01

    work page 2024

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . ArXiv preprint, abs/2303.08774

    arXiv 2023

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proc. of ACL, pages 3119--3137. Association for Computational ...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024. https://arxiv.org/abs/2406.02069 Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling . ArXiv preprint, abs/2406.02069

    arXiv 2024

  6. [6]

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. 2025. https://openreview.net/forum?id=ALzTQUgW8a Magic PIG : LSH sampling for efficient LLM generation . In The Thirteenth International Conference on Learning Representations

    work page 2025

  7. [7]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  8. [8]

    Tri Dao. 2024. https://openreview.net/forum?id=mZn2Xyh9Ec Flashattention-2: Faster attention with better parallelism and work partitioning . In The Twelfth International Conference on Learning Representations

    2024

Show all 42 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . ArXiv preprint, abs/2407.21783

    2024 arXiv

  2. [10]

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2024. https://arxiv.org/abs/2407.11550 Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference . ArXiv preprint, abs/2407.11550

    2024 arXiv

  3. [11]

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. https://openreview.net/forum?id=5t4ZAkPiJs Zipcache: Accurate and efficient KV cache quantization with salient token identification . In The Thirty-eighth Annual Conference on Neural Information P...

    2024

  4. [12]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270--1303

    2024

  5. [13]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. https://openreview.net/forum?id=kIoBbc76Sy RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling

    2024

  6. [14]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. https://openreview.net/forum?id=fPBACAbqSN MI nference 1.0: Accelerating pre-filling for long-context LLM ...

    2024

  7. [15]

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. https://arxiv.org/abs/2403.05527 Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm . ArXiv preprint, abs/2403.05527

    2024 arXiv

  8. [16]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  9. [17]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. Infinigen: Efficient generative inference of large language models with dynamic kv cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155--172

    2024

  10. [18]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. https://openreview.net/forum?id=poE54GOq2l Snap KV : LLM knows what you are looking for before generation . In The Thirty-eighth Annual Confer...

    2024

  11. [19]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

    2024

  12. [20]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  13. [21]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. 2024 b . Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997--140031

    2024

  14. [22]

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. 2024 c . https://arxiv.org/abs/2409.10516 Retrievalattention: Accelerating long-context llm inference via vector retrieval . ArXiv preprint, abs...

    2024 arXiv

  15. [23]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a452a7c6c463e4ae8fbdc614c6e983e6-Abstract-Conference.html Scissorhands: Exploiting the per...

    2023

  16. [24]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 d . Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  17. [25]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. https://www.usenix.org/conference/fast25/presentation/qin Mooncake: Trading more storage for less computation a KVCache-centric architecture for serving LLM c...

    2025

  18. [26]

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2024. https://openreview.net/forum?id=Ue8EHzaFI4 Sparq attention: Bandwidth-efficient LLM inference . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  19. [27]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \' e , Ion Stoica, and Ce Zhang. 2023. https://proceedings.mlr.press/v202/sheng23a.html Flexgen: High-throughput generative inference of large language models with a singl...

    2023

  20. [28]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: query-aware sparsity for efficient long-context llm inference. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  21. [29]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  22. [30]

    Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. 2025. https://openreview.net/forum?id=HzBfoUdjHt \ text\ D\ \_\ 2\ text\ O\ \ : Dynamic discriminative operations for efficient long-context infere...

    2025

  23. [31]

    Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. 2019. https://arxiv.org/abs/1909.01315 Deep graph library: A graph-centric, highly-performant package for graph neural networks . ArXiv preprint, abs/1909.01315

    2019 arXiv

  24. [32]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024 a . https://openreview.net/forum?id=bTHFrqhASY Inf LLM : Training-free long-context extrapolation for LLM s with an efficient context memory . In The Thirty-eight...

    2024

  25. [33]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024 b . https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  26. [34]

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024 a . https://arxiv.org/abs/2402.18096 No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization . ArXiv preprint, ...

    2024 arXiv

  27. [35]

    Shuo Yang, Ying Sheng, Joseph E Gonzalez, Ion Stoica, and Lianmin Zheng. 2024 b . https://arxiv.org/abs/2408.07092 Post-training sparse attention with double sparsity . ArXiv preprint, abs/2408.07092

    2024 arXiv

  28. [36]

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2024 a . https://arxiv.org/abs/2407.12820 Pqcache: Product quantization-based kvcache for long context llm inference . ArXiv preprint, abs/2407.12820

    2024 arXiv

  29. [37]

    Libo Zhang, Zhaoning Zhang, Baizhou Xu, Songzhu Mei, and Dongsheng Li. 2024 b . Dovetail: A cpu/gpu heterogeneous speculative decoding for llm inference. arXiv preprint arXiv:2412.18934

    2024 arXiv

  30. [38]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.814 B ench: Extending long context evaluation beyond 100 K tokens . In Proceedings of the 6...

    2024 doi

  31. [39]

    Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, and Min Lin. 2024 d . https://arxiv.org/abs/2410.13846 Simlayerkv: A simple framework for layer-level kv cache reduction . ArXiv preprint, abs/2410.13846

    2024 arXiv

  32. [40]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-C...

    2023

  33. [41]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.