Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LFPS claims that the Top-k index set for each decoding step can be built from two historical attention patterns, vertical and slash, instead of scanning all keys, giving up to 22.8x speedup over full attention and 9.6x over exact Top-k…

desk verdict The speedup mechanism is plausible, but the paper's own RULER table contradicts the abstract's claim of preserving generation accuracy, showing a 12-point drop. read the letter →

arxiv 2506.15704 v1 pith:OT6KG5FY submitted 2025-05-30 cs.LG cs.AIcs.CL

Feiyu Yao , Qian Wang This is my paper
classification cs.LGcs.AIcs.CL
keywords sparseattentionKVcacheoffloadingdecodingaccelerationlong-contextLLMinferencesinkhistoricalreuseTop-kretrievalCPU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

During decoding, an LLM with an offloaded key-value cache must decide which cached keys are relevant at each step; exact Top-k retrieval requires a dot product against every key and dominates CPU time. LFPS tries to make that step cheap by predicting the Top-k set from the recent past: it maintains two running score tables per attention head, one for fixed absolute positions (vertical patterns) and one for relative positions (slash patterns), then uses these tables plus a local positional expansion to form a small candidate set before the exact dot product. The paper reports that this keeps output close to exact Top-k while cutting indexing work, with up to 22.8x speedup over full attention and 9.6x over exact Top-k retrieval on a single CPU core. A companion bypass sends heads whose attention is almost entirely on the earliest tokens straight to a mean-value output, saving about 12% of head computations. If the method holds, it makes CPU-side sparse decoding a practical route for long-context inference.

What carries the argument

The central object is the pair of running score tables $\Phi_{\mathrm{ver}}(i)$ and $\Phi_{\mathrm{sla}}(i)$, which track, for each key position $i$, how often that absolute position and how often a fixed relative offset have received attention in recent decoding steps. They are updated after every step with a decay factor $r=0.95$ and half-normalized attention weights, and they determine the initial candidate set $C_0^{(t)}$ through a kurtosis-adaptive threshold. Two further mechanisms carry the argument: a positional expansion that adds offsets $\{-1,0,1,2\}$ to each candidate to capture clustered important keys, and a head-sparsity estimator $\rho_h^{(t)}$ that compares sink, global, and local attention estimates and, above threshold $\varepsilon=0.85$, replaces the head output with the mean value vector and freezes that head's score tables.

What would settle it

Measure, for every head of Llama-3.1-8B-Instruct during a RULER run, the correlation between LFPS's $\rho_h^{(t)}$ and the true fraction of attention mass on the sink token computed with full attention; if the correlation is weak, or if forcing all bypassed heads to use exact Top-k attention changes RULER accuracy by more than the paper's reported margins, the sparsity-estimation step is the point of failure.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Top-k index set for a decoding step is largely a function of the vertical and slash attention patterns seen in previous steps, so the exact dot product against all keys can be replaced by an exact dot product over a much smaller candidate set built from those patterns. LFPS embodies this with position-indexed score tables updated by exponential decay, kurtosis-based thresholds that adapt to pattern sharpness, and a positional expansion that captures the observed clustering of important keys. It also claims that attention heads dominated by the sink token can be bypassed: their output is a mean value vector, and updating their score tables is skipped to avoid noise. On Llama-3.1-8B-Instruct over LongBench and RULER, LFPS reports accuracy near the exact Top-k upper bound and substantially above block-based and sampling baselines, with overlap to the true Top-k around 85% on the token-level candidate set.

Load-bearing premise

The load-bearing premise is that the sparsity estimator $\rho_h^{(t)}$ correctly identifies sink-dominated heads from raw unnormalized attention scores under a log-normal assumption; if it misclassifies a head, the bypass freezes that head's score tables and the error compounds through later candidate sets.

Editorial extensions

If this is right

  • At a single CPU core, the paper's measured speedups grow with context length: 4.3x over exact Top-k at 4K, 8.1x at 16K, 14.7x at 64K, and 22.8x over full attention at 128K.
  • The candidate-set fraction shrinks as the context grows, from 6.0% of positions at 4K to 1.7% at 128K for one threshold setting, so indexing cost scales sublinearly in context length.
  • Highly sparse heads are skipped without updating their score tables, cutting head computation by about 12% at $\varepsilon=0.85$ while preserving measured accuracy on LongBench tasks such as Qasper.
  • LFPS beats block-based Quest on RULER averages at a 2% budget and uses only 1.7% of positions at 128K while staying above Quest's accuracy.
  • On eight CPU cores with batch size 8, LFPS reaches 45 tokens/s, about 50% higher than Quest, so the benefit is not limited to single-core decoding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the log-normal assumption used to estimate global attention is the least defended piece of the method; comparing $\rho_h^{(t)}$ with the true fraction of attention mass on the sink token across layers would show how much accuracy depends on that distributional guess.
  • Editorial inference: because LFPS only builds candidates from two one-dimensional patterns, it may miss keys that matter for retrieval-style tasks where relevant tokens sit at arbitrary positions; a test that places a single crucial fact at a position with low vertical and slash scores would probe this blind spot.
  • Editorial inference: the vertical and slash score tables are a lightweight online predictor that could be stacked with low-rank key compression or block-level indexing to shrink the remaining exact dot product, but the paper does not explore such combinations.
  • Editorial inference: the experiments cover one instruction-tuned 8B model, so cross-model transfer to denser or mixture-of-experts architectures is untested; a sweep over model families would show whether the roughly 85% overlap with exact Top-k is a general property of decoder attention.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. LFPS is a sparse-attention indexing acceleration method for LLM decoding when the KV cache is offloaded to host memory. The method maintains two positional score tables—vertical and slash—that aggregate historical attention weights, uses them to select an initial candidate index set, expands the set by neighboring offsets, and then computes exact Top-k inner products only within this set. For heads estimated to have very sparse attention, a bypass returns a mean value vector without updating the tables. Experiments on LongBench and RULER with Llama-3.1-8B-Instruct compare against full attention, exact Top-k, Quest, and MagicPIG, reporting per-token speedups up to 22.8x over full attention and claiming accuracy preservation. The central claim is that historical patterns can predict current Top-k indices cheaply without sacrificing output quality.

Significance. If the accuracy-preservation claim held, LFPS would be a practical and novel contribution to CPU-offloaded sparse inference, since it avoids the full-length dot product that dominates indexing cost. The temporal reuse idea and the positional expansion are plausible and the speedup measurements are substantial. However, the paper's own RULER results show a large accuracy drop relative to full attention and exact Top-k, and the bypass mechanism does not implement the sink-average approximation used to justify it. The method is better described as a speed-accuracy tradeoff than as an accuracy-preserving acceleration. The core idea remains worth exploring, but the current evidence does not support the advertised claim.

major comments (3)
  1. [Abstract and Table 2] The abstract states that LFPS achieves its speedups 'while preserving generation accuracy,' but Table 2 shows LFPS(a=0.2) with an average RULER score of 81.01 compared to 93.15 for full attention and 91.07 for exact Top-k, a degradation of 12.14 and 10.06 points, respectively. The gap persists at every evaluated context length, including 4K (77.84 vs 96.96 for Top-k) and 128K (73.21 vs 77.94 for Top-k). The RULER paragraph in Section 5 claims LFPS 'demonstrates outstanding adaptability to long contexts' without acknowledging this gap, so the central accuracy-preservation claim is contradicted by the paper's own measurements.
  2. [Section 4.2 vs Section 3] The head-sparsity bypass in Algorithm 1 (step 1) returns the mean value vector \bar{V}_h when the sparsity ratio ρ exceeds ε, but Section 3, Finding 3, motivates the bypass with the 'sink-average output' that combines the sink token with the mean of the remaining KV pairs and shows this combination has lower error than returning only the sink value. The implemented bypass uses only the mean, which is a different approximation and is not the one argued to preserve accuracy. This is a load-bearing discrepancy because the bypass is applied to heads with ρ>0.85, and the paper does not assess how this implementation choice affects final generation accuracy.
  3. [Appendix B.2 and Table 1] The sparsity threshold ε is selected by measuring accuracy on Qasper(8K+) in Table 3, and Qasper (reported as 'Qas' in Table 1) is also one of the LongBench tasks included in the average in Table 1. Tuning a hyperparameter on a task that is later included in the reported benchmark average constitutes selection on the test set, and the paper does not report that a separate validation split was used. This inflates the reported LongBench accuracy and weakens the comparison against baselines, all of which are evaluated with fixed hyperparameters.
minor comments (6)
  1. [Abstract and Section 5] The abstract's 9.6x speedup over exact Top-k is not reported in Section 5; the latency paragraph gives 4.3x, 8.1x, and 14.7x at 4K, 16K, and 64K, respectively. Please specify the settings for the 9.6x figure.
  2. [Section 4.2, Eq. (8)] Equation (8) refers to an 'approximate kurtosis' of the score distributions, but no estimator or computational procedure is defined; please specify how κ(x) is computed online and its cost.
  3. [References] Reference [3] (Loki) cites a paper on Jungian psychology rather than a low-rank KV-cache compression method; if this is the intended Loki reference, please provide the correct citation.
  4. [Figure 2c] Figure 2c's y-axis label says 'Error with full Attention' while the caption says 'Error relative to Full Attention'; please unify the notation and include units.
  5. [Table 4] Table 4 reports the dot-product budget for different a but does not report the resulting RULER accuracy for a=0.3 or a=0.4; adding this would help justify the choice of a=0.2.
  6. [General] There are several typos, e.g., 'vertor' in Section 1 and 'the system offloads the model’s KV cache... while extracting' in Section 4; please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LFPS's candidate-set prediction is a heuristic built from externally measured attention patterns and is validated against external benchmarks; the accuracy caveat is a correctness issue, not a circularity issue.

full rationale

The LFPS derivation chain is self-contained. Candidate sets are constructed from score tables (Equations 4, 7, 9, 10) that aggregate historical attention weights, and the final Top-k selection is then computed by exact dot-product within that candidate set; the claimed speedups are measured against full attention and exact Top-k on external benchmarks rather than derived from fitted parameters. The sparsity estimator in Equation (6) is a stated modeling heuristic with distributional assumptions (Appendix B.1), not a quantity fitted to the target output. Hyperparameters epsilon and a are tuned in Appendix B.2, and the tuning set includes Qasper, which also appears in the LongBench evaluation; this is benchmark tuning rather than circular reasoning. The abstract's unqualified 'preserving generation accuracy' claim is internally contradicted by the RULER results in Table 2, but that is an internal-consistency or correctness concern, not a circular step in the derivation. There are no load-bearing self-citations: the vertical/slash pattern terminology is attributed to external prior work [10], and the authors cite no prior work of their own as the basis for the method. Therefore the paper's core derivation does not reduce to its inputs by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

LFPS rests on several hand-chosen hyperparameters (r, epsilon, a, s, expansion offsets, local window size) and on domain assumptions about attention-pattern persistence, log-normal attention weights, comparability of unnormalized exp scores, and local clustering of important keys. No new entities are introduced.

free parameters (6)
  • decay factor r = 0.95
    Controls memory length of vertical/slash score tables in Eq. (10) and the normalization in Eq. (4); set by hand.
  • sparsity threshold epsilon = 0.85
    Used in Algorithm 1 and Eq. (6); tuned against Qasper accuracy in Table 3.
  • threshold adaptation parameter a = 0.2 and 0.3
    Controls candidate-set size via tau in Eq. (8); two values are reported, with an ablation in Table 4.
  • recent steps s = 32
    Number of prefill steps used to initialize score tables in Eq. (4); set by hand.
  • expansion offsets = {-1,0,1,2}
    Position expansion window in Eq. (9); chosen by hand.
  • local window size = 6
    Recent-token window in Eq. (5) for w_local; Appendix B.1 says last five, Eq. (5) uses indices t-6 through t-1.
assumptions (4)
  • domain assumption Vertical and slash attention patterns observed during prefill persist during decoding with only slow drift, so score tables updated by Eq. (10) remain predictive.
    The whole candidate-set construction relies on temporal stability of attention patterns; the paper supports it with overlap measurements in Figure 2a but assumes it across all benchmarks.
  • ad hoc to paper For non-sink tokens, attention weight distribution is log-normal, justifying the global term exp(qKbar/sqrt(d) + ||q||^2 sigma_hat^2/2) n in Eq. (5).
    Introduced in Appendix B.1, Eqs. (11)-(14), without empirical validation; used in the sparsity ratio that decides which heads can be bypassed.
  • ad hoc to paper Unnormalized exp scores from different contexts can be compared directly to estimate the sink-token proportion in Eq. (6).
    Softmax denominators are ignored; the estimator treats raw exp values as if they were normalized probabilities, which is not justified in the text.
  • domain assumption Important KV pairs cluster locally, so expanding candidates by offsets {-1,0,1,2} captures additional true Top-k positions.
    Based on the clustering observation in Figure 2b; used in Eq. (9) expansion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding." pith.science (2026). https://pith.science/paper/OT6KG5FY

@misc{pith2026250615704,
  author       = {Pith},
  title        = {Pith review of: Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OT6KG5FY}},
  note         = {Machine review of arXiv:2506.15704}
}
abstract

As large language models (LLMs) continue to support increasingly longer contexts, the memory demand for key-value (KV) caches during decoding grows rapidly, becoming a critical bottleneck in both GPU memory capacity and PCIe bandwidth. Sparse attention mechanisms alleviate this issue by computing attention weights only for selected key-value pairs. However, their indexing computation typically requires traversing all key vectors, resulting in significant computational and data transfer overhead. To reduce the cost of index retrieval, existing methods often treat each decoding step as an independent process, failing to exploit the temporal correlations embedded in historical decoding information. To this end, we propose LFPS(Learn From the Past for Sparse Indexing), an acceleration method that dynamically constructs sparse indexing candidates based on historical attention patterns. LFPS captures two prevalent trends in decoder attention -vertical patterns (attending to fixed positions) and slash patterns (attending to relative positions) -and incorporates a positional expansion strategy to effectively predict the Top-k indices for the current step. We validate LFPS on challenging long-context benchmarks such as LongBench-RULER, using Llama-3.1-8B-Instruct as the base model. Experimental results show that LFPS achieves up to 22.8$\times$ speedup over full attention and 9.6$\times$ speedup over exact Top-k retrieval on an RTX 4090 GPU and a single CPU core of a Xeon Gold 6430, respectively, while preserving generation accuracy. These results demonstrate that LFPS offers a practical and efficient solution for decoding optimization in long-context LLM inference.

Figures

Figures reproduced from arXiv: 2506.15704 by the authors.

Figure 1
Figure 1. (a) At a PCIe bandwidth of 32 GB/s, the time required to transfer 25% of the key vectors [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Across various sequence lengths and budget settings, our method achieves a higher over [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The overview of LFPS. During the prefilling phase, the system offloads the model’s KV cache to host memory while extracting two characteristic attention patterns: vertical and slash. In the decoding phase, prior to generating the current query vector, the system leverages pre-collected attention patterns to facilitate rapid retrieval of extended indices. Subsequent processing involves executing inner-product operati… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: For example, given the vertical and slash score table, we first compute their thresh￾olds and means using Equation (8). The initial candidate index is obtained based on the thresh￾olds and then expanded with offsets {-1,0,1,2}, keeping only indices whose scores exceed …
Figure 5
Figure 5. Figure 5: We evaluate the decoding throughput of Llama-3.1-8B-Instruct with and without LFPS [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: We selected the top 5% of tokens from Llama [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: This heatmap illustrates the cosine similarity between aggregated attention patterns across [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visualization of output vectors from different attention heads at various decoding steps in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 4 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    arXiv 2023

  3. [3]

    Loki then and now: the trickster against civilization

    Helena Bassil-Morozow. Loki then and now: the trickster against civilization. International Journal of Jungian Studies, 9(2):84–96, 2017

    work page 2017

  4. [4]

    Magicpig: Lsh sampling for efficient llm generation

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient llm generation. arXiv preprint arXiv:2410.16179, 2024. 9

    arXiv

  5. [5]

    A dataset of information-seeking questions and answers anchored in research papers

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021

    arXiv 2021

  6. [6]

    Human-like episodic memory for infinite context llms

    Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context llms. arXiv preprint arXiv:2407.09450, 2024

    arXiv 2024

  7. [7]

    Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines

    Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421, 2024

    arXiv 2024

  8. [8]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    arXiv 2024

Show all 33 references
  1. [9]

    Efficient llm inference with i/o-aware partial kv cache recomputation

    Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, and Murali Annavaram. Efficient llm inference with i/o-aware partial kv cache recomputation. arXiv preprint arXiv:2411.17089, 2024

    2024 arXiv

  2. [10]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  3. [11]

    Neo: Saving gpu memory crisis with cpu offloading for online llm inference

    Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. Neo: Saving gpu memory crisis with cpu offloading for online llm inference. arXiv preprint arXiv:2411.01142, 2024

    2024 arXiv

  4. [12]

    Compute or load kv cache? why not both? arXiv preprint arXiv:2410.03065, 2024

    Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z Morley Mao. Compute or load kv cache? why not both? arXiv preprint arXiv:2410.03065, 2024

    2024 arXiv

  5. [13]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  6. [14]

    {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 155–172, 2024

    2024

  7. [15]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems , 37:22947–22970, 2024

    2024

  8. [16]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  9. [17]

    Retrievalattention: Accelerating long- context llm inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long- context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024

    2024 arXiv

  10. [18]

    Clusterkv: Manipulating llm kv cache in semantic space for recallable compression

    Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating llm kv cache in semantic space for recallable compression. arXiv preprint arXiv:2412.03213, 2024

    2024 arXiv

  11. [19]

    Moba: Mixture of block attention for long-context llms

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189, 2025

    2025 arXiv

  12. [20]

    Sparq attention: Bandwidth-efficient llm inference

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient llm inference. arXiv preprint arXiv:2312.04985 , 2023

    2023 arXiv

  13. [21]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. 10

    2023 arXiv

  14. [22]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, page...

    2023

  15. [23]

    Shadowkv: Kv cache in shadows for high-throughput long-context llm inference

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference. arXiv preprint arXiv:2410.21465, 2024

    arXiv

  16. [24]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  17. [25]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  18. [26]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  19. [27]

    Infllm: Unveiling the intrinsic capacity of llms for under- standing extremely long sequences with training-free memory.arXiv e-prints, pages arXiv–2402, 2024

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for under- standing extremely long sequences with training-free memory.arXiv e-prints, pages arXiv–2402, 2024

    2024

  20. [28]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024

    arXiv

  21. [29]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  22. [30]

    Qwen2.5-1m technical report

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang R...

    2025 arXiv

  23. [31]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, 2022

    2022

  24. [32]

    Pqcache: Product quantization-based kvcache for long context llm inference

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization-based kvcache for long context llm inference. arXiv preprint arXiv:2407.12820, 2024

    2024 arXiv

  25. [33]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Syste...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.