Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frozen query-key geometry alone can classify attention heads as retrieval or streaming, enabling data-free sparse attention.

desk verdict A weight-only, data-free head classifier that mostly works; the caveats are evaluation rigor, not the core idea. read the letter →

arxiv 2608.06849 v1 pith:J4Z2PTZT submitted 2026-08-07 cs.CL cs.AI

Yehan Yang , Junyuan Shang , Yang Li , Guanqun Zhao , Shuohuan Wang , Dianhai Yu This is my paper · ORCID
classification cs.CLcs.AI
keywords long-contextLLMinferencesparseattentioneffectiverankretrievalheadsstreamingKVcachecompressiondata-freeheadclassificationquery-keygeometry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the functional role of an attention head—whether it searches the full context for specific content or mainly streams over nearby and sink tokens—is written into the model's frozen query and key projection weights, before any input is seen. It proposes Autonomy-of-Heads (AoH), a data-free and training-free classifier that labels low-effective-rank heads as retrieval heads needing global attention and high-effective-rank heads as streaming heads served by sink-plus-recent attention. Running long-context inference with this split is claimed to retain roughly 96.5% of full-attention performance at 50% sparsity while cutting prefill and decode latency and halving KV-cache memory at 256K tokens. If the paper is right, head diagnosis for sparse attention no longer needs runtime scores, calibration prompts, or learned gates.

What carries the argument

The load-bearing object is the kernel attention matrix $M_h = W_K^{h\top} W_Q^h \in \mathbb{R}^{d_{\text{model}}\times d_{\text{model}}}$, which is exactly the head-specific middle operator in the attention-score map $X_{\text{ctx}} W_K^{h\top} W_Q^h x_i$. Its nonzero singular spectrum is computed efficiently from the $d_{\text{head}}\times d_{\text{head}}$ proxy $C_h = (W_Q^h W_Q^{h\top})(W_K^h W_K^{h\top})$, whose eigenvalues are the squared singular values of $M_h$ by Sylvester's determinant theorem. From that spectrum AoH computes the effective rank, the exponential of the Shannon entropy of the normalized singular values, and uses it as a per-layer ranking: the lowest-$k$ heads (or, under GQA, the KV groups whose member heads have the lowest mean effective rank) are labeled retrieval heads and the rest streaming heads. This ranking is what carries the entire argument: it turns a qualitative retrieval-versus-streaming distinction into a one-time, input-free ordering of heads.

What would settle it

On a held-out long-context model, compute the effective-rank ordering of heads and then run the passkey-conversion experiment: restrict the lowest-effective-rank half of heads to sink-plus-recent attention and measure exact-match retrieval. If converting low-ER heads loses no more accuracy than converting the high-ER half, or if the Spearman correlation between effective rank and average attention distance becomes positive on a substantial task suite, the paper's central association is false.

Watch

Extended reading notes

Core claim

The central claim is that the effective rank of the kernel attention operator $M_h = W_K^{h\top} W_Q^h$, formed from frozen weights, predicts each head's contribution to long-context behavior. A concentrated singular spectrum (low effective rank) means the head has a few dominant query-key matching directions; AoH identifies these as retrieval heads, which should keep full global attention. A diffuse spectrum (high effective rank) means the head has no dominant global matching direction; AoH identifies these as streaming heads, which can safely attend only to sink tokens and a recent window. The paper supports the label with three kinds of evidence: per-head attention-distance patterns are stable across context lengths and negatively correlated with effective rank; converting low-effective-rank heads to streaming attention collapses passkey retrieval while converting high-effective-rank heads does not; and on LongBench, the effective-rank ordering beats random and reversed head selection and several alternative spectral metrics. As a direct corollary, sparse attention can begin at prefill, before any input is observed.

Load-bearing premise

Head role is a stable, input-independent property: the effective rank of the frozen query-key product, computed without positional rotations and without the value/output projections, keeps predicting whether a head needs global attention at the 32K–256K contexts where AoH is deployed.

Editorial extensions

If this is right

  • At 50% sparsity AoH retains about 96.5% of Full Attention performance on average across the tested Qwen2.5-7B, Qwen3-8B, and Llama3.1-8B-Instruct models, while cutting the KV-cache budget roughly in half.
  • Prefill and decode latency decrease by up to 41.4% and 66.0% at 50% sparsity, with larger gains reported at 75% sparsity (up to 3.24$\times$ and 9.14$\times$ at 256K context on Llama3.1-8B).
  • Because labels come from frozen weights, AoH applies sparse attention from prefill, with no calibration prompts, runtime attention traces, or trained head gates.
  • The effective-rank ordering is behaviorally meaningful: random head selection and reversed ordering (high effective rank as retrieval) degrade LongBench scores substantially, and effective rank outperforms Frobenius norm, spectral norm, and stable rank as a selection metric.
  • As a data-free head prior, AoH can be composed with other sparse-attention techniques; the paper demonstrates this by adding a cross-layer index-reuse variant (H2Share).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same frozen-weight criterion may transfer to encoder-decoder or multimodal models, but only if their query-key projections carry the same low-rank structure; the paper's limitation section explicitly says this was not tested.
  • An adaptive per-layer budget could be chosen from the distribution of effective ranks within each layer—layers with many low-ER heads keep more retrieval heads—rather than the fixed 50% split, which the paper's limitations section concedes is manually chosen.
  • The low-ER ordering could also seed token selection for neighboring layers: top-k positions attended by a retrieval head in one layer could serve as lazy indices for reuse layers, an idea the paper's H2Share appendix starts to develop but does not claim as a general method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Autonomy-of-Heads (AoH), a training-free and data-free method that classifies attention heads into retrieval heads (served with full attention) and streaming heads (served with sink-plus-recent-window attention) using the effective rank of the frozen kernel matrix M_h = W_K^{hT} W_Q^h. The authors derive an efficient d_head-dimensional proxy for the singular spectrum, present LongBench evaluations on Qwen2.5-7B, Qwen3-8B, Llama3.1-8B-Instruct, and a MoE model, report passkey and attention-distance evidence for the retrieval/streaming distinction, and measure latency and KV-cache savings. The central claim is that at 50% sparsity AoH retains 96.5% of Full Attention performance on average while outperforming random/reversed head selection and sparse baselines.

Significance. If the central assumption holds—that a head's retrieval or streaming role is encoded in the frozen query-key spectrum and is stable across inputs—AoH would be a practically valuable contribution, since it removes runtime score computation, calibration prompts, and learned gates from head diagnosis. The paper has several genuine strengths: the trace-reduction and AB-BA eigenvalue arguments in Appendix L are mathematically sound; the d_head-dimensional proxy (Eq. 5) is a clean and useful reduction; the passkey conversion experiment (Figure 4) is a well-designed causal-style check that low-effective-rank heads matter for retrieval; the Random and Reverse ablations are appropriately controlled; and the RoPE-aware ranking stability analysis (Table 6) is a thoughtful robustness check. However, the load-bearing empirical generalization (input-independent head roles) is supported only by aggregate, single-seed evidence, and there are notable per-category failures that the paper does not discuss.

major comments (4)
  1. [§6.1, §6.2, Tables 2 and 9] The paper reports no error bars, no multiple seeds, and no statistical tests. The headline comparisons against Full Attention are small (e.g., Qwen3-8B: 38.78 vs. 39.10, a 0.32-point gap; Llama3.1-8B: 47.55 vs. 48.65, a 1.10-point gap), and the differences against DuoAttention and other baselines are also of single-digit size. Without variance estimates, it is impossible to tell whether these gaps reflect method behavior or run-to-run noise. The authors should report standard deviations or confidence intervals over at least several seeds, or, if this is too expensive, on a representative subset of tasks and models.
  2. [§3, §6.2, Table 9, Figure 4] The central assumption that head roles are stable across inputs is only tested through aggregate statistics. Figure 3 and Figure 7 report per-head average attention distances or correlations, which average over many prompts and contexts; Figure 4 is a single-model (Llama3.1-8B-Instruct) passkey study. These do not rule out per-input role switching. A concrete warning sign is the Synthetic category on Qwen2.5-7B in Table 9: Full Attention scores 28.51, AoH at 50% sparsity scores 19.11 (a 9.4-point drop), and AoH is far below SnapKV (29.47) and Quest (20.37) on this category. This is consistent either with head-role instability or with a budget that is too aggressive for retrieval-heavy inputs, but the paper does not distinguish these possibilities. The authors should provide per-task or per-input evidence of label stability—for example, per-task correlations between effective rank and attention distance, or an analysis of how the low-effective-rank head set performs on individual prompts—or explicitly temper the claim that the frozen label is input-independent.
  3. [§6.4, Figure 6, Limitations] The 50% sparsity budget is a manually chosen hyperparameter, as the Limitations section admits, and Figure 6 shows that this value is selected from the LongBench accuracy-sparsity trade-off curves. This means that although the head ordering is data-free, the method's overall configuration is calibrated on the evaluation benchmark. The paper should clearly separate these two aspects and either (a) justify the budget as a fixed prior or deployment constraint, or (b) provide a sensitivity analysis that does not use the test benchmark to choose the budget. Without this, the 'data-free' claim in the title and abstract is overstated.
  4. [Abstract, §6.3, Table 11] The abstract states that AoH reduces prefill and decode latency by up to 41.4% and 66.0%, respectively. This is inconsistent with the reported efficiency tables: for Llama3.1-8B at 75% sparsity and 256K context, Table 11 reports a 3.24x prefill speedup (about 69.1% reduction) and a 9.14x decode speedup (about 89.1% reduction), and Table 12 reports a 14.5x decode speedup on Qwen2.5-7B. The authors should clarify the exact configuration used for the abstract's numbers or update them to match the maximum speedups reported in the tables.
minor comments (5)
  1. [Conclusion] The Conclusion contains a typo: 'trainging-free baselines' should be 'training-free baselines'.
  2. [Appendix F] The heading 'Reults for MoE models' contains a typo ('Reults' should be 'Results').
  3. [Figure 3] The label 'Effecive Rank' in Figure 3 contains a typo; it should be 'Effective Rank'. Additionally, the color scale on the heatmap is not described, making it difficult to interpret the magnitude of the effective-rank values.
  4. [Table 10] The MMLU accuracy values in Table 10 are suspiciously constant (0.600, 0.595, 0.605) across all sink-size, recent-window, and sparsity combinations. The authors should explain whether this is finite-precision rounding or a genuine lack of sensitivity, and report standard deviations.
  5. [§5, Eq. (7)] In Eq. (7), the notation is dense and the normalization of the singular values is implicit. It would clarify the presentation to explicitly define the normalized singular values \hat{\sigma}_k before writing the effective-rank formula, as is done in Eq. (4).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Frozen-weight ER ordering predicts head role and is validated on external LongBench/passkey behavior; no target quantity is an input by construction.

full rationale

AoH's derivation is self-contained against external behavior. The classification signal is the effective rank of M_h = W_K^{hT}W_Q^h (Eqs. 2-4), computed solely from frozen query/key projections; the paper never fits the ER rule to the LongBench or passkey outcomes it later reports. The retrieval/streaming association is introduced as an empirical observation (Figures 3-4) and tested by the accuracy drop under head-to-streaming conversion and by LongBench, so the target result is not an input to the classifier. The d_head proxy (Eqs. 5-6) follows from the AB/BA nonzero-eigenvalue identity and is a computational reduction, not a restatement of the claim. The only same-author citation (NACL, Chen et al. 2024) appears in related work and is not load-bearing. The manual 50% budget and the choice of effective rank over alternative spectral metrics (Table 4) on the same LongBench suite are selection effects rather than circular reductions; they affect generalizability, not the derivation. Hence no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It relies on the frozen query-key product as a proxy for head function, which is an interpretative assumption rather than a fitted parameter or invented object. The only fitted choices are deployment hyperparameters.

free parameters (3)
  • Sparsity budget (retrieval head fraction) = 50% (with sweeps from 0 to 100%)
    Per-layer fraction of heads or KV groups labelled retrieval; selected from the LongBench accuracy-efficiency trade-off in Figure 6 and kept constant across layers, as stated in the Limitations section.
  • Sink cache size = 128 tokens
    Chosen from the LongBench-Qasper sensitivity table (Table 3) as part of the default streaming-head cache.
  • Recent window size = 256 tokens
    Chosen from Table 3; affects streaming-head KV cache and accuracy, especially at high sparsity.
assumptions (5)
  • standard math Nonzero singular values of M_h are computable from the d_head-dimensional Gram proxy C_h = (W_Q W_Q^T)(W_K W_K^T) via AB and BA sharing nonzero eigenvalues.
    Invoked in Section 4 and Algorithm 1; used for the efficient spectrum computation.
  • domain assumption Head role (retrieval vs streaming) is a stable property of the frozen QK geometry and does not change with input, context length, or RoPE phase.
    Core premise tested in Section 3 (Figure 3) and Section 6.4 (Table 6); if false, data-free labels misassign heads.
  • domain assumption Low-effective-rank heads are functionally necessary for long-range retrieval, as suggested by passkey conversion experiments.
    Figure 4 tests one model (Llama3.1-8B-Instruct) and one task; generalization to all LongBench tasks is assumed.
  • domain assumption Sink and recent tokens suffice for streaming heads.
    Adopted from StreamingLLM-style attention and used to bound streaming-cache size in Section 5.
  • domain assumption GQA group labels can be aggregated by the mean of member query-head effective ranks without losing retrieval information.
    Supported by the GQA aggregation ablation in Table 5, but only tested on one model and one benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry." pith.science (2026). https://pith.science/paper/J4Z2PTZT

@misc{pith2026260806849,
  author       = {Pith},
  title        = {Pith review of: Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4Z2PTZT}},
  note         = {Machine review of arXiv:2608.06849}
}
abstract

Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input-dependent and costly to deploy. We propose Autonomy-of-Heads (AoH), a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections. AoH defines the kernel attention operator $M_h = W_K^{h\top}W_Q^h$ and uses its effective-rank as a weight-space measure of head function: concentrated spectra indicate a small number of dominant query-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads. We further derive an efficient $d_\text{head}$-dimensional computation that avoids constructing the full $d_\text{model}\times d_\text{model}$ matrix. We conducted extensive experiments across models demonstrating that at 50\% sparsity, AoH retains 96.5\% of Full Attention performance on average while reducing prefill and decode latency by up to 41.4\% and 66.0\%, respectively, and KV-cache memory by 50.0\% at 256K tokens.

Figures

Figures reproduced from arXiv: 2608.06849 by the authors.

Figure 1
Figure 1. Input-Dependent Methods vs. AoH. mass (Zhang et al., 2023; Wan et al., 2024), local windows or selectors (Xiao et al., 2024; Fu et al., 2025; Mohtashami and Jaggi, 2023; DeepSeek-AI et al., 2025), cross-layer reuse (Deshmukh et al., 2025; Gao et al., 2026; Bai et al., 2026; Brandon et al., 2024), or learned head gates (Xiao et al., 2025; Lin et al., 2026). As shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the Mh singular value spectrum and attention maps in the Qwen2.5-7B model for the sentence “The best fruit is orange. What is the best fruit? Orange.”, showing that concentrated spectra correspond to retrieval heads while uniform spectra correspond to streaming heads. Left: Retrieval heads (e.g., Layer 15, Head 20) attend selectively to contextually relevant tokens, requiring full attention.Center: … view at source ↗
Figure 3
Figure 3. Empirical relationship between effective-rank and head attention distance on Qwen2.5-7B. Left: Per-head [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Passkey retrieval under progressive head-to [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: AoH Prefill and Decoding Efficiency Across Context Lengths vs. Full Attention. Gray and red bars [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of three LLMs under [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Correlation between effective-rank and empirical attention behavior. Negative correlations indicate that [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Accuracy-efficiency trade-off of AoH. tion only at the most challenging 128K context and 100% insertion depth. H Additional Efficiency Results Measurement protocol. To isolate the effect of head-level sparsity from low-level kernel engineer￾ing, the decode latencies re…
Figure 10
Figure 10. Figure 10: Ablation of AoH head classification under [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Overview of H2Share. Left: Offline preprocessing classifies heads with AoH, partitions layers into anchor–reuse blocks, and builds cross-layer head mappings. Right: During decoding, anchor retrieval heads compute full attention and select important-token indices; reus…
Figure 12
Figure 12. Figure 12: Ablation Experiments on LongBench. H2Share offers a better KV budget-accuracy trade-off. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    arXiv preprint arXiv:2602.03560 , year=

    HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing , author=. arXiv preprint arXiv:2602.03560 , year=

    arXiv

  2. [2]

    DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads , url =

    Xiao, Guangxuan and Tang, Jiaming and Zuo, Jingwei and Guo, Junxian and Yang, Shang and Tang, Haotian and Fu, Yao and Han, Song , booktitle =. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads , url =

    work page

  3. [3]

    arXiv preprint arXiv:2602.04541 , year=

    LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding , author=. arXiv preprint arXiv:2602.04541 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2512.16391 , year=

    Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference , author=. arXiv preprint arXiv:2512.16391 , year=

    arXiv

  5. [5]

    TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention , url =

    Yang, Lijie and Zhang, Zhihao and Chen, Zhuofu and Li, Zikun and Jia, Zhihao , booktitle =. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention , url =

    work page

  6. [6]

    arXiv preprint arXiv:2603.12201 , year=

    IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse , author=. arXiv preprint arXiv:2603.12201 , year=

    arXiv

  7. [7]

    2: Pushing the Frontier of Open Large Language Models , author=

    DeepSeek-V3. 2: Pushing the Frontier of Open Large Language Models , author=. arXiv preprint arXiv:2512.02556 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2603.13314 , year=

    Linear Predictability of Attention Heads in Large Language Models , author=. arXiv preprint arXiv:2603.13314 , year=

    arXiv

Show all 43 references
  1. [9]

    arXiv preprint arXiv:2412.15115 , year =

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =

    arXiv

  2. [10]

    arXiv preprint arXiv:2510.26692 , year=

    Kimi linear: An expressive, efficient attention architecture , author=. arXiv preprint arXiv:2510.26692 , year=

    arXiv

  3. [11]

    https://www.anthropic.com/engineering/building-effective-agents , year=

    Building effective agents , author=. https://www.anthropic.com/engineering/building-effective-agents , year=

  4. [12]

    Advances in Neural Information Processing Systems , volume=

    H2o: Heavy-hitter oracle for efficient generative inference of large language models , author=. Advances in Neural Information Processing Systems , volume=

  5. [13]

    arXiv preprint arXiv:2406.13035 , year=

    D2o: Dynamic discriminative operations for efficient long-context inference of large language models , author=. arXiv preprint arXiv:2406.13035 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2502.18845 , year=

    Sliding window attention training for efficient large language models , author=. arXiv preprint arXiv:2502.18845 , year=

    arXiv

  7. [15]

    Advances in Neural Information Processing Systems , volume=

    Random-access infinite context length for transformers , author=. Advances in Neural Information Processing Systems , volume=

  8. [16]

    Advances in Neural Information Processing Systems , volume=

    Reducing transformer key-value cache size with cross-layer attention , author=. Advances in Neural Information Processing Systems , volume=

  9. [17]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  10. [18]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  11. [19]

    Advances in neural information processing systems , volume=

    Pytorch: An imperative style, high-performance deep learning library , author=. Advances in neural information processing systems , volume=

  12. [20]

    arXiv preprint arXiv:1503.00519 , year=

    Generalizations of Sylvester's determinantal identity , author=. arXiv preprint arXiv:1503.00519 , year=

    arXiv

  13. [21]

    Advances in neural information processing systems , volume=

    Root mean square layer normalization , author=. Advances in neural information processing systems , volume=

  14. [22]

    arXiv preprint arXiv:2004.05150 , year=

    Longformer: The Long-Document Transformer , author=. arXiv preprint arXiv:2004.05150 , year=

    2004 arXiv

  15. [23]

    Advances in Neural Information Processing Systems , volume=

    Big Bird: Transformers for Longer Sequences , author=. Advances in Neural Information Processing Systems , volume=

  16. [24]

    Efficient Streaming Language Models with Attention Sinks , url =

    Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike , booktitle =. Efficient Streaming Language Models with Attention Sinks , url =

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    You only cache once: Decoder-decoder architectures for language models , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    arXiv preprint arXiv:2503.18893 , year=

    xkv: Cross-layer svd for kv-cache compression , author=. arXiv preprint arXiv:2503.18893 , year=

    arXiv

  19. [27]

    arXiv preprint arXiv:2508.16134 , year=

    Commonkv: Compressing kv cache with cross-layer parameter sharing , author=. arXiv preprint arXiv:2508.16134 , year=

    arXiv

  20. [28]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    arXiv

  21. [29]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  22. [30]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  23. [31]

    2026 , howpublished=

    MiMo-V2.5 , author=. 2026 , howpublished=

    2026

  24. [32]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  25. [33]

    SnapKV: LLM Knows What You are Looking for Before Generation , url =

    Li, Yuhong and Huang, Yingbing and Yang, Bowen and Venkitesh, Bharat and Locatelli, Acyr and Ye, Hanchen and Cai, Tianle and Lewis, Patrick and Chen, Deming , booktitle =. SnapKV: LLM Knows What You are Looking for Before Generation , url =. doi:10.52202/079017-0722 , editor =

    doi

  26. [34]

    2024 , cdate=

    Jiaming Tang and Yilong Zhao and Kan Zhu and Guangxuan Xiao and Baris Kasikci and Song Han , title=. 2024 , cdate=

    2024

  27. [35]

    RazorAttention: Efficient

    Hanlin Tang and Yang Lin and Jing Lin and Qingsen Han and Danning Ke and Shikuan Hong and Yiwu Yao and Gongyi Wang , booktitle=. RazorAttention: Efficient. 2025 , url=

    2025

  28. [36]

    NACL : A General and Effective KV Cache Eviction Framework for LLM at Inference Time

    Chen, Yilong and Wang, Guoxia and Shang, Junyuan and Cui, Shiyao and Zhang, Zhenyu and Liu, Tingwen and Wang, Shuohuan and Sun, Yu and Yu, Dianhai and Wu, Hua. NACL : A General and Effective KV Cache Eviction Framework for LLM at Inference Time. Proceedings of the 62nd Annual ...

    2024 doi

  29. [37]

    arXiv preprint arXiv:2006.16362 , year=

    Multi-head attention: Collaborate instead of concatenate , author=. arXiv preprint arXiv:2006.16362 , year=

    2006 arXiv

  30. [38]

    arXiv preprint arXiv:2308.16369 , year=

    Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills , author=. arXiv preprint arXiv:2308.16369 , year=

    arXiv

  31. [39]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  32. [40]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  33. [41]

    Transformers: State-of-the-Art Natural Language Processing

    Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Pierric and Rault, Tim and Louf, Remi and Funtowicz, Morgan and Davison, Joe and Shleifer, Sam and von Platen, Patrick and Ma, Clara and Jernite, Yacine an...

    2020

  34. [42]

    2007 15th European signal processing conference , pages=

    The effective rank: A measure of effective dimensionality , author=. 2007 15th European signal processing conference , pages=. 2007 , organization=

    2007

  35. [43]

    Journal of the ACM (JACM) , volume=

    Sampling from large matrices: An approach through geometric functional analysis , author=. Journal of the ACM (JACM) , volume=. 2007 , publisher=

    2007

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.