Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Supervising keep-or-drop decisions with future attention lets language models discard most of the KV cache while keeping nearly full reasoning accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 09:23 UTC pith:ELHFPE4V

load-bearing objection Solid systems paper: future-attention boundary supervision plus delayed scoring actually moves fixed-budget KV eviction on hard math, with matched baselines that make the gains believable. the 3 major comments →

arxiv 2607.05061 v2 pith:ELHFPE4V submitted 2026-07-06 cs.LG cs.AI

KVpop -- Key-Value Cache Compression with Predictive Online Pruning

Lukas Hauzenberger , Niklas Schmidinger , Anamaria-Roberta Hartl , David Stap , Thomas Schmied , Sebastian B\"ock , G\"unter Klambauer , Sepp Hochreiter This is my paper
classification cs.LG cs.AI
keywords KV cache compressionpredictive online pruningfuture-attention targetlearned evictiondelayed scoringsparse attentionfixed-budget cacheautoregressive decoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Autoregressive transformers store a growing key-value cache so later tokens can attend to earlier ones, but that cache quickly becomes a memory bottleneck. Most eviction methods drop tokens using fixed heuristics or local scores that do not track whether a token will still matter later. KVpop instead trains a lightweight scorer to decide, at a fixed budget, which older tokens to keep by supervising the keep-or-drop decision with the future attention mass each token receives after it leaves a short protected window. That target is computed during training with a transposed attention pass that never materializes a dense map, and the scorer can optionally wait until the eviction boundary so near-future context can inform the decision. On hard math reasoning, the method keeps roughly 97–98 percent of full-cache performance on Qwen3-4B at 75 percent and 88 percent compression, and approaches full teacher performance on Qwen3-8B, while also generalizing to code and science tasks and keeping peak memory nearly flat for long generations.

Core claim

Fixed-budget KV eviction can be cast as a supervised prediction problem at the eviction boundary: a scorer trained against each token’s post-window future-attention mass, optionally delayed until near-future context is available, recovers nearly full dense-attention accuracy under aggressive compression and outperforms both heuristic and prior learned eviction baselines on mathematical reasoning.

What carries the argument

Future-attention target plus boundary-aware retention loss: each token’s keep/drop label is defined by the mean attention mass it receives after leaving the protected window (aggregated across query heads sharing a KV head), recovered efficiently by a transposed-attention pass that reuses sparse log-sum-exp normalizers; the scorer is trained with a pairwise logistic loss only against the teacher’s top-k cutoff token, and may use a delayed memory-based (mLSTM) readout that scores the token only when it becomes eligible.

Load-bearing premise

The paper assumes that how much future attention a token receives after leaving the protected window is a good enough proxy for whether keeping that token actually helps the final answer, even when the normalizers used to measure that attention come from the sparse student rather than full dense attention.

What would settle it

Train the same scorers and budgets but replace the future-attention target with a pure recency or random ranking (or with dense-LSE targets that systematically disagree with the sparse-LSE approximation) and check whether pass@1 on AIME/HMMT collapses relative to the reported KVpop numbers under identical compression ratios.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Aggressive fixed KV budgets (75–88 percent compression) can preserve near-teacher math-reasoning accuracy when eviction is supervised by future attention rather than local proxies.
  • Delaying the scoring decision until a token leaves a short protected window lets memory-based scorers use near-future context and improves token accuracy over immediate scoring.
  • The same learned policy, trained only on math traces, remains competitive on out-of-domain code generation and STEM reasoning without retraining the eviction rule.
  • Homogeneous per-head fixed budgets yield more regular sparse patterns and lower long-generation latency than dynamic gates that concentrate capacity in a few heads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If future-attention mass is a transferable utility signal, the same boundary-supervised scorer could be distilled once and reused across related model families without re-collecting dense teacher maps.
  • The method suggests that irreversible eviction can approach sparse-retrieval quality on reasoning workloads provided the keep decision is postponed until a short future window is observed.
  • Hybrid designs that give a few critical layers denser budgets while keeping most heads under KVpop-style fixed top-k may further close residual quality gaps without returning to linear memory growth.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes KVpop, a post-training retrofit for fixed-budget KV-cache eviction in pretrained Transformers. Each head keeps sink tokens, a protected recent window, and a learned long-range top-k set. Scorers are supervised at the eviction boundary by a future-attention mass target (Eqs. 2–3, 5–7), computed via a training-only transposed-attention pass that reuses sparse LSE normalizers and avoids materializing dense S×S maps (Eqs. 8–9, App. D). A delayed mLSTM scorer can defer the keep/drop decision until the protected-window exit, using near-future context. On AIME/HMMT with Qwen3-4B/8B, KVpop and a stateless MLP variant outperform StreamingLLM, TOVA, a trained StreamingLLM+, and parameter-matched DMS under matched budgets (Table 1), with near-teacher OOD results on GPQA-D and LiveCodeBench (Table 2) and favorable latency/VRAM at long generation lengths (Fig. 4).

Significance. KV-cache growth is a central systems bottleneck for long-context decoding; a learned fixed-budget eviction policy that preserves hard reasoning quality would be practically valuable. The work’s main technical contributions—boundary-aware supervision with an efficiently computed future-attention target, and delayed stateful scoring that uses protected-window context before the decision—are concrete and well-motivated relative to heuristic eviction and to DMS-style relaxed gates. Empirically, the matched-budget comparisons (especially StreamingLLM+ isolating pattern adaptation, and DMS isolating the retention objective) and the homogeneous per-head budget that yields cleaner long-generation throughput are strengths. If the reported near-teacher AIME/HMMT retention under 75–88% compression holds under broader evaluation, the method is a meaningful advance for learned eviction retrofits.

major comments (3)
  1. [Abstract; §1 Contributions; Table 1; §5 Conclusion] Relative-performance numbers are inconsistent across the manuscript and do not always match Table 1. The abstract reports 98%/97% of teacher on Qwen3-4B at 75%/88% compression; §1 Contributions and §5 Conclusion report 95%/94% (and 95%/99% for Qwen3-8B). Table 1 gives Rel. 0.98/0.97 for Qwen3-4B and 1.00/1.00 for Qwen3-8B. These are load-bearing headline claims; please recompute from the Abs. columns, reconcile abstract/intro/conclusion with Table 1, and state the exact aggregation (mean of four pass@1s vs. other).
  2. [§4.2; Figure 5; Table 1] The delayed-scoring claim is central (“uniquely among learned eviction methods”) but the only direct ablation (Fig. 5) reports token accuracy (+0.2 points after 2k steps), not AIME/HMMT pass@1 or other task metrics under the same budget. Without a task-level delayed vs. non-delayed comparison for the stateful scorer, it is hard to judge how much of Table 1’s KVpop vs. KVpopmlp gap is due to delay rather than recurrence capacity or the reduced top-k used to match memory. Please add (or clearly state the absence of) a task-level ablation.
  3. [§3.2; Eqs. (8)–(9); Appendix D.3] The future-attention target uses sparse-query LSE as a proxy for dense causal LSE (Eqs. 8–9; App. D.3), and the paper asserts this “matches the dense-LSE target in downstream performance.” That approximation is an axiom of the training signal, yet no main-text or appendix table quantifies target agreement or task delta under dense vs. sparse LSE. A short ablation (or correlation of targets) would make the supervision claim more solid; if already run, report it explicitly.
minor comments (5)
  1. [§4.1; Table 1] Pass@1 is estimated with 16 rollouts; please report uncertainty (e.g., bootstrap SE or std over seeds) for Table 1 averages, especially where gaps to DMS are a few points.
  2. [§4.3; Figure 6; Appendix Figure 9] Figure 6 / Appendix Fig. 9 eviction visualizations are informative but qualitative; a simple quantitative breakdown (e.g., retention rates for numeric vs. discourse/operator tokens) would strengthen §4.3.
  3. [§4 Baselines; Appendix F] Clarify whether StreamingLLM+ and DMS use identical sink/window/top-k (or equivalent B) and the same Nemotron-Math training mixture and KL setup as KVpop; §4 states matching budget and scorer parameter count for DMS, but a one-row hyperparameter table would help reproducibility.
  4. [§3.1; Eq. (3); Appendix B.1] Notation: r̂ vs. r_tgt and effective scores with decay (Eq. 5) are clear, but Agg_g is only specified as max in the main text with alternatives in App. B; a brief main-text sentence on why max was chosen would help.
  5. [Throughout; Algorithm 1] Typos / polish: “KVcache” spacing is inconsistent; “themLSTM” often lacks a space after “the”; Algorithm 1 line breaks and the “˜LSE” notation in the PDF are hard to read.

Circularity Check

1 steps flagged

No significant circularity: ordinary teacher-supervised distillation of a keep/drop scorer, evaluated on held-out task accuracy rather than reconstruction of the training target.

specific steps
  1. other [Sec. 3.2 / Eq. (9) and Appendix D.3]
    "Using the dense causal LSE would make this identity exact. Instead, we reuse the sparse query LSE values ˜LSE already returned by the student attention pass... We find empirically that this sparse-LSE approximation matches the dense-LSE target in downstream performance"

    The future-attention target used to supervise the scorer is computed with the student's own sparse-attention normalizers rather than independent dense LSE. This is a mild self-reference in the training signal (target depends on the current sparse policy). It is not a tautology of the main claim: the claim is held-out task accuracy under fixed budgets, the paper reports the approximation matches dense-LSE downstream, and baselines/ablations still separate learned ranking from the sparse pattern itself.

full rationale

KVpop trains lightweight scorers with a boundary-aware pairwise loss against a future-attention mass target derived from the dense (or sparse-LSE-approximated) teacher attention, while the base model is updated only by KL distillation to the dense teacher (Algorithm 1; Eqs. 2–3, 5–7). The sparse-LSE reuse (Eq. 9, App. D.3) is an efficiency approximation for the normalizer, not a definition of the claimed result; the paper reports that it matches dense-LSE downstream performance. Evaluation is pass@1 on AIME/HMMT and OOD GPQA-D/LCB under fixed budgets, with ablations (StreamingLLM+ vs learned ranking, DMS under matched budget/params, delayed vs non-delayed mLSTM) that isolate the supervision target from mere pattern adaptation. There is no self-definitional loop, no fitted parameter renamed as a prediction of the same quantity, and no load-bearing uniqueness theorem imported from overlapping authors. Self-citations (e.g., mLSTM background, FlexAttention) supply architecture or tooling, not the central claim. Score 1 only for the minor self-referential sparse-LSE approximation, which is not load-bearing against the task-accuracy claim.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard Transformer/GQA attention math, the modeling choice that future attention mass after a protected window is the right keep/drop label, several hand-chosen cache and training hyperparameters, and two invented algorithmic constructs (the future-attention target pipeline and the delayed mLSTM scorer). No new physical entities are postulated; free parameters are the usual retrofit knobs plus the learned per-head decay.

free parameters (4)
  • per-head long-range top-k budget k (and sinks s, window w) = s=4, w=256, k∈{2016,4032}
    Fixed cache composition B=s+w+k is chosen by hand to hit 75%/88% compression (e.g., sinks=4, window=256, top-k 2016/4032); results are reported only at these operating points.
  • per-head decay factor γ_h (log-decay range) = γ_min=0.999, γ_max=0.999999
    Learned per head inside a fixed [γ_min, γ_max] interval to impose recency bias on effective scores; range is a design choice that shapes ranking.
  • scorer and base learning rates / schedule = scorer 1e-3→8e-5; base 8e-5; 2k steps
    Scorer peak LR 1e-3 cosine, base LR fixed 8e-5, 2000 steps, KL top-256; chosen for the retrofit and not derived.
  • boundary-loss temperature τ and margin/balance weights = τ=1.0; margin weighting enabled
    Pairwise logistic temperature and optional ω_margin / ω_bal reweight ambiguous keep/drop labels; affect which boundary decisions dominate training.
axioms (4)
  • domain assumption Mean future attention mass a key receives after leaving the protected window is a valid supervision signal for keep-or-drop decisions that preserve generation quality.
    Core modeling choice in §3.1 Eqs. (2)–(3); the entire training objective is built on this proxy rather than direct task reward.
  • ad hoc to paper Sparse-query LSE normalizers from the fixed-budget student attention closely approximate dense causal LSE for the purpose of the future-attention target.
    Appendix D.3 and §3.2; exact dense LSE is replaced for efficiency and claimed to match downstream performance without a quantitative agreement study.
  • domain assumption Homogeneous fixed per-head budgets are preferable for GPU execution even if dynamic per-head allocation (as in DMS) could concentrate capacity.
    Stated in efficiency discussion and limitations; underpins the latency advantage claim versus ragged DMS caches.
  • standard math Standard causal multi-head / grouped-query attention and KL distillation to a dense teacher are valid training substrates for a sparse retrofit.
    Background Transformer machinery (Vaswani et al., Ainslie GQA) used throughout Algorithm 1.
invented entities (2)
  • Future-attention target with transposed-attention computation no independent evidence
    purpose: Provide a dense-map-free training label equal to post-window attention mass for each key, used to supervise the eviction boundary decision.
    Defined in Eqs. (2)–(3) and implemented via swapped Q/K FlexAttention pass (Fig. 2, Appendix D); not a physical entity but a new algorithmic construct the method depends on.
  • Delayed mLSTM importance scorer no independent evidence
    purpose: Maintain a recurrent memory and score a token only when it exits the protected window, incorporating near-future context into the keep/drop score.
    §3.3 and Fig. 3; claimed unique among learned eviction methods. Ablation shows a small token-accuracy gain; no external validation outside this paper.

pith-pipeline@v1.1.0-grok45 · 27526 in / 3832 out tokens · 37448 ms · 2026-07-11T09:23:42.792638+00:00 · methodology

0 comments
read the original abstract

Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.

Figures

Figures reproduced from arXiv: 2607.05061 by Anamaria-Roberta Hartl, David Stap, G\"unter Klambauer, Lukas Hauzenberger, Niklas Schmidinger, Sebastian B\"ock, Sepp Hochreiter, Thomas Schmied.

Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Transposed Attention. (1) Blue tiles show the future attention mass for token t. (2) Transposing q and k turns the per-key column-sum into a per-query row-sum, which attention kernels return as their auxiliary LSE. Future-attention target. Let S be the training sequence length and let g ∈ {1, . . . , G} index the query heads that share KV head h under grouped-query attention (Ainslie et al., 2023). Let p (… view at source ↗
Figure 3
Figure 3. Figure 3: KVpop stateful eviction policy. (1) The mLSTM-memory is updated with the most recent KV pair. (2) For delayed scoring, the memory is read with the token at position q − w that has just exited the protected sliding window. (3) The mLSTM emits an importance score, and ranking determines whether the token is kept or evicted. Stateless scorers. The simplest scorer predicts token importance from the token’s own… view at source ↗
Figure 4
Figure 4. Figure 4: Inference efficiency on Qwen3-8B. We measure end-to-end decoding latency and peak allocated VRAM at batch size 1, 75% KV compression, and varying generation lengths. Dense attention incurs growing memory cost as the KV cache expands, whereas DMS and KVpop grow more slowly. KVpop further reduces latency, maintaining stable throughput across long generations and outperforming both dense attention and DMS at … view at source ↗
Figure 6
Figure 6. Figure 6: Eviction patterns in a mathematical reasoning trace. We visualize the last 112 tokens of a randomly sampled sequence. Rows denote layers, columns denote token strings, and color indicates how many attention heads in a layer retained the corresponding token. KVpop tends to evict purely numeric tokens more often, while retaining reasoning-structural tokens. 4.2 Delayed scoring improves stateful eviction 500 … view at source ↗
Figure 5
Figure 5. Figure 5: Effect of delayed scoring. To￾ken accuracy with and without delayed scor￾ing. We ablate the effect of delayed scoring in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Alignment with the future-attention oracle. We plot the top-k retention recall distribution over heads for all layers of Qwen3-4B at 75% KV-cache compression. The metric measures what fraction of the full-attention teacher’s top-k retained tokens are also kept by the learned KVpopscorer under the same cache budget. Higher values indicate better agreement with the teacher retention policy. 0 1 2 3 4 5 6 7 8… view at source ↗
Figure 8
Figure 8. Figure 8: DMS evict ratio. We compute the median eviction ratio over 100 randomly sampled sequences for Qwen3-4B with DMS at a 75% compression ratio. A value of 0 indicates dense/full attention, whereas a value of 1 indicates that attention is restricted to the local sliding window. DMS exhibits highly heterogeneous eviction patterns: some heads use the full long-range token budget and behave close to dense-attentio… view at source ↗
Figure 9
Figure 9. Figure 9: KVpopeviction patterns (Qwen3-4B, CR=75%, final 112 tokens of five sampled sequences). Color shows how many heads retained each token (x-axis) per layer (y-axis). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 1 canonical work pages

  1. [1]

    2024 , pages =

    Advances in Neural Information Processing Systems , author =. 2024 , pages =. doi:10.52202/079017-3417 , language =

    doi:10.52202/079017-3417 2024

  2. [2]

    Neural computation , keywords =

    Hochreiter, Sepp and Schmidhuber, J. Neural computation , keywords =

  3. [3]

    Michael Zhang and Simran Arora and Rahul Chalamala and Benjamin Frederick Spector and Alan Wu and Krithik Ramesh and Aaryan Singhal and Christopher Re , booktitle=. Lo. 2025 , url=

    2025

  4. [4]

    The Twelfth International Conference on Learning Representations , year=

    The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry , author=. The Twelfth International Conference on Learning Representations , year=

  5. [5]

    2023 , eprint=

    Gaussian Error Linear Units (GELUs) , author=. 2023 , eprint=

    2023

  6. [6]

    2023 , eprint=

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints , author=. 2023 , eprint=

    2023

  7. [7]

    2025 , eprint=

    LoLA: Low-Rank Linear Attention With Sparse Caching , author=. 2025 , eprint=

    2025

  8. [8]

    2025 , eprint=

    Olmo 3 , author=. 2025 , eprint=

    2025

  9. [9]

    T oken S elect: Efficient Long-Context Inference and Length Extrapolation for LLM s via Dynamic Token-Level KV Cache Selection

    Wu, Wei and Pan, Zhuoshi and Fu, Kun and Wang, Chao and Chen, Liyi and Bai, Yunchu and Wang, Tianfu and Wang, Zheng and Xiong, Hui. T oken S elect: Efficient Long-Context Inference and Length Extrapolation for LLM s via Dynamic Token-Level KV Cache Selection. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi...

    work page doi:10.18653/v1/2025.emnlp-main.1079 2025

  10. [10]

    and Soloveychik, Ilya and Kamath, Purushotham , booktitle =

    Adnan, Muhammad and Arunkumar, Akhil and Jain, Gaurav and Nair, Prashant J. and Soloveychik, Ilya and Kamath, Purushotham , booktitle =. Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference , volume =

  11. [11]

    Attention

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and. Attention. 2017 , booktitle =

    2017

  12. [12]

    2023 , pages =

    Advances in Neural Information Processing Systems , author =. 2023 , pages =

    2023

  13. [13]

    The Twelfth International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. The Twelfth International Conference on Learning Representations , year=

  14. [14]

    arXiv preprint arXiv:2512.15489 , year =

    Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision , author =. arXiv preprint arXiv:2512.15489 , year =

    arXiv

  15. [15]

    Forty-second International Conference on Machine Learning , year=

    Understanding and Improving Length Generalization in Recurrent Models , author=. Forty-second International Conference on Machine Learning , year=

  16. [16]

    Inference-

    Łańcucki, Adrian and Staniszewski, Konrad and Nawrot, Piotr and Ponti, Edoardo , month = oct, year =. Inference-

  17. [17]

    , month = apr, year =

    Nawrot, Piotr and Li, Robert and Huang, Renjie and Ruder, Sebastian and Marchisio, Kelly and Ponti, Edoardo M. , month = apr, year =. The. doi:10.48550/arXiv.2504.17768 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.17768

  18. [18]

    Overcoming

    Zhan, Zhihao and Zhao, Jianan and Zhu, Zhaocheng and Tang, Jian , month = oct, year =. Overcoming

  19. [19]

    Nawrot, Piotr and Łańcucki, Adrian and Chochowski, Marcin and Tarjan, David and Ponti, Edoardo , month = jul, year =. Dynamic. Proceedings of the 41st

  20. [20]

    Yuhong Li and Yingbing Huang and Bowen Yang and Bharat Venkitesh and Acyr Locatelli and Hanchen Ye and Tianle Cai and Patrick Lewis and Deming Chen , booktitle=. Snap. 2024 , url=

    2024

  21. [21]

    Proceedings of the 41st

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , month = jul, year =. Proceedings of the 41st

  22. [22]

    doi:10.48550/arXiv.2512.02556 , abstract =

    DeepSeek Team , month = dec, year =. doi:10.48550/arXiv.2512.02556 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2512.02556

  23. [23]

    Efficient

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , month = oct, year =. Efficient. Proceedings of the 29th. doi:10.1145/3600006.3613165 , abstract =

    doi:10.1145/3600006.3613165

  24. [24]

    Proceedings of Machine Learning and Systems , author =

    Keyformer:. Proceedings of Machine Learning and Systems , author =. 2024 , pages =

    2024

  25. [25]

    doi:10.48550/arXiv.2503.08879 , abstract =

    Wang, Guangtao and Upasani, Shubhangi and Wu, Chen and Gandhi, Darshan and Li, Jonathan and Hu, Changran and Li, Bo and Thakker, Urmish , month = mar, year =. doi:10.48550/arXiv.2503.08879 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.08879

  26. [26]

    Efficient

    Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike , month = oct, year =. Efficient

  27. [27]

    In-context

    Zeng, Zihao and Lin, Bokai and Hou, Tianqi and Zhang, Hao and Deng, Zhijie , month = apr, year =. In-context. doi:10.48550/arXiv.2410.12876 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.12876

  28. [28]

    and Gao, Yifei and Chang, Chi-Chih and Jain, Nilesh and Abdelfattah, Mohamed S

    Akhauri, Yash and AbouElhamayed, Ahmed F. and Gao, Yifei and Chang, Chi-Chih and Jain, Nilesh and Abdelfattah, Mohamed S. , month = mar, year =. doi:10.48550/arXiv.2503.07518 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.07518

  29. [29]

    , title =

    Fenwick, Peter M. , title =. Software---Practice and Experience , volume =. 1994 , doi =

    1994

  30. [30]

    , author=

    FlexAttention: A Programming Model for Generating Fused Attention Variants. , author=. Eighth Conference on Machine Learning and Systems , year=

  31. [31]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  32. [32]

    arXiv:2004.05150 , year=

    Longformer: The Long-Document Transformer , author=. arXiv:2004.05150 , year=

    Pith/arXiv arXiv 2004

  33. [33]

    arXiv preprint arXiv:2603.15590 , year=

    Effective Distillation to Hybrid xLSTM Architectures , author=. arXiv preprint arXiv:2603.15590 , year=

    Pith/arXiv arXiv

  34. [34]

    2024 , booktitle =

    Yang, Songlin and Wang, Bailin and Shen, Yikang and Panda, Rameswar and Kim, Yoon , title =. 2024 , booktitle =

    2024

  35. [35]

    arXiv preprint arXiv:2503.13427 , year=

    xlstm 7b: A recurrent llm for fast and efficient inference , author=. arXiv preprint arXiv:2503.13427 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2411.04165 , year=

    Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences , author=. arXiv preprint arXiv:2411.04165 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2410.22391 , year=

    A large recurrent action model: xLSTM enables fast inference for robotics tasks , author=. arXiv preprint arXiv:2410.22391 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2406.04303 , year=

    Vision-lstm: xlstm as generic vision backbone , author=. arXiv preprint arXiv:2406.04303 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2505.23719 , year=

    Tirex: Zero-shot forecasting across long and short horizons with enhanced in-context learning , author=. arXiv preprint arXiv:2505.23719 , year=

    arXiv

  40. [40]

    2020 , booktitle =

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, François , title =. 2020 , booktitle =

    2020

  41. [41]

    arXiv preprint arXiv:2510.00636 , year=

    Expected attention: Kv cache compression by estimating attention from future queries distribution , author=. arXiv preprint arXiv:2510.00636 , year=

    arXiv

  42. [42]

    arXiv preprint arXiv:2412.06464 , year=

    Gated delta networks: Improving mamba2 with delta rule , author=. arXiv preprint arXiv:2412.06464 , year=

    Pith/arXiv arXiv

  43. [43]

    2024 , booktitle =

    Gu, Albert and Dao, Tri , title =. 2024 , booktitle =

    2024

  44. [44]

    Transformers are Multi-State RNN s

    Oren, Matanel and Hassid, Michael and Yarden, Nir and Adi, Yossi and Schwartz, Roy. Transformers are Multi-State RNN s. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.1043

    doi:10.18653/v1/2024.emnlp-main.1043 2024

  45. [45]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Random-Access Infinite Context Length for Transformers , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  46. [46]

    2025 , eprint=

    Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention , author=. 2025 , eprint=

    2025

  47. [47]

    Bowman , booktitle=

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle=. 2024 , url=

    2024

  48. [48]

    The Thirteenth International Conference on Learning Representations , year=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. The Thirteenth International Conference on Learning Representations , year=

  49. [49]

    Zefan Cai and Yichi Zhang and Bofei Gao and Yuliang Liu and Yucheng Li and Tianyu Liu and Keming Lu and Wayne Xiong and Yue Dong and Junjie Hu and Wen Xiao , booktitle=. Pyramid. 2025 , url=

    2025

  50. [50]

    2025 , eprint=

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=. 2025 , eprint=

    2025

  51. [51]

    BERT Rediscovers the Classical NLP Pipeline

    Tenney, Ian and Das, Dipanjan and Pavlick, Ellie. BERT Rediscovers the Classical NLP Pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1452

    doi:10.18653/v1/p19-1452 2019

  52. [52]

    Gonzalez and Clark Barrett and Ying Sheng , booktitle=

    Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng , booktitle=. 2024 , url=

    2024