Pith. sign in

REVIEW 4 major objections 5 minor 67 references

The paper claims that memory collision in linear attention is a spherical-packing problem, and that rank-one positive semidefinite features square interference to break the capacity wall.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:01 UTC pith:ECAEQBYQ

load-bearing objection A genuinely new geometric framework for linear-attention capacity with solid theory and honest experiments, but the headline empirical wins are not yet cleanly linked to the theory. the 4 major comments →

arxiv 2607.17419 v1 pith:ECAEQBYQ submitted 2026-07-19 cs.LG cs.AIcs.ITmath.ITstat.ML

Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

Ayoub Ghriss , Sourav Chakraborty This is my paper
classification cs.LG cs.AIcs.ITmath.ITstat.ML MSC 68T0752C1715B48
keywords linear attentionassociative recallspherical packingsymmetric conespositive semidefinite featuresWelch boundmemory collisionkernelized attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Associative recall in linear attention collapses because many keys share one fixed state and interfere; the paper argues this is fundamentally a spherical-packing problem. Once nonnegative attention weights are certified by a self-dual homogeneous cone, the admissible feature geometries are organized by the Koecher–Vinberg classification, and the rank-one positive semidefinite map emerges as the favorable choice: it squares the interference between keys. Above the Welch interference floor, Theorem 4 shows such features can pack exponentially many key directions in the projection dimension, and Theorem 7 raises the idealized retrieval threshold from Θ(p) to Θ(p²). The paper implements this as KATA, whose fused kernels run at competitive or faster throughput than standard attention, and reports that KATA-M1 retains 0.985 MQAR at 16× the training context with roughly one quarter of softmax's KV-cache entries. A sympathetic reader should care because this replaces ad hoc feature-map choices with a principled capacity theory and gives concrete prescriptions for architecture design.

Core claim

The central claim, stated on the paper's own terms, is that memory collision is the saturation of a spherical-packing problem, and that 'rank-one positive semi-definite (PSD) features offer a favorable capacity–interference tradeoff.' The key objects are the feature map ψ(u)=uuᵀ and the interference floor from Welch's bound. Theorem 4 proves that, for any fixed tolerance above the floor, the PSD cone packs at least (1/2)(1−ε)^(−(p−1)/2) keys, exponential in the projection dimension p, while Theorem 7 shows the idealized pSNR retrieval threshold becomes Θ(p²) rather than Θ(p). The normalized denominator of linear attention is recovered as a parameter-free convex output gate. Experiments on lo

What carries the argument

The rank-one PSD feature lift ψ(u)=uuᵀ, which embeds p-dimensional unit vectors into the p(p+1)/2-dimensional symmetric PSD cone so that the kernel becomes the squared cosine ⟨ψ(u),ψ(v)⟩=(uᵀv)². This squaring converts distractor interference from ε to ε² and is what upgrades the retrieval threshold to Θ(p²). It is supported by a spherical-cap greedy packing argument (Theorem 4) and by a pSNR analysis at the Welch scale (Theorem 7); the Koecher–Vinberg classification of self-dual homogeneous cones certifies nonnegativity and organizes the admissible geometries.

Load-bearing premise

The load-bearing premise is that trained query and key directions really behave like the unit-norm spherical points the packing theory assumes; the main KATA runs do not enforce per-head ℓ2 normalization (it appears only as an ablation), so the empirical gains could in principle come from the larger state rather than from the certified packing.

What would settle it

Run the KATA-M1 MQAR protocol with per-head ℓ2 normalization of queries and keys enforced versus removed; if 0.985 accuracy at 16× context persists without normalization while the measured pairwise key-feature interference stays above the Welch floor, then the spherical-packing model is not what is driving recall.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Feature maps for linear attention can be chosen from cone geometry rather than heuristics: self-duality certifies nonnegative weights, and the normalization denominator is a parameter-free convex output gate.
  • Above the Welch floor, rank-one PSD features pack exponentially many keys in the projection dimension; below it, capacity is polynomial—so operating above the floor is essential for long-context recall.
  • The idealized retrieval threshold rises from Θ(p) to Θ(p²), meaning the same raw key dimension can support quadratically more distinct addresses before interference dominates.
  • KATA-M1 holds 0.985 MQAR at 16× training context with roughly one quarter of softmax's KV-cache entries, approaching softmax-level associative recall at fixed state size.
  • The same pSNR framework gives a softmax capacity law: at fixed temperature, capacity is finite and grows exponentially in 1/τ; unconstrained temperature needs Θ(log T) to hold a fixed margin.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the theory: sweep head dimension p for PSD features and measure MQAR; if the packing model transfers, recall should improve roughly quadratically with p above the Welch floor.
  • The theory suggests that gates and delta rules are not additional capacity mechanisms but learnability/overwrite controls layered on top of the address geometry; combining a PSD address map with gating should dominate either alone.
  • Only the real cone factors were implemented; complex, quaternionic, and exceptional factors from the classification remain untested, and the same proof machinery predicts denser spherical codes there.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Kernelized Linear Attention Activations (KATA), a framework that derives linear-attention feature maps from symmetric-cone geometry. The central claim is that associative recall in linear attention is governed by a spherical-packing capacity, and that rank-one PSD features (ψ(u)=uu^T) provide a better capacity–interference tradeoff than raw linear or Lorentz features. The authors prove that, above the Welch interference floor, rank-one PSD features admit exponentially many near-orthogonal keys in the projection dimension (Theorem 4), and that an idealized pSNR analysis raises the retrieval threshold from Θ(p) to Θ(p²) (Theorem 7). They implement KATA in fused Triton kernels, report competitive throughput against FlashAttention-2 and Gated DeltaNet, and present MQAR, repeated-key overwrite, NIAH, and 340M-parameter LM pretraining experiments. The paper is notable for its clear separation of geometric capacity, learnability, and contextual fluency, and for reporting state sizes and parameter counts alongside accuracy.

Significance. If the central claims are correct, this is a significant contribution: it provides a principled geometric account of why certain linear-attention variants fail on associative recall, gives an explicit construction with exponential capacity in a fixed projection dimension, and demonstrates hardware-feasible implementations. The paper is also commendable for shipping reproducible kernel code, reporting state sizes and parameter counts, and including honest discussion of learnability and contextual-fluency confounds. The theoretical results appear internally consistent, and the empirical gains on MQAR and NIAH are substantial. However, the load-bearing assumption of unit-norm query/key geometry is not enforced in the primary experiments, and the central empirical comparisons confound feature geometry with state size. Thus the significance is conditional on closing the theory–experiment gap.

major comments (4)
  1. [§3, §5.1, Appendix F.4] The capacity theory assumes unit-norm query and key directions: the interference definition in Eq. (5) restricts to S^{d-1}, Theorem 4 uses u∈S^{p-1}, and Theorem 7 normalizes the matched score to one. However, the primary MQAR runs in Table 2 use the 'Plain KATA' configuration of Appendix F.4, where q and k are raw learned projections and the optional per-head ℓ2 normalization is disabled. Appendix A.1's RMSNorm lift applies to the 340M pretraining setup, not to the MQAR rows. For unnormalized features the squared kernel is ||q||²||k||²cos²θ, so learned norms are multiplicative retrieval weights. The observed MQAR gains could therefore arise from separating key/query norms rather than from the certified PSD packing. This is a load-bearing gap between Assumptions 1–3 and Tables 2/4; the primary experiments should either enforce per-head ℓ2 normalization or the theory must be extended to
  2. [Table 2] The empirical comparison confounds feature geometry with state size. The best KATA rows use states of 135,168 or 266,240 entries (KATA-M2, KATA-M1), while the Gated DeltaNet baselines at comparable parameter counts use 8,192 or 32,768-entry states. The paper's central claim is a favorable capacity–interference tradeoff 'at a fixed state size,' but the main table does not hold state size fixed. A matched-state ablation (e.g., a PSD variant with the same state budget as GDN, or a larger-state GDN baseline) is needed to attribute the accuracy gains to the PSD feature geometry rather than to the larger recurrent state.
  3. [Table 4] The only experiment that explicitly enforces unit-norm geometry contradicts the theory's mechanism. In Table 4, the 'ℓ2-norm (KATA-M2)' ablation has UUID recall of 0.224 at 2K, while the primary KATA-M2 (which uses RMSNorm but not per-head ℓ2 normalization) reaches 0.616. If the certified packing were the driver of recall, enforcing the unit-norm precondition should help, not sharply hurt. This suggests that the empirical recall behavior depends on other aspects of the learned geometry, and that the theory–experiment link in the pretrained setting is not secured by the current evidence.
  4. [§3.3, §5.1] Theorem 7's pSNR criterion is an idealized model, but the paper does not report any direct measurement of trained-model key/query coherence, norm distributions, or pSNR values. The MQAR results in Table 2 are only architectures' accuracies; no evidence is provided that the trained KATA keys approximate the assumed packing, or that the transition from near-perfect to failing recall occurs near the predicted Θ(p²) threshold. To make the capacity theory load-bearing, the authors should measure (or at least report) the learned pairwise coherence and norm statistics for the KATA variants, and compare the measured pSNR against the theoretical threshold at the lengths where accuracy drops.
minor comments (5)
  1. [§2.3 vs Appendix F.4] The Lorentz feature map is defined differently in the main text (ψ_L(y)=(y,||y||_2)) and in F.4 (ψ(x)=(y,||y||_2(1+s^2))). Please reconcile the notation and define the exact map once.
  2. [Equation (22)] The direct-sum symbol appears to be typeset incorrectly (shown as 'L_{r≥0}'). Use a proper direct sum or tensor-sum notation.
  3. [Table 4 header] The header 'Arora’24(Based) cloze' is visually cramped and should be formatted more clearly, e.g., 'Based cloze [Arora et al., 2024]'.
  4. [Appendix A.1] The construction of M as max_j ||k̄_j||_2 requires a finite dictionary; if D_γ = 0 the bound is undefined. This is a minor edge case but should be stated.
  5. [General] Some table numbers are referenced only loosely in the text (e.g., 'full table in Appendix F'). Please ensure all appendix tables have explicit numbers and are referenced consistently.

Circularity Check

0 steps flagged

No circularity: the capacity derivations are self-contained and benchmarked against external classical bounds; the theory–experiment normalization gap is a validity concern, not a circular reduction.

full rationale

The paper's derivation chain does not reduce to its own inputs. The linear-attention recurrence (Eq. 2) is an algebraic substitution; Proposition 1 is an identity from normalized linear attention. Assumptions 1–3 are stated modeling assumptions, and the Koecher–Vinberg classification is an external mathematical result, not a self-citation. The capacity theory (Theorems 3, 4, 5, 6, 13) is built on standard packing bounds (Welch, Rankin, spherical-cap estimates, MUB and DeVore constructions) and is not fitted to the experiments. Theorem 7 is an explicitly idealized pSNR calculation with stated isotropic-value and Welch-scale assumptions, so its Θ(p) vs Θ(p^2) threshold is a direct consequence of the squared kernel, not a fitted parameter renamed as a prediction. Proposition 8 similarly derives softmax sharpening from the pSNR model. The empirical MQAR, NIAH, and Based results do not set any theory constants; they are compared with baselines under matched configurations. The skeptic's concern that the main KATA-M1 runs do not apply per-head ℓ2 normalization while the theory assumes unit-norm directions is a legitimate assumption-validity or theory–experiment interface gap, but it is not circularity: nothing in the derivation is defined in terms of the empirical outcomes, and the RMSNorm lifting argument in Appendix A.1 is stated rather than used to redefine the experimental claim. There are no load-bearing self-citations and no renamed fitted quantities. Accordingly, the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 0 invented entities

The theory imports standard results (Koecher–Vinberg, Welch, Rankin, DeVore, MUB) and adds explicit modeling assumptions (unit-sphere geometry, isotropic values, self-dual cone feature images). No free parameters are fitted to make the derivations work; the empirical hyperparameter sweeps are not part of the theory.

axioms (7)
  • standard math Koecher–Vinberg classification of symmetric cones (orthant, Lorentz, PSD factors).
    Invoked in Section 2.3 to reduce admissible feature geometries to the ordinary real irreducible factors. Taken as background.
  • standard math Welch bound on maximum pairwise inner products of unit vectors.
    Theorem 3 and Appendix A.3; the base capacity floor for orthant and PSD raw vectors.
  • standard math Rankin's strict spherical-cap bound (T ≤ d+1 for mutually obtuse vectors).
    Appendix C.2; used to prove the Lorentz Rankin wall (Theorem 13).
  • domain assumption Assumptions 1–3: self-dual nonnegative geometry, isometric invariance, homogeneity.
    Section 2.1; imposed so that Koecher–Vinberg applies. Not derived from attention itself, and Assumption 2 in particular is strong.
  • domain assumption Query and key directions are effectively unit-norm; RMSNorm logits lift to a unit sphere in one higher dimension.
    Section 3 opening and Appendix A.1. Primary KATA experiments do not enforce per-head ℓ2 norm, so the theory applies only approximately to the reported trained models.
  • domain assumption Isotropic independent Gaussian value model with ideal retrieval iff pSNR > 1.
    Equation (10) and Section 3.3. A modeling criterion for the capacity theorems; trained readouts and correlated values need not obey it.
  • standard math Greedy spherical-cap construction achieves the exponential packing lower bound.
    Theorem 4 and Lemma 10; a standard volumetric covering argument, independent of attention.

pith-pipeline@v1.3.0-alltime-deepseek · 39025 in / 21565 out tokens · 202904 ms · 2026-08-01T18:01:09.988777+00:00 · methodology

0 comments
read the original abstract

Linear attention promises constant-time recurrent inference but degrades sharply on associative recall. We formulate attention recall as a spherical-packing problem and introduce Kernelized Linear Attention Activations (KATA), a framework whose feature maps are derived from first principles by certifying nonnegative attention weights through a self-dual homogeneous cone. Building on this observation, we show that rank-one positive semi-definite (PSD) features offer a favorable capacity--interference tradeoff. KATA recovers a parameter-free convex output gate and characterizes associative capacity through the Welch interference floor. For tolerances above this floor, KATA enlarges the state without adding parameters and admits spherical codes with exponentially many keys in the projection dimension. We implement KATA as fused Triton kernels at two operating points: a flash-attention-style forward up to ${\sim}1.6\times$ FlashAttention-2 throughput, and an exact $O(T)$ chunked-state form that reaches ${\sim}11\times$ FlashAttention-2 forward throughput at $131$k tokens. An associative scan of the first-order feature lowers the inter-chunk recurrence depth to $O(\log(T/C))$ for chunk size $C$ and averages ${\sim}2.4\times$ the throughput of a matched sequential linear-attention baseline. On long-range MQAR and repeated-key overwrite, several KATA variants outperform Gated DeltaNet, with parameter counts and state sizes reported alongside accuracy. Induction preserves near-perfect recall, while kernel benchmarks show that the maps can be implemented efficiently. KATA retains $0.985$ MQAR at a $16\times$ out-of-distribution length, approaching the softmax with roughly one quarter of the KV-cache entries. Experiments on 340M-parameter LLMs reveal a feature-dependent fluency trade-off and clarify how positional embeddings, delta rules, and decay gates interact with feature geometry.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 1 canonical work pages

  1. [1]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages =

    Shen, Zhuoran and Zhang, Mingyuan and Zhao, Haiyu and Yi, Shuai and Li, Hongsheng , title =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages =. 2021 , doi =

    2021

  2. [2]

    Transformers are RNNs: fast autoregressive transformers with linear attention , year =

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are RNNs: fast autoregressive transformers with linear attention , year =. Proceedings of the 37th International Conference on Machine Learning , articleno =

  3. [3]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  4. [4]

    arXiv preprint arXiv:2402.01032 , year=

    Repeat after me: Transformers are better than state space models at copying , author=. arXiv preprint arXiv:2402.01032 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2502.13685 , year=

    Mom: Linear sequence modeling with mixture-of-memories , author=. arXiv preprint arXiv:2502.13685 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2307.08621 , year =

    Retentive Network: A Successor to Transformer for Large Language Models , author =. arXiv preprint arXiv:2307.08621 , year =. 2307.08621 , archivePrefix =

    Pith/arXiv arXiv

  7. [7]

    International Conference on Learning Representations , year=

    Rethinking Attention with Performers , author=. International Conference on Learning Representations , year=

  8. [8]

    Linear Transformers Are Secretly Fast Weight Programmers , booktitle =

    Schlag, Imanol and Irie, Kazuki and Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers , booktitle =. 2021 , url =

    2021

  9. [9]

    International Conference on Learning Representations , year=

    Efficiently Modeling Long Sequences with Structured State Spaces , author=. International Conference on Learning Representations , year=

  10. [10]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Dao, Tri and Gu, Albert , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  11. [11]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Yang, Songlin and Wang, Bailin and Shen, Yikang and Panda, Rameswar and Kim, Yoon , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  12. [12]

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length , url =

    Yang, Songlin and Wang, Bailin and Zhang, Yu and Shen, Yikang and Kim, Yoon , booktitle =. Parallelizing Linear Transformers with the Delta Rule over Sequence Length , url =. doi:10.52202/079017-3668 , editor =

    doi:10.52202/079017-3668

  13. [13]

    The Thirteenth International Conference on Learning Representations , year=

    Gated Delta Networks: Improving Mamba2 with Delta Rule , author=. The Thirteenth International Conference on Learning Representations , year=

  14. [14]

    SIAM Journal on Scientific and Statistical Computing , volume =

    Schreiber, Robert and Van Loan, Charles , title =. SIAM Journal on Scientific and Statistical Computing , volume =. 1989 , doi =

    1989

  15. [15]

    American Journal of Mathematics , volume =

    Koecher, Max , title =. American Journal of Mathematics , volume =. 1957 , publisher =

    1957

  16. [16]

    Vinberg, E. B. , title =. Trudy Moskovskogo Matematicheskogo Obshchestva , volume =. 1963 , url =

    1963

  17. [17]

    Analysis on Symmetric Cones , series =

    Faraut, Jacques and Kor. Analysis on Symmetric Cones , series =. 1994 , isbn =

    1994

  18. [18]

    and Majumdar, Satya N

    Dean, David S. and Majumdar, Satya N. , title =. Physical Review Letters , volume =. 2006 , doi =

    2006

  19. [19]

    Israel Journal of Mathematics , volume =

    Blekherman, Grigoriy , title =. Israel Journal of Mathematics , volume =. 2006 , doi =

    2006

  20. [20]

    Hopfield Networks Is All You Need , booktitle =

    Ramsauer, Hubert and Sch. Hopfield Networks Is All You Need , booktitle =. 2021 , url =

    2021

  21. [21]

    , title =

    Krotov, Dmitry and Hopfield, John J. , title =. Advances in Neural Information Processing Systems 29 (

  22. [22]

    On a Model of Associative Memory with Huge Storage Capacity , journal =

    Demircigil, Mete and Heusel, Judith and L. On a Model of Associative Memory with Huge Storage Capacity , journal =. 2017 , doi =

    2017

  23. [23]

    First Conference on Language Modeling (

    Gu, Albert and Dao, Tri , title =. First Conference on Language Modeling (. 2024 , url =

    2024

  24. [24]

    Findings of the Association for Computational Linguistics:

    Peng, Bo and Alcaide, Eric and Anthony, Quentin and Albalak, Alon and Arcadinho, Samuel and Biderman, Stella and Cao, Huanqi and Cheng, Xin and Chung, Michael and Derczynski, Leon and Du, Xingjian and Grella, Matteo and Gv, Kranthi and He, Xuzheng and Hou, Haowen and Kazienko, Przemyslaw and Kocon, Jan and Kong, Jiaming and Koptyra, Bart. Findings of the ...

    2023

  25. [25]

    and Kong, Lingpeng , title =

    Peng, Hao and Pappas, Nikolaos and Yogatama, Dani and Schwartz, Roy and Smith, Noah A. and Kong, Lingpeng , title =. 9th International Conference on Learning Representations (. 2021 , url =

    2021

  26. [26]

    The Tenth International Conference on Learning Representations (

    Qin, Zhen and Sun, Weixuan and Deng, Hui and Li, Dongxu and Wei, Yunshen and Lv, Baohong and Yan, Junjie and Kong, Lingpeng and Zhong, Yiran , title =. The Tenth International Conference on Learning Representations (. 2022 , url =

    2022

  27. [27]

    Transformer Circuits Thread , year =

    Olsson, Catherine and Elhage, Nelson and Nanda, Neel and Joseph, Nicholas and DasSarma, Nova and Henighan, Tom and Mann, Ben and Askell, Amanda and Bai, Yuntao and Chen, Anna and Conerly, Tom and Drain, Dawn and Ganguli, Deep and Hatfield-Dodds, Zac and Hernandez, Danny and Johnston, Scott and Jones, Andy and Kernion, Jackson and Lovitt, Liane and Ndousse...

  28. [28]

    Rankin, R. A. , title =. Proceedings of the Glasgow Mathematical Association , volume =

  29. [29]

    , title =

    Welch, Lloyd R. , title =. IEEE Transactions on Information Theory , volume =

  30. [30]

    , title =

    Hopfield, John J. , title =. Proceedings of the National Academy of Sciences , volume =

  31. [31]

    , title =

    Oseledets, Ivan V. , title =. SIAM Journal on Scientific Computing , volume =

  32. [32]

    , title =

    Schölkopf, Bernhard and Smola, Alexander J. , title =. 2001 , month =. doi:10.7551/mitpress/4175.001.0001 , url =

    doi:10.7551/mitpress/4175.001.0001 2001

  33. [33]

    2023 , eprint=

    Zoology: Measuring and Improving Recall in Efficient Language Models , author=. 2023 , eprint=

    2023

  34. [34]

    2025 , eprint=

    Understanding Transformer from the Perspective of Associative Memory , author=. 2025 , eprint=

    2025

  35. [35]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  36. [36]

    2024 , eprint =

    Simple linear attention language models balance the recall-throughput tradeoff , author =. 2024 , eprint =

    2024

  37. [37]

    2024 , eprint =

    Titans: Learning to Memorize at Test Time , author =. 2024 , eprint =

    2024

  38. [38]

    arXiv preprint arXiv:2505.23666 , year=

    Lola: Low-rank linear attention with sparse caching , author=. arXiv preprint arXiv:2505.23666 , year=

    arXiv

  39. [39]

    2025 , eprint=

    Modern Methods in Associative Memory , author=. 2025 , eprint=

    2025

  40. [40]

    arXiv preprint arXiv:2510.01450 , year=

    Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression , author=. arXiv preprint arXiv:2510.01450 , year=

    arXiv

  41. [41]

    Annals of Physics , volume =

    Optimal state-determination by mutually unbiased measurements , author =. Annals of Physics , volume =. 1989 , doi =

    1989

  42. [42]

    Algorithmica , volume =

    A new proof for the existence of mutually unbiased bases , author =. Algorithmica , volume =. 2002 , doi =

    2002

  43. [43]

    Journal of Complexity , volume =

    Deterministic constructions of compressed sensing matrices , author =. Journal of Complexity , volume =. 2007 , doi =

    2007

  44. [44]

    2024 , url =

    FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , author =. 2024 , url =

    2024

  45. [45]

    Proceedings of the 39th International Conference on Machine Learning (

    Transformer Quality in Linear Time , author =. Proceedings of the 39th International Conference on Machine Learning (

  46. [46]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  47. [47]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  48. [48]

    2026 , eprint=

    The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence , author=. 2026 , eprint=

    2026

  49. [49]

    arXiv preprint arXiv:2410.05258 , year =

    Differential Transformer , author =. arXiv preprint arXiv:2410.05258 , year =

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2310.04625 , year =

    Copy Suppression: Comprehensively Understanding an Attention Head , author =. arXiv preprint arXiv:2310.04625 , year =

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2410.02703 , year =

    Selective Attention Improves Transformer , author =. arXiv preprint arXiv:2410.02703 , year =

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2411.07176 , year =

    More Expressive Attention with Negative Weights , author =. arXiv preprint arXiv:2411.07176 , year =

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2309.17453 , year =

    Efficient Streaming Language Models with Attention Sinks , author =. arXiv preprint arXiv:2309.17453 , year =

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2103.03404 , year =

    Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth , author =. arXiv preprint arXiv:2103.03404 , year =

    Pith/arXiv arXiv

  55. [55]

    First Conference on Language Modeling , year=

    RULER: What's the Real Context Size of Your Long-Context Language Models? , author=. First Conference on Language Modeling , year=

  56. [56]

    2023 , howpublished=

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , author=. 2023 , howpublished=

    2023

  57. [57]

    2401.02385 , archivePrefix=

    Peiyuan Zhang and Guangtao Zeng and Tianduo Wang and Wei Lu , year=. 2401.02385 , archivePrefix=

    Pith/arXiv arXiv

  58. [58]

    2016 , eprint =

    Tables of the Existence of Equiangular Tight Frames , author =. 2016 , eprint =

    2016

  59. [59]

    1993 , issn =

    Capacity of Associative Memory Using a Nonmonotonic Neuron Model , journal =. 1993 , issn =. doi:10.1016/0893-6080(93)90014-N , url =

    work page doi:10.1016/0893-6080(93)90014-n 1993

  60. [60]

    Physical Review Letters , volume =

    Exponential Capacity of Dense Associative Memories , author =. Physical Review Letters , volume =. 2024 , doi =

    2024

  61. [61]

    2026 , eprint =

    Mamba-3: Improved Sequence Modeling Using State Space Principles , author =. 2026 , eprint =

    2026

  62. [62]

    and Lindenstrauss, Joram , title =

    Johnson, William B. and Lindenstrauss, Joram , title =. Contemporary Mathematics , volume =. 1984 , doi =

    1984

  63. [63]

    Journal of Computer and System Sciences , volume =

    Achlioptas, Dimitris , title =. Journal of Computer and System Sciences , volume =. 2003 , doi =

    2003

  64. [64]

    Combinatorics, Probability and Computing , volume =

    Alon, Noga , title =. Combinatorics, Probability and Computing , volume =. 2009 , doi =

    2009

  65. [65]

    and Elad, Michael , title =

    Donoho, David L. and Elad, Michael , title =. Proceedings of the National Academy of Sciences , volume =. 2003 , month = mar, doi =

    2003

  66. [66]

    2025 , eprint=

    Critical attention scaling in long-context transformers , author=. 2025 , eprint=

    2025

  67. [67]

    2023 , eprint=

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning , author=. 2023 , eprint=

    2023