pith. sign in

arxiv: 2606.21848 · v1 · pith:7WBXE6Z5new · submitted 2026-06-20 · 💻 cs.CL · cs.AI

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

Xin Gao This is my paper

Pith reviewed 2026-06-26 12:18 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords keyless attentionvalue-only cachekv cache reductionattention factorizationtransformer efficiencylanguage modelinginference optimization
0
0 comments X

The pith

Keyless Attention removes the key projection to create a value-only cache that halves KV memory use.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Keyless Attention, which drops the key projection from the standard QKV setup and works only with queries and values. This produces a Value-Only Cache that cuts KV cache memory and access overhead by exactly 50 percent while keeping decode speed the same or better. The change is framed as moving from a depth-2 to a depth-3 factorization of the attention bilinear form, where a value-space routing matrix takes the place of the key projection and couples routing with retrieval. Across five models spanning GPT-2, Pythia, Qwen2, and Llama, the method matches or beats standard attention on perplexity in four cases and improves zero-shot commonsense scores in four of five benchmarks for the 557M model.

Core claim

Standard attention realizes a depth-2 factorization of the attention bilinear form; Keyless Attention at m=3 realizes a depth-3 instance by replacing the key projection with a value-space routing matrix that maintains the same total projection count and produces a Value-Only Cache.

What carries the argument

The value-space routing matrix that substitutes for the key projection and couples routing to retrieval in the depth-m attention factorization.

If this is right

  • KV cache memory and bandwidth are reduced by exactly 50 percent at inference time.
  • Decode throughput stays the same or increases because of lower cache access cost.
  • The same projection matrix budget as standard attention is used, so no extra parameters are introduced.
  • The method applies across GPT-2, Pythia, Qwen2, and Llama architectures without architecture-specific changes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Key projections may be redundant when routing information can be learned directly in value space.
  • Memory-limited devices could support longer contexts by halving the cache footprint per layer.
  • Other members of the depth-m factorization family might trade different numbers of projections for further efficiency gains.

Load-bearing premise

A learned value-space routing matrix can fully replace the key projection while preserving model capacity without extra parameters or changed training dynamics.

What would settle it

Train a Keyless Attention model from scratch on the same data and architecture as a standard QKV model and measure whether perplexity rises or downstream zero-shot scores fall below the baseline.

Figures

Figures reproduced from arXiv: 2606.21848 by Xin Gao.

Figure 1
Figure 1. Figure 1: Training dynamics of QKV and QVV(3) across GPT-2 model depths. Top row: 12-layer (280M parameters); [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation loss comparison of QVV and QKV variants across factorization depths [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Perplexity over 4 epochs across three GQA architectures. Keyless Attention matches or outperforms standard [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Decode throughput and KV cache size for Keyless vs. QKV attention under GQA. Left: decode throughput (mean ˘ std over 3 seeds) as a function of context length; Keyless exceeds QKV. Right: KV cache size vs Value-Only Cache size; Keyless reduces cache memory by exactly 50% at every context length, since it stores only value states. parameters induced by value-space routing, though we leave a direct mechanist… view at source ↗
read the original abstract

We propose Keyless Attention, an attention mechanism that eliminates the key projection entirely, operating over queries and values only. This yields a Value-Only Cache that reduces KV cache memory and access overhead by exactly 50% over standard attention, while matching or exceeding standard attention's decode throughput. Beyond efficiency, we introduce Depth-$m$ Attention Factorization: standard attention computes a depth-2 factorization of the attention bilinear form, while Keyless Attention realizes a depth-$m$ instance of this family. At m=3, Keyless Attention matches the projection matrix count of standard attention via a value-space routing matrix that replaces the key projection and introduces a coupling between routing and retrieval. Experiments across five models and four architectures (GPT-2 280M, GPT-2 557M, Pythia 410M, Qwen2 1.5B, and Llama 3.2 1B) show that Keyless Attention matches or outperforms standard QKV attention on perplexity in 4 out of 5 models. On downstream zero-shot evaluation (GPT-2 557M), Keyless Attention outperforms on 4 out of 5 commonsense reasoning benchmarks, while achieving 50% KV cache reduction throughout.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes Keyless Attention, which removes the key projection from standard QKV attention and operates only on queries and values. It introduces Depth-m Attention Factorization, realizing a depth-3 instance via a value-space routing matrix that replaces the key projection while matching its parameter count. This enables a Value-Only Cache that reduces KV cache memory and access by exactly 50%. Experiments across GPT-2 (280M, 557M), Pythia 410M, Qwen2 1.5B, and Llama 3.2 1B report that Keyless Attention matches or exceeds standard attention perplexity in 4/5 models and outperforms on 4/5 downstream zero-shot commonsense benchmarks for GPT-2 557M.

Significance. If the value-space routing substitution preserves model capacity at matched parameter count and training dynamics, the 50% KV cache reduction would be a substantial efficiency gain for transformer inference, particularly at long contexts, without throughput loss. The depth-m factorization framing offers a conceptual generalization of attention bilinear forms that could inspire further variants.

major comments (2)
  1. [Abstract, §3] Abstract and §3 (implied by description of m=3 factorization): the central claim that the value-space routing matrix fully substitutes the removed key projection at equal parameter count is load-bearing for both the efficiency and performance results, yet no derivation or explicit matrix-dimension comparison is provided to confirm the substitution does not reduce expressivity or require compensatory changes to training.
  2. [Experiments] Experiments (across five models): the reported perplexity matching in 4/5 models and downstream gains lack any mention of hyperparameter matching, training procedure details, statistical tests, or ablations isolating the routing matrix effect, making it impossible to verify whether the substitution succeeds or whether unstated adjustments enabled the outcomes.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for highlighting the potential significance of the 50% KV cache reduction. We respond to each major comment below, indicating where revisions will be made.

read point-by-point responses

  1. Referee: [Abstract, §3] Abstract and §3 (implied by description of m=3 factorization): the central claim that the value-space routing matrix fully substitutes the removed key projection at equal parameter count is load-bearing for both the efficiency and performance results, yet no derivation or explicit matrix-dimension comparison is provided to confirm the substitution does not reduce expressivity or require compensatory changes to training.

    Authors: We agree an explicit derivation strengthens the presentation. Section 3 already frames Keyless Attention as a depth-3 factorization where the value-space routing matrix (d_model × d_model) replaces the key projection while preserving total projection parameters; the coupling term does not alter the count. In revision we will add a dedicated matrix-dimension paragraph and a short proof sketch confirming equal parameter count and that no training adjustments were introduced. revision: yes

  2. Referee: [Experiments] Experiments (across five models): the reported perplexity matching in 4/5 models and downstream gains lack any mention of hyperparameter matching, training procedure details, statistical tests, or ablations isolating the routing matrix effect, making it impossible to verify whether the substitution succeeds or whether unstated adjustments enabled the outcomes.

    Authors: All models were trained with the identical hyperparameter sets and optimization schedules used for the corresponding standard-attention baselines (taken directly from the original GPT-2, Pythia, Qwen2, and Llama recipes). We will insert an explicit statement of this matching and a short ablation table isolating the routing matrix. Statistical tests were omitted because results are reported across five architecturally diverse models; we can add per-run variance if space permits. revision: partial

Circularity Check

0 steps flagged

No circularity in derivation or claims

full rationale

The paper proposes Keyless Attention as a new mechanism eliminating the key projection and introduces Depth-m Attention Factorization as a conceptual framing, then reports direct empirical results on perplexity and downstream tasks across multiple models. No equations, predictions, or first-principles derivations are presented that reduce by construction to fitted inputs, self-citations, or renamed known results. The performance claims rest on experimental comparisons rather than any load-bearing self-referential step. This is the normal case of an empirical architecture paper whose central claims remain independently falsifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Abstract-only review; full derivation of depth-m factorization and training details unavailable. The routing matrix is introduced without independent evidence of its necessity beyond the reported experiments.

axioms (1)
  • domain assumption Standard attention admits a depth-m factorization family of which depth-2 is the conventional QKV form
    Stated in abstract as the mathematical framing for Keyless Attention at m=3.
invented entities (1)
  • Value-space routing matrix no independent evidence
    purpose: Replaces key projection and couples routing with value retrieval
    New component required to enable keyless operation; no external falsifiable prediction supplied in abstract.

pith-pipeline@v0.9.1-grok · 5744 in / 1304 out tokens · 24932 ms · 2026-06-26T12:18:38.453771+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 18 canonical work pages · 9 internal anchors

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Proteins: Structure, Function, and Bioinformatics , year =

    Klausen, Michael Schantz and Jespersen, Martin Closter and Nielsen, Henrik and Jensen, Kamilla Kjaergaard and Jurtz, Vanessa Isabell and Soenderby, Casper Kaae and Sommer, Morten Otto Alexander and Winther, Ole and Nielsen, Morten and Petersen, Bent and others , title =. Proteins: Structure, Function, and Bioinformatics , year =

  9. [9]

    Proteins: Structure, Function, and Bioinformatics , year =

    Moult, John and Fidelis, Krzysztof and Kryshtafovych, Andriy and Schwede, Torsten and Tramontano, Anna , title =. Proteins: Structure, Function, and Bioinformatics , year =. doi:10.1002/prot.25415 , issn =

    work page doi:10.1002/prot.25415

  10. [10]

    and Bolotin, Dmitry A

    Sarkisyan, Karen S. and Bolotin, Dmitry A. and Meer, Margarita V. and Usmanova, Dinara R. and Mishin, Alexander S. and Sharonov, George V. and Ivankov, Dmitry N. and Bozhanova, Nina G. and Baranov, Mikhail S. and Soylemez, Onuralp and others , title =. Nature , volume =. 2016 , publisher =

    2016

  11. [11]

    and Chidyausiku, Tamuka M

    Rocklin, Gabriel J. and Chidyausiku, Tamuka M. and Goreshnik, Inna and Ford, Alex and Houliston, Scott and Lemak, Alexander and Carter, Lauren and Ravichandran, Rashmi and Mulligan, Vikram K. and Chevalier, Aaron and others , title =. Science , volume =. 2017 , publisher =

    2017

  12. [12]

    2019 , volume =

    UniProt: a worldwide hub of protein knowledge , journal =. 2019 , volume =

    2019

  13. [13]

    1984 , volume =

    Nomenclature and symbolism for amino acids and peptides (Recommendations 1983) , journal =. 1984 , volume =

    1983

  14. [14]

    and Barton, Geoffrey J

    Cuff, James A. and Barton, Geoffrey J. , title =. Proteins: Structure, Function, and Bioinformatics , year =

  15. [15]

    Briefings in Bioinformatics , year =

    Yang, Yuedong and Gao, Jianzhao and Wang, Jihua and Heffernan, Rhys and Hanson, Jack and Paliwal, Kuldip and Zhou, Yaoqi , title =. Briefings in Bioinformatics , year =

  16. [16]

    The catalytic triad of serine peptidases , journal =

    Polg. The catalytic triad of serine peptidases , journal =. 2005 , volume =

    2005

  17. [17]

    and Tokuriki, Nobuhiko , title =

    Miton, Christina M. and Tokuriki, Nobuhiko , title =. Protein Science , year =

  18. [18]

    , title =

    Halabi, Nadav and et al. , title =. Cell , year =

  19. [19]

    Physical Review Letters , year =

    Rivoire, Olivier , title =. Physical Review Letters , year =

  20. [20]

    , title =

    McGill, William J. , title =. Psychometrika , year =

  21. [21]

    , title =

    Horovitz, Amnon and Fersht, Alan R. , title =. Journal of Molecular Biology , volume =

  22. [22]

    , title =

    Wells, James A. , title =. Biochemistry , volume =

  23. [23]

    and Krishna, Vijay and Ranganathan, Rama , title =

    Poelwijk, Frank J. and Krishna, Vijay and Ranganathan, Rama , title =. PLOS Computational Biology , volume =. 2016 , doi =

    2016

  24. [24]

    and Socolich, Michael and Ranganathan, Rama , title =

    Poelwijk, Frank J. and Socolich, Michael and Ranganathan, Rama , title =. Nature Communications , volume =. 2019 , doi =

    2019

  25. [25]

    and Lan, Y

    Weinreich, Daniel M. and Lan, Y. and Wylie, Christopher S. and Heckendorn, Robert B. , title =. Current Opinion in Genetics & Development , volume =

  26. [26]

    and Harms, Michael J

    Sailer, Zachary R. and Harms, Michael J. , title =. Genetics , volume =. 2017 , publisher =

    2017

  27. [27]

    and Harms, Michael J

    Sailer, Zachary R. and Harms, Michael J. , title =. PLOS Computational Biology , volume =. 2017 , doi =

    2017

  28. [28]

    and Dai, Liang and Olson, Catherine A

    Wu, Nicholas C. and Dai, Liang and Olson, Catherine A. and Lloyd-Smith, James O. and Sun, Ren , title =. eLife , volume =. 2016 , doi =

    2016

  29. [29]

    , title =

    Kauffman, Stuart A. , title =

  30. [30]

    Gomez and Lukasz Kaiser and Illia Polosukhin , title =

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , title =. Advances in Neural Information Processing Systems , year =

  31. [31]

    International Conference on Learning Representations , year =

    Clift, James and Murfet, Daniel , title =. International Conference on Learning Representations , year =

  32. [32]

    arXiv preprint arXiv:2306.02896 , year=

    Sanford, Clayton and Hsu, Daniel and Telgarsky, Matus , title =. arXiv preprint arXiv:2306.02896 , year =

    work page arXiv

  33. [33]

    Advances in Neural Information Processing Systems , year =

    Sanford, Clayton and Hsu, Daniel and Telgarsky, Matus , title =. Advances in Neural Information Processing Systems , year =

  34. [34]

    arXiv preprint , year =

    Kozachinskiy, Alexander and others , title =. arXiv preprint , year =

  35. [35]

    arXiv preprint arXiv:2602.02422 , year =

    Chakrabarti, Sayak and Pitassi, Toniann and Alman, Josh , title =. arXiv preprint arXiv:2602.02422 , year =

    work page arXiv

  36. [36]

    Bai, Song and Zhang, Feihu and Torr, Philip H. S. , title =. Pattern Recognition , volume =. 2021 , publisher =

    2021

  37. [37]

    Longformer: The Long-Document Transformer

    Beltagy, Iz and Peters, Matthew E. and Cohan, Arman , title =. arXiv preprint arXiv:2004.05150 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  38. [38]

    Advances in Neural Information Processing Systems , year =

    Zaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon, Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others , title =. Advances in Neural Information Processing Systems , year =

  39. [39]

    Reformer: The Efficient Transformer , booktitle =

    Kitaev, Nikita and Kaiser,. Reformer: The Efficient Transformer , booktitle =

  40. [40]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , booktitle =

    Dao, Tri and Fu, Dan and Ermon, Stefano and Rudra, Atri and R. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , booktitle =

  41. [41]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and others , title =. arXiv preprint arXiv:2010.11929 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  42. [42]

    AAAI Conference on Artificial Intelligence , year =

    Sekhon, Arshdeep and others , title =. AAAI Conference on Artificial Intelligence , year =

  43. [43]

    International Conference on Learning Representations (ICLR) , year=

    Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling , author=. International Conference on Learning Representations (ICLR) , year=

  44. [44]

    Findings of the Association for Computational Linguistics , year=

    Blockwise Self-Attention for Long Document Understanding , author=. Findings of the Association for Computational Linguistics , year=

  45. [45]

    Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

    Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention , author=. arXiv preprint arXiv:2502.11089 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  46. [46]

    Generating Long Sequences with Sparse Transformers

    Generating Long Sequences with Sparse Transformers , author=. arXiv preprint arXiv:1904.10509 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  47. [47]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. Proceedings of the 37th International Conference on Machine Learning (ICML) , pages =. 2020 , volume =

    2020

  48. [48]

    International Conference on Learning Representations (ICLR) , year =

    Rethinking Attention with Performers , author =. International Conference on Learning Representations (ICLR) , year =

  49. [49]

    Linformer: Self-Attention with Linear Complexity

    Linformer: Self-Attention with Linear Complexity , author =. arXiv preprint arXiv:2006.04768 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  50. [50]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  51. [51]

    and Zhang, Hao and Stoica, Ion , booktitle =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with. 2023 , url =

    2023

  52. [52]

    2024 , eprint =

    Prabhu, Ramya and Nayak, Ajay and Mohan, Jayashree and Ramjee, Ramachandran and Panwar, Ashish , journal =. 2024 , eprint =

    2024

  53. [53]

    Zipage: Maintain High Request Concurrency for

    Liao, Mengqi and Wang, Lu and Zhang, Chaoyun and Qiao, Bo and Qin, Si and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Wan, Huaiyu , journal =. Zipage: Maintain High Request Concurrency for. 2026 , eprint =

    2026

  54. [54]

    2022 , url =

    Dettmers, Tim and Lewis, Mike and Belkada, Younes and Zettlemoyer, Luke , booktitle =. 2022 , url =

    2022

  55. [55]

    2023 , url =

    Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan , booktitle =. 2023 , url =

    2023

  56. [56]

    arXiv preprint arXiv:2412.08585 , eprint =

    Kang, Hao and Bharadwaj, Srikant and Hensman, James and Krishna, Tushar and Ruhle, Victor and Rajmohan, Saravan , year =. arXiv preprint arXiv:2412.08585 , eprint =

    work page arXiv

  57. [57]

    Fast Transformer Decoding: One Write-Head is All You Need

    Fast Transformer Decoding: One Write-Head is All You Need , author =. arXiv preprint arXiv:1911.02150 , year =. 1911.02150 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  58. [58]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =. 2023 , address =

    2023

  59. [59]

    2024 , eprint =

    Tang, Hanlin and Lin, Yang and Lin, Jing and Han, Qingsen and Hong, Shikuan and Yao, Yiwu and Wang, Gongyi , journal =. 2024 , eprint =

    2024

  60. [60]

    2025 , journal =

    Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? , author=. 2025 , journal =. 2502.11501 , archivePrefix=

    work page arXiv 2025

  61. [61]

    Xu, Yuhui and Jie, Zhanming and Dong, Hanze and Wang, Lei and Lu, Xudong and Zhou, Aojun and Saha, Amrita and Xiong, Caiming and Sahoo, Doyen , booktitle =. Thin

  62. [62]

    arXiv preprint arXiv:2512.17452 , year=

    KV Admission: Learning What to Write for Efficient Long-Context Inference , author=. arXiv preprint arXiv:2512.17452 , year=

    work page arXiv

  63. [63]

    Proceedings of Machine Learning and Systems , volume =

    Efficiently Scaling Transformer Inference , author =. Proceedings of Machine Learning and Systems , volume =

  64. [64]

    PaLM: Scaling Language Modeling with Pathways

    PaLM: Scaling Language Modeling with Pathways , author=. 2022 , journal =. 2204.02311 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  65. [65]

    DeepSeek-AI , journal =

  66. [66]

    Advances in Neural Information Processing Systems , volume =

    Implicit Regularization in Deep Matrix Factorization , author =. Advances in Neural Information Processing Systems , volume =

  67. [67]

    Advances in Neural Information Processing Systems , volume =

    Matrix Completion Has No Spurious Local Minimum , author =. Advances in Neural Information Processing Systems , volume =

  68. [68]

    Proceedings of The 28th Conference on Learning Theory , pages =

    Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition , author =. Proceedings of The 28th Conference on Learning Theory , pages =

  69. [69]

    Advances in Neural Information Processing Systems , volume =

    Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced , author =. Advances in Neural Information Processing Systems , volume =

  70. [70]

    International Conference on Learning Representations , year =

    Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks , author =. International Conference on Learning Representations , year =

  71. [71]

    Neural Networks , volume =

    High-Dimensional Dynamics of Generalization Error in Neural Networks , author =. Neural Networks , volume =

  72. [72]

    2025 , journal =

    EcoTransformer: Attention without Multiplication , author=. 2025 , journal =. 2507.20096 , archivePrefix=

    work page arXiv 2025

  73. [73]

    Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages =

    Effective Approaches to Attention-based Neural Machine Translation , author =. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages =. 2015 , publisher =

    2015

  74. [74]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year =

    HellaSwag: Can a Machine Really Finish Your Sentence? , author =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year =

  75. [75]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv:1803.05457v1 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  76. [76]

    Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , month =

    Mostafazadeh, Nasrin and Chambers, Nathanael and He, Xiaodong and Parikh, Devi and Batra, Dhruv and Vanderwende, Lucy and Kohli, Pushmeet and Allen, James , title =. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , month =. 2016 , address =

    2016

  77. [77]

    Proceedings of the 3rd Workshop on Noisy User-generated Text , year =

    Crowdsourcing Multiple Choice Science Questions , author =. Proceedings of the 3rd Workshop on Noisy User-generated Text , year =

  78. [78]

    NAACL , year =

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , author =. NAACL , year =

  79. [79]

    OpenAI Blog , volume =

    Language Models are Unsupervised Multitask Learners , author =. OpenAI Blog , volume =

  80. [80]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , year =

    Transformers: State-of-the-Art Natural Language Processing , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , year =

    2020

Showing first 80 references.