Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

KV Shifting Attention Enhances Language Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that shifting keys and values by one position per head lets one attention layer implement induction heads exactly, a task standard attention needs two layers and twice the width to approximate.

desk verdict Cheap, well-motivated attention modification with clean toy evidence, but Theorem 2's exact equality is wrong and the large-scale comparison is confounded; needs major revision before it's citable. read the letter →

arxiv 2411.19574 v2 pith:4GXF6KPD submitted 2024-11-29 cs.CL

Mingyu Xu , Wei Cheng , Bingning Wang , Weipeng Chen This is my paper · ORCID
classification cs.CL MSC 68T0768T50
keywords KVshiftingattentioninductionheadsin-contextlearningtransformerdepthwidthlanguagemodelpretrainingAlibipositionalbiasmechanism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a small, cheap change to attention—shifting each token's key and value by one position via four learnable scalars per head—removes the usual depth and width requirements for induction heads (the circuit that predicts a token by finding its earlier occurrence and copying what followed), so a single attention layer can implement the mechanism exactly. If the claim is right, transformers can learn in-context patterns faster and reach better language-modeling performance with the same compute. The paper supports this with a theoretical construction, toy induction experiments, and pretraining runs up to 19B parameters.

What carries the argument

The central object is the KV shifting operation: before attention, the key and value are recomputed as $\alpha_1 K + \alpha_2 \operatorname{Shift}(K)$ and $\beta_1 V + \beta_2 \operatorname{Shift}(V)$, where Shift discards the last token and pads a zero at the front, and $\alpha_1, \alpha_2, \beta_1, \beta_2$ are four learnable scalars per head. This lets the current token attend to the key of token $i$ but read the value of token $i-1$, which is exactly the copy step an induction head needs, so the usual second layer that copies the previous token's hidden state is no longer required.

What would settle it

Train a one-layer KV shifting transformer on synthetic induction data under RoPE with learned projections, and check whether the learned attention pattern reproduces the paper's exact construction ($\alpha_1=0$, $\alpha_2=1$, $\beta_1=1/\sigma$, $\beta_2=0$). If a RoPE model cannot achieve the same near-zero induction error that the Alibi construction achieves, the theoretical support for the tested architecture fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a one-layer, single-head transformer using KV shifting attention—where each token's key and value are each a learned mix of that token's and the previous token's key and value—can implement the induction heads mechanism exactly, with no approximation error and no wider hidden dimension. Standard attention provably needs two layers and twice the width for the same task, and even then only approximates it. The paper proves this equality in the Alibi relative-position-bias setting and demonstrates experimentally that the modification learns induction faster in toy models and improves pretraining loss and benchmarks from 1.5B to 19B parameters.

Load-bearing premise

The load-bearing premise is that the exact one-layer construction proven for Alibi position bias and identity projections transfers to the RoPE positional encoding with learned projections used in all the large-scale experiments; the paper does not prove that transfer.

Editorial extensions

If this is right

  • A one-layer transformer with KV shifting can, in principle, perform induction heads exactly, so the standard claim that induction requires at least two attention layers no longer applies to this architecture.
  • Pretraining with KV shifting reaches lower loss or the same loss in fewer tokens across model sizes from 1.5B to 19B parameters, and it stays stable at learning rates where vanilla attention diverges.
  • The modification is cheap—four scalars per head and $O(ND)$ extra compute per layer—so it can be dropped into existing LLM training and inference code with minimal overhead.
  • The ablation shows both the K-shift and the V-shift are necessary; shifting only one of them loses most of the benefit, and shifting over a longer window does not help.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the theory is proven with Alibi position bias and identity projections, while every large-scale experiment uses RoPE and learned projections; whether the exact one-layer construction survives under RoPE is an unstated transfer assumption that a direct experiment could settle.
  • Beyond the paper: the appendix results on multi-hop tasks and grade-school math suggest the benefit may extend beyond induction to multi-step reasoning, but the paper does not claim that as a main result.
  • Beyond the paper: if the convergence speedup persists at larger scale and with open datasets, then induction heads are a real pretraining bottleneck and targeted inductive biases of this kind could be an alternative to adding more layers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes KV shifting attention, a modification of the standard attention mechanism in which the keys and values are formed as learned combinations of the current token's K/V and a one-position-shifted version (Eqs. 4-6). The authors claim that this modification reduces the depth and width required for a transformer to implement induction heads, prove a representational theorem (Theorem 2) and a toy learning analysis (Theorem 3), and report experiments ranging from toy induction tasks to pretraining runs at 1.5B, 2.9B, 6.7B, 13B, and 19B parameters. The central empirical claim is that KV shifting attention yields faster convergence and better language modeling performance than vanilla attention at these scales.

Significance. If the claims were established, the paper would offer a strikingly cheap architectural change: four learnable scalars per head plus a shift operation, improving induction-head formation and language modeling at scale. The toy induction experiments are clear and the paper releases code and model checkpoints, which is a genuine strength. However, the main theoretical result is not proven as stated, the headline large-scale comparison is not controlled, and the theory is developed under assumptions that do not match the experiments. The idea is promising, but the current evidence is not sufficient to support the abstract's claims.

major comments (4)
  1. [§3.1, Appendix B] Theorem 2's claimed exact equality IH = KVSA is not established. Under the construction α1=0, α2=1, β1=1/σ, β2=0 with identity projections, the last-token output of KV shifting attention is sum_{s=1}^{L} softmax(x_L^T x_{s-1}/σ - m|L-s|)(x_s/σ), whereas Definition 1 restricts the sum to s=2,...,L-1 and normalizes only over those positions. The boundary terms s=1 and s=L are generically nonzero, and because the normalization sets differ, the weights on the common terms also change. For example, with L=3 and x1=A, x2=B, x3=A, Definition 1 gives exactly B, while the constructed KVSA outputs a mixture of A and B. A mask excluding positions 1 and L would be needed to make the equality hold, but that mask is not part of Eq. (5)-(6) or the released implementation in Appendix F. The central claim that KV shifting attention reduces depth and width requirements for induction heads is therefore not proven by this construction.
  2. [§4.1, Table 6] The main large-scale comparison is confounded or at least described inconsistently. The text says the production baseline uses GQA with a 48,000-token vocabulary, while the KV shifting experiment is described as using Llama's 36,000-token vocabulary with standard MHA; Table 6, however, lists both the 2.9B and 19B models with vocab size 48,000 and 4 KV heads. If the runs differ in attention type and vocabulary, the improvements in Table 2 cannot be attributed to KV shifting. If they do not differ, the text needs to be corrected. In either case, the manuscript does not provide a controlled comparison at the scales used for the headline result, and the 2.9B and 19B results are single runs.
  3. [§3.1, §4.1, Appendix F] The theoretical results are stated for Alibi relative position encoding and, in Theorem 3, for identity projections with residual connections, MLPs, and position embeddings removed, but all experiments, including the toy induction experiments, use Llama-style RoPE with learned projections. No argument is given that the Alibi-based construction transfers to RoPE or to learned Wq, Wk, Wv, Wo. Consequently, the theorems do not directly support the architecture actually tested, and the claimed 'bias towards learning induction' remains an empirical observation without the stated theoretical backing.
  4. [§3.2, Theorem 3] Theorem 3 computes a loss expression in a d→∞ limit under highly simplified conditions; it does not prove convergence speed or show that gradient descent reaches the induction-head configuration (α1,β1)=(0,1). The surrounding text interprets Figure 2 as evidence about the dynamics of learning induction heads, but the theorem provides only a static contour of the loss, not a dynamical statement. This is not a load-bearing proof of faster convergence, despite the abstract's wording.
minor comments (6)
  1. [Definition 1] The notation softmax(...)_i in Eq. (7) has no defined index i, and the sentence 'when T > 0 and m > 0, which infinitely approach 0' is unclear; presumably a limit is intended, but it is never formalized.
  2. [§2.2, Eq. (6)] The causal mask M is said to be in R^{D×D}, but it should be in R^{N×N}; the scaling by 1/σ inside the softmax is also written ambiguously.
  3. [Various] There are numerous typos and inconsistencies, including 'KV shfting', 'KV SHITING', 'Vallina', 'Lambda' for Lambada, 'decent derection', 'β = 0.95' for β2, and '∑i αi and ∑i αi' in §4.6.
  4. [Appendix C] The proof of Theorem 3 contains garbled O(·) placement and undefined intermediate expressions, which makes Eq. (10) difficult to verify; it should be rewritten with consistent notation and a clear statement of which terms are kept in the d→∞ limit.
  5. [Table 2, Table 3] The benchmark tables contain header typos such as 'A RC-C' and report no variance or significance measures, so the claimed improvements, especially the smaller 19B differences, should be interpreted cautiously.
  6. [Footnote 3] The footnote admits that a lower-bound comparison for Theorem 1 versus Theorem 2 is missing; this is important because the 'reduced width' claim is comparative, and the limitation should be discussed in the main text rather than relegated to a footnote.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 2 is a constructive existence result, and the language-modeling claims are independently benchmarked against vanilla transformers.

full rationale

KV shifting attention is defined independently in Eqs. (4)-(6), and the induction-head target IH is given by an explicit formalization in Definition 1, not as a restatement of KVSA. Theorem 2 is a constructive existence proof: setting alpha1=0, alpha2=1, beta1=1/sigma, beta2=0 and identity projections makes shifted-key attention reproduce the IH sum. This is a representation theorem rather than a circular definition, because the architecture class and the target function are specified separately and the equality is derived (even if the proof is one line). The central empirical claim, that KV shifting attention improves induction learning and language modeling, is tested against vanilla baselines at multiple scales and is independent of the theorem. The only self-citation (Xu et al. 2024) concerns RoPE base selection in Appendix G and is not load-bearing. The boundary mismatch between the KVSA sum over s=1..L and Definition 1's sum over s=2..L-1 is a correctness concern, not circularity. No prediction or derived result reduces by construction to its own input.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on the transfer of an Alibi-based representational result to RoPE-based models and on a simplified learning analysis. No external fitted constants are introduced; the only tunable quantities are the architectural per-head scalars.

assumptions (4)
  • domain assumption Theoretical results are stated for infinite-precision transformers.
    Section 3.1 states 'We use infinite precision transformers in this article.' This ignores numerical precision effects.
  • domain assumption Alibi RPE can be substituted for the positional encoding used in experiments (RoPE) without loss of validity.
    Definition 1 and Theorem 2 use Alibi bias; experiments use Llama/RoPE (Table 6). The paper does not prove the transfer.
  • domain assumption The simplified learning model with Wq=Wk=Wv=Wo=I and d approaching infinity captures the behavior of real trained transformers.
    Theorem 3 is derived under these conditions (Appendix C); the paper asserts this reflects practice but provides only contour plots.
  • domain assumption The induction-head machine IH with Alibi RPE (Definition 1) adequately formalizes the induction heads mechanism described in prior work.
    The paper uses its own definition and does not prove equivalence to the original induction-head circuit (Elhage et al. 2021; Olsson et al. 2022).

how reviews work

0 comments
Cite this review

Pith. "Pith review of KV Shifting Attention Enhances Language Modeling." pith.science (2026). https://pith.science/paper/4GXF6KPD

@misc{pith2026241119574,
  author       = {Pith},
  title        = {Pith review of: KV Shifting Attention Enhances Language Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GXF6KPD}},
  note         = {Machine review of arXiv:2411.19574}
}
read the original abstract

The current large language models are mainly based on decode-only structure transformers, which have great in-context learning (ICL) capabilities. It is generally believed that the important foundation of its ICL capability is the induction heads mechanism, which requires at least two layers attention. In order to more efficiently implement the ability of the model's induction, we revisit the induction heads mechanism and proposed a KV shifting attention. We theoretically prove that the KV shifting attention reducing the model's requirements for the depth and width of the induction heads mechanism. Our experimental results demonstrate that KV shifting attention is beneficial to learning induction heads and language modeling, which lead to better performance or faster convergence from toy models to the pre-training models with more than 10 B parameters.

Figures

Figures reproduced from arXiv: 2411.19574 by the authors.

Figure 1
Figure 1. On the left, as the training step size increases, the accuracy of induction varies [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Contour lines and gradient decent derection of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of learning 3-gram text using models of different sizes. In this experi [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Training loss curve. We train 2.9B model with 500B tokens, and 19B models with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Training loss of 1.5B parameters model among random seeds and learning rate [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Training loss comparison between different size. All models are trained on 10B [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Validation loss across different size and training tokens. For scaling law, while [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Further experiments are conducted on a 1.5B model, where we trained 10B tokens. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Error Rates under Hop k Tasks. The smaller the error, the better [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: QKV shifting attention vs KV shifting attention in model with 19B parameters. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A gated Shift-FFN adapter that adds the previous token's representation to the current token's before the feedforward layer reduces repetitive looping and improves math accuracy in LoRA fine-tuned models trained on lo...

  2. Understanding Transformer from the Perspective of Associative Memory

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Frames the Transformer as associative memory, derives retrieval SNR for linear, softmax, ReLU, and SoLU kernels, and proposes DeltaFormer, a softmax-plus-delta-rule variant claimed to exceed TC0 expressivity.

Reference graph

Works this paper leans on

29 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [4]

    Unveiling induction heads: Provable training dynamics and feature learning in transformers

    Chen, S., Sheen, H., Wang, T., and Yang, Z. Unveiling induction heads: Provable training dynamics and feature learning in transformers. arXiv preprint arXiv:2409.10559,

    arXiv

  3. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P ., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  4. [6]

    and Shutova, E

    Crosbie, J. and Shutova, E. Induction heads as an essential mechanism for pattern matching in in-context learning. arXiv preprint arXiv:2407.07011,

    arXiv

  5. [7]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  6. [9]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  7. [10]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math data...

    arXiv

  8. [11]

    M., et al

    14 Jelassi, S., Brandfonbrener, D., Kakade, S. M., et al. Repeat after me: Transformers are better than state space models at copying. In Forty-first International Conference on Machine Learning. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mode...

    arXiv 2001

Show all 29 references
  1. [13]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., and Baldwin, T. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023a. Li, Y., Wu, Y., Li, J., and Liu, S. Accelerating transducers through adjacent toke...

    2023 arXiv

  2. [15]

    Rwkv: Reinventing rnns for the transformer era

    Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

    arXiv

  3. [16]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are uns...

    arXiv

  4. [17]

    Identifying semantic induction heads to understand in-context learning

    Ren, J., Guo, Q., Yan, H., Liu, D., Zhang, Q., Qiu, X., and Lin, D. Identifying semantic induction heads to understand in-context learning. arXiv preprint arXiv:2402.13055,

    arXiv

  5. [18]

    One-layer transformers fail to solve the induction heads task

    Sanford, C., Hsu, D., and Telgarsky, M. One-layer transformers fail to solve the induction heads task. arXiv preprint arXiv:2408.14332, 2024a. 15 Sanford, C., Hsu, D., and Telgarsky, M. Transformers, parallel computation, and logarithmic depth. In Forty-first International Con...

    arXiv

  6. [19]

    Retentive network: A successor to transformer for large language models

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621,

    arXiv

  7. [20]

    Lstm neural networks for language modeling

    Sundermeyer, M., Schlüter, R., and Ney, H. Lstm neural networks for language modeling. In Interspeech, volume 2012, pp. 194–197,

    2012

  8. [23]

    R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J

    Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations. Wang, M., Yu, R., Wu, L., et al. How trans...

    arXiv

  9. [24]

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al

    URL https://openreview.net/forum?id=EiIelh2t7S. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671,

    arXiv

  10. [25]

    Differential transformer

    Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential transformer. arXiv preprint arXiv:2410.05258, 2024a. Ye, T., Xu, Z., Li, Y., and Allen-Zhu, Z. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv prepr...

    arXiv

  11. [26]

    u nsq ue ez e (0)

    # ( bs , seq +w -1 , h , d , w ) K _ e x p a n d e d = K . u nsq ue ez e (0) . u ns qu ee ze (0) . un sq ue eze ( -2) # (1 , 1 , h , 1 , w ) V _ u n f o l d e d = U _ u n f o l d e d * K _ e x p a n d e d # ( bs , seq , h , d , w ) V = V _ u n f o l d e d . sum ( dim = -1) # (...

    2024

  12. [27]

    Therefore, a relatively large value has been uniformly set here

    has shown that the longer the context length, the larger the base required, while the default base=10,000 is relatively small, even for 2048 windows. Therefore, a relatively large value has been uniformly set here. Table 6: Configuration. PARAMETERS 1.5B 2.9B 6.7B 13B 19B HIDD...

    2024

  13. [28]

    Therefore, we present the experimental results of mathematical ability in the next section

    This powerful ability to perform implicit reasoning implies that KV shifting attention may achieve better results in mathematical or reasoning abilities. Therefore, we present the experimental results of mathematical ability in the next section. 23 I Grade-School Math As a mor...

    2024

  14. [29]

    after Add

    It can be seen that the performance of shifting QKV is not as good as shifting KV , and even not as good as the vanilla. From the perspective of induction heads, the shifting of Q is difficult to contribute to the formation of the induction heads mechanism. Finally, I will ask...

    2021

  15. [2008]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P ., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P ., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  16. [2017]

    An empirical study of mamba-based language models

    Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V ., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887,

    arXiv

  17. [2020]

    Selective attention improves transformer.arXiv preprint arXiv:2410.02703,

    Leviathan, Y., Kalman, M., and Matias, Y. Selective attention improves transformer.arXiv preprint arXiv:2410.02703,

    arXiv

  18. [2021]

    Friedman, D., Wettig, A., and Chen, D

    https://transformer- circuits.pub/2021/framework/index.html. Friedman, D., Wettig, A., and Chen, D. Learning transformer programs. Advances in Neural Information Processing Systems, 36,

    2021

  19. [2022]

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R

    https://transformer-circuits.pub/2022/in- context-learning-and-induction-heads/index.html. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad disc...

    2022

  20. [2023]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901,

    2023

  21. [2024]

    and Yu, H

    Alman, J. and Yu, H. Fundamental limitations on subquadratic alternatives to transformers. arXiv preprint arXiv:2410.04271,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.