Pith. sign in

REVIEW 3 major objections 4 minor 30 references

Selective Induction Heads: How Transformers Select Causal Structures In Context

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-layer attention-only transformer can select the correct causal lag in context and converge to maximum-likelihood selection.

desk verdict The selective induction head construction and the interleaved-Markov-chain task are real contributions, but the paper's headline proof of asymptotic MLE equivalence rests on an unproved inequality (Claim 1), so the abstract overstates what is established. read the letter →

arxiv 2509.08184 v1 pith:KZRV7XD2 submitted 2025-09-09 cs.LG stat.ME

Francesco D'Angelo , Francesco Croce , Nicolas Flammarion This is my paper
classification cs.LGstat.ME
keywords interleavedMarkovchainsselectiveinductionheadsin-contextlearningcausalstructureselectionlagattention-onlytransformersmaximumlikelihoodmechanisticinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a transformer can choose, purely from context, which past token actually predicts the next one. It studies sequences generated by interleaved Markov chains that share one transition matrix but differ in their lag, so the model must identify the true causal structure before predicting. The authors construct a three-layer attention-only transformer that does this: the first layer extracts normalized transition probabilities for each candidate lag, the second aggregates those probabilities from disjoint chunks of history, and the third copies the token whose lag accumulated the most evidence. They argue that this 'selective induction head' converges asymptotically to maximum-likelihood selection, and report that transformers trained from scratch match the construction. If correct, the paper gives a concrete mechanism for in-context model selection rather than fixed-rule copying.

What carries the argument

The mechanism is the selective induction head, an attention circuit whose copy source is chosen by an evidence score rather than by a fixed token match. The score at position i for lag k is the normalized transition probability p̃ᵢ,ₖ = Xᵢ₋ₖᵀ P* Xᵢ / Σₗ Xᵢ₋ₗᵀ P* Xᵢ; the head watches cumulative sums over i. The construction uses three layers: layer 1 puts log P* and a lag-selective position mask in the attention matrix so each token stores normalized transitions for every lag; layer 2 uses K shifted diagonal attention heads to aggregate non-overlapping chunks of history without mixing lags; layer 3 combines an A matrix that opens only diagonals corresponding to lags with B matrices that select

What would settle it

Compute, for randomly sampled transition matrices P* and lag sets K (including lag sets with close or nested lags), the stationary expectation E[ X_{i-k}^T P* X_i / Σ_l X_{i-l}^T P* X_i ] for the true lag k and for each alternative r. If any alternative has a larger expectation, Claim 1 is false; the paper itself reports a histogram of this difference and all values are positive, but that histogram is finite. The cleanest test is to search adversarially for a counterexample and, if found, to check whether trained transformers still select the wrong lag asymptotically.

Watch

Extended reading notes

Core claim

The central claim is Proposition 1: for any contiguous set of lags K, there exists a three-layer disentangled attention-only transformer whose final-token prediction is a weighted mixture over lags, with weights proportional to exp(β/(T−k̂) Σᵢ p̃ᵢ,ₖ), where p̃ᵢ,ₖ is the normalized transition probability of lag k at position i. In the large-β limit this becomes a hard selection of the lag with the largest cumulative normalized transition probability, and the final attention copies the corresponding past token—the selective induction head. Section 5.3 then claims this selection coincides with maximum-likelihood selection asymptotically: under Claim 1 the true lag has the largest expected norma

Load-bearing premise

Claim 1: for an interleaved Markov chain generated with true lag k, the expected normalized transition probability of lag k is at least that of every other lag; the paper proves this only for two lags, independent lags, and the unnormalized case, and leaves the general proof to future work.

Editorial extensions

If this is right

  • If the construction is what transformers learn, then three attention layers suffice for in-context causal-structure selection, and two layers do not.
  • The selection rule is not Bayesian model averaging but a softmax over accumulated normalized transition evidence; as context grows it approaches the maximum-likelihood lag, not the Bayes average.
  • Standard trained transformers, not just the interpretable disentangled variant, show attention maps matching the constructed circuit, so the mechanism is a plausible account of actual trained models.
  • The same circuit adapts to different lag sets: contiguous lags need K heads, non-contiguous lags need more, and the two-lag case needs only one head while retaining optimal sample complexity.
  • A trained single-head transformer can outperform the interpreted construction, implying gradient descent finds other aggregation strategies with comparable or better sample efficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unproven Claim 1 is the load-bearing inequality for the MLE-convergence result; a single transition matrix and lag set where a wrong lag has higher expected normalized transition probability would refute the asymptotic claim even though the circuit itself still exists.
  • The aggregate-then-select pattern is attention's natural computation, so the same circuit may explain selection among other candidate statistical structures, such as n-gram orders or mixture components, whenever each candidate can be scored by a per-position normalized probability.
  • The observed performance gap between single-head trained transformers and the interpretable single-head construction suggests that superposition-like compression in the aggregation layer is worth reverse-engineering.
  • The setup offers a controlled testbed to measure the transition from Bayesian averaging at short lengths to hard MLE-like selection at long lengths, which could inform when LLMs switch between in-context strategies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a synthetic framework in which sequences are generated from interleaved Markov chains with a fixed transition matrix P* but an unknown, context-dependent lag k drawn from a set K. The task is to predict the next token by selecting the correct causal structure in-context. The authors define 'selective induction heads', propose a detailed 3-layer disentangled-transformer construction that extracts normalized transition probabilities, aggregates them with multiple attention heads, and selects a past token via a softmax-weighted score, and empirically compare trained transformers with the construction. The theoretical section (Sec. 5.3) claims that the constructed predictor asymptotically converges to the maximum-likelihood solution. The paper also provides special-case proofs in App. A, experiments on several lag sets, and attention-map comparisons.

Significance. If the theoretical claim were fully established, the paper would make a useful contribution: it identifies a concrete mechanism by which transformers can select among causal structures, gives an explicit circuit-level construction, and provides evidence that trained transformers implement the same algorithm. The construction itself is detailed and the experiments are extensive, including head/layer ablations and attention-map alignments. The two-lag proof, the ergodicity argument, and the empirical validation of the score are valuable. However, the headline theoretical contribution — asymptotic convergence to the MLE — is conditional on an explicitly unproved inequality, Claim 1, and the exact statement of Proposition 1 requires the λ→∞ limit. The paper is therefore stronger as an empirical/mechanistic study than as a proof of MLE equivalence.

major comments (3)
  1. [Sec. 5.3, App. A] The central asymptotic claim — that the cumulative normalized-transition score selects the true lag and hence matches MLE — is exactly Claim 1. The paper explicitly states 'we leave the complete proof of Claim 1 for future work' and provides only the two-lag case, the independent-lag case, and the unnormalized case. For |K|≥3, the denominator couples all lags, so the pairwise Lemma 1 cannot be directly applied. Since Claim 1 is the only bridge between the construction and MLE equivalence, the abstract's phrase 'a theoretical analysis proving that this mechanism asymptotically converges to the maximum likelihood solution' is stronger than what the manuscript establishes. Please either prove Claim 1 under stated conditions or explicitly restate the theoretical conclusion as conditional on Claim 1.
  2. [Prop. 1, Sec. 5.1, Eq. (3)] Proposition 1 is stated as an exact existence result: 'there exists a three-layer disentangled transformer' whose output equals Eq. (3). However, the construction derives Eq. (3) only in the λ→∞ limit; at finite λ the softmax attention gives a smoothed approximation, not the exact normalized probabilities p̃_{i,k} and exact weights w̃_k. To justify the proposition as stated, either formulate it as a limit statement (λ→∞, possibly β→∞) or provide finite-λ error bounds that vanish as λ grows. This is a load-bearing gap between the statement and the proof.
  3. [Sec. 5.3, Eq. (3), Sec. 5.2] The claimed convergence to MLE requires specifying the limiting order of T→∞ and β→∞. Eq. (3) already contains the temperature β; for a fixed finite β the softmax weights do not (generically) converge to the hard argmax that defines k*. The manuscript says 'for large β' in Sec. 5.2 and 'high-temperature limit' in App. A, which is inconsistent. Please define precisely the asymptotic regime (e.g., T→∞ then β→∞, or β growing with T) under which Eq. (3) reduces to MLE selection.
minor comments (4)
  1. [Abstract, Conclusion] The abstract says 'a theoretical analysis proving that this mechanism asymptotically converges to the maximum likelihood solution', while the Conclusion says 'in certain cases, it asymptotically converges to maximum likelihood'. These are materially different. Adjust the abstract to match the actual conditional statement.
  2. [Fig. 3, App. E] The empirical validation of Claim 1 reports histograms of a positive difference over sampled matrices. Please include error bars or worst-case gaps over the sampled transition matrices to make the strength of the empirical support clearer.
  3. [App. A, Lemma 2 proof] The notation in the Cauchy-Schwarz chain in Lemma 2 is difficult to parse (e.g., the lines beginning with square-root sums and the marginalization step). Please restate the proof with explicit sums and a clear statement of the coupling/marginalization argument.
  4. [Sec. 3, Eq. (3)] The temperature notation is inconsistent: in the softmax definition α>0 is the temperature, while Eq. (3) uses β as an inverse temperature. Clarify the relationship and use consistent terminology for the 'large-β' / 'high-temperature' limits.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline derivation rests on an unproved inequality (Claim 1), not on a step that reduces to its own inputs.

full rationale

The derivation chain is constructive and self-contained: explicit attention matrices in Sec. 5.1 produce the estimator in Eq. (3), and the claimed asymptotic agreement with MLE in Sec. 5.3 is conditional on Claim 1, a substantive probabilistic inequality about interleaved Markov chains. Claim 1 is honestly labeled as unproved in general ("we leave the complete proof of Claim 1 for future work") and supported only by special cases and empirical histograms. This is a proof gap or an overstatement of what is proven, but it is not circular: the true lag is not defined as the maximizer of E[\tilde p_{i,k}], and the inequality is not an identity arrived at by construction. Nor is any parameter fitted to the benchmark data and then renamed a prediction. The paper contains no load-bearing self-citations; the cited prior work (Friedman et al., Nichani et al., Olsson et al., Rousseau and Mengersen) supplies background architecture, known consistency of MLE, and induction-head context, none of which smuggles in the paper's own conclusion. The secondary issue that Eq. (3) is formally obtained in the \lambda\to\infty limit while Proposition 1 states finite transformer weights is a rigor gap, not a circular reduction. Overall, the central mechanism has independent constructive content, and the unproved Claim 1 is an assumption, not an input-output equivalence.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The construction is the main contribution and is mostly self-contained, but the asymptotic optimality claim relies on an unproven inequality (Claim 1). The hand-set constants lambda and beta are needed for the finite implementation, and P* is an input to the construction that trained models must estimate.

free parameters (2)
  • lambda (attention sharpness) = 500
    Hand-chosen constant for finite-length implementation; the proof requires the limit lambda -> infinity.
  • beta (selection temperature) = 100
    Hand-chosen constant; the hardmax selection analysis requires beta -> infinity.
assumptions (4)
  • domain assumption Interleaved Markov chain is ergodic with stationary distribution pi; the first k_hat samples of each sequence are drawn i.i.d. from pi.
    Used in Sec 4 and Algorithm 1 to define the data generating process and to justify ergodic convergence of cumulative averages.
  • domain assumption The transition matrix P* is known to the constructed transformer, which places log P*^T and P*^T into its weights.
    The construction in Sec 5.1 uses P* directly in layer 1 and the output layer; trained transformers must estimate P* from data, which the paper shows empirically but does not prove.
  • ad hoc to paper Claim 1: the expected normalized transition probability is maximized by the true lag.
    Unproven inequality underpinning the asymptotic-MLE claim; only special cases are proven in App A, and the full proof is deferred to future work.
  • standard math Softmax limits lambda -> infinity and beta -> infinity are taken and are assumed to preserve the finite-behavior mechanism.
    Used throughout the construction to convert soft attention into hard selection; in experiments finite values are used, so the proof is asymptotic.
invented entities (1)
  • Selective induction head
    purpose: Name for the three-layer circuit that selects the lag to copy from based on aggregated transition scores.
    Introduced in this paper as a new mechanistic concept; the only evidence offered is the synthetic task and attention-map similarities in trained transformers, with no external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Selective Induction Heads: How Transformers Select Causal Structures In Context." pith.science (2026). https://pith.science/paper/KZRV7XD2

@misc{pith2026250908184,
  author       = {Pith},
  title        = {Pith review of: Selective Induction Heads: How Transformers Select Causal Structures In Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZRV7XD2}},
  note         = {Machine review of arXiv:2509.08184}
}
read the original abstract

Transformers have exhibited exceptional capabilities in sequence modeling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, attention circuits that enable copying tokens based on their previous occurrences. In this work, we introduce a novel framework that showcases transformers' ability to dynamically handle causal structures. Existing works rely on Markov Chains to study the formation of induction heads, revealing how transformers capture causal dependencies and learn transition probabilities in-context. However, they rely on a fixed causal structure that fails to capture the complexity of natural languages, where the relationship between tokens dynamically changes with context. To this end, our framework varies the causal structure through interleaved Markov chains with different lags while keeping the transition probabilities fixed. This setting unveils the formation of Selective Induction Heads, a new circuit that endows transformers with the ability to select the correct causal structure in-context. We empirically demonstrate that transformers learn this mechanism to predict the next token by identifying the correct lag and copying the corresponding token from the past. We provide a detailed construction of a 3-layer transformer to implement the selective induction head, and a theoretical analysis proving that this mechanism asymptotically converges to the maximum likelihood solution. Our findings advance the understanding of how transformers select causal structures, providing new insights into their functioning and interpretability.

Figures

Figures reproduced from arXiv: 2509.08184 by the authors.

Figure 1
Figure 1. Summary of the framework. Top: We define a new task based on Interleaved Markov Chains of different lags (k = 1 and k = 2 in the example). Middle: given a sequence generated from a chain of unknown lag, the model has to identify the true lag, and use it to predict the distribution of the next token. Bottom: attention-only transformers can solve this task with 3 layers. The first computes the transition probabilities… view at source ↗
Figure 2
Figure 2. Visualization of the mechanism of the second attention layer in our construction. The matrix represents the input of the second layer h (1) whereas the single vectors represent the output for the 10th token hˆ(2,1) 10 , hˆ(2,2) 10 , hˆ(2,3) 10 . Each of the three attention heads (arrows of different colors) copies non-overlapping transition probabilities at distance 3 from each other from the past. By doing this, th… view at source ↗
Figure 3
Figure 3. Left difference of the expected normalized transition probabilities for the true lag and the maximum over all other lags for |S| = 10. Right: the estimator in Eq. 3 matches MLE for large β. 5.4. Generalizations and special cases Single-head transformers. The construction above allows the model to store all the past transition probabilities in the embedding of the current token by scaling the number of heads with the… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Performance of our constructed transformers, trained transformers, and theoretical estimator (BMA, ML). First plot: lags 1,2,3. Second: the model solves the task with non-contiguous lags. Third: the model is effective with additional lags. Fourth: one head is enough fo…
Figure 5
Figure 5. Figure 5: Attention Maps: Trained versus Constructed Transformers. Heatmaps depicting the atten￾tion mechanisms of the trained standard transformer, disentangled transformer, and our constructed model are presented for training lags 3, 5, and 7 and the true lag that generated th…
Figure 6
Figure 6. Figure 6: (right) instead we consider the task defined by K = {1, 2, 3, 4, 5} and train transformers with fewer, equal to, or more than K heads. As predicted by our construction increasing the number of heads leads to performances that get closer to the maximum likelihood up to …
Figure 7
Figure 7. Figure 7: Attention maps for different heads in the second layer and K = {1, 2, 3, 4, 5}: we observe how even with fewer heads the transformer learns layers which are consistent with the operations in our construction. In particular the first layer is still extracting the transi…
Figure 8
Figure 8. Figure 8: Attention maps K = {1, 2} and true lag k = 2 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Attention maps K = {1, 2, 3} and true lag k = 1. 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained standard transformer 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained disentangled transformer 0 5 10 15 20 25 30 35 40 45 Attention Layer 1 0 5 10 15 20 25 30 35 40 4…
Figure 10
Figure 10. Figure 10: Attention maps K = {1, 2, 3} and true lag k = 2 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Attention maps K = {1, 2, 3} and true lag k = 3. 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained standard transformer 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained disentangled transformer 0 5 10 15 20 25 30 35 40 45 Attention Layer 1 0 5 10 15 20 25 30 35 40 …
Figure 12
Figure 12. Figure 12: Attention maps K = {1, 3, 4} and true lag k = 1 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Attention maps K = {1, 3, 4} and true lag k = 3. 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained standard transformer 0 5 10 15 20 25 30 35 40 45 Sequence Index Trained disentangled transformer 0 5 10 15 20 25 30 35 40 45 Attention Layer 1 0 5 10 15 20 25 30 35 40 …
Figure 14
Figure 14. Figure 14: Attention maps K = {1, 3, 4} and true lag k = 4 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Expected Normalized Transition Probabilities for |S| = 10: The sampled set of lags is K = {1, 2, 7, 9, 10, 11, 13, 15, 16, 22, 26, 28}, we sampled 10 different transition matrices and for each lag and each matrix sampled 1000 sequences of length 1000. The expected nor…
Figure 16
Figure 16. Figure 16: Expected Normalized Transition Probabilities for |S| = 25: The sampled set of lags is K = {1, 2, 7, 9, 10, 11, 13, 15, 16, 22, 26, 28}, we sampled 10 different transition matrices and for each lag and each matrix sampled 1000 sequences of length 1000. The expected nor…
Figure 17
Figure 17. Figure 17: Cumulative average of Normalized Transition Probabilities for |S| = 10: The sampled set of lags is K = {1, 2, 7, 9, 10, 11, 13, 15, 16, 22, 26, 28}, we report one sequence sampled according to one the transition matrix. The cumulative average of normalized transition …
Figure 18
Figure 18. Figure 18: Cumulative average of Normalized Transition Probabilities for |S| = 25: The sampled set of lags is K = {1, 2, 7, 9, 10, 11, 13, 15, 16, 22, 26, 28}, we report one sequence sampled according to one the transition matrix. The cumulative average of normalized transition …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [1]

    K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. InThirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URLhttps://openreview.net/forum?id=LziniAXEI9

    work page 2023

  2. [2]

    Akyürek, D

    E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models.arXiv preprint arXiv:2211.15661, 2022

    arXiv 2022

  3. [3]

    Y. Bai, F. Chen, H. Wang, C. Xiong, and S. Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id=liMSqUuVg9

    work page 2023

  4. [4]

    Y. Bai, F. Chen, H. Wang, C. Xiong, and S. Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection.Advances in neural information processing systems, 36, 2023

    work page 2023

  5. [5]

    T. Batu, S. Guha, and S. Kannan. Inferring mixtures of markov chains. InLearning Theory: 17th Annual Conference on Learning Theory, COLT 2004, Banff, Canada, July 1-4, 2004. Proceedings 17, pages 186–199. Springer, 2004

    work page 2004

  6. [6]

    Berchtold and A

    A. Berchtold and A. Raftery. The mixture transition distribution model for high-order markov chains and non-gaussian time series.Statistical Science, 17(3):328–356, 2002

    work page 2002

  7. [7]

    Bietti, V

    A. Bietti, V. Cabannes, D. Bouchacourt, H. Jegou, and L. Bottou. Birth of a transformer: A memory viewpoint.Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

  8. [8]

    T. B. Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

Show all 30 references
  1. [9]

    S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers.Advances in neural information processing systems, 35:18878–18891, 2022

    2022

  2. [10]

    B. L. Edelman, E. Edelman, S. Goel, E. Malach, and N. Tsilivis. The evolution of statistical induction heads: In-context learning markov chains.arXiv preprint arXiv:2402.11004, 2024

    2024 arXiv

  3. [11]

    Friedman, A

    D. Friedman, A. Wettig, and D. Chen. Learning transformer programs. InThirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id= Pe9WxkN8Ff

    2023

  4. [12]

    S. Garg, D. Tsipras, P. S. Liang, and G. Valiant. What can transformers learn in-context? a case study of simple function classes.Advances in Neural Information Processing Systems, 35:30583– 30598, 2022

    2022

  5. [13]

    J. Hu, Q. Liu, and C. Jin. On limitation of transformer for learning hmms.arXiv preprint arXiv:2406.04089, 2024

    2024 arXiv

  6. [14]

    M. E. Ildiz, Y. HUANG, Y. Li, A. S. Rawat, and S. Oymak. From self-attention to markov mod- els: Unveiling the dynamics of generative transformers. InForty-first International Conference on Machine Learning, 2024. URLhttps://openreview.net/forum?id=72oT4mPLUb

    2024

  7. [15]

    A. V. Makkuva, M. Bondaschi, A. Girish, A. Nagle, M. Jaggi, H. Kim, and M. Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains.arXiv preprint arXiv:2402.04161, 2024

    2024 arXiv

  8. [16]

    Minot and Y

    A. Minot and Y. M. Lu. Separation of interleaved markov chains. In2014 48th Asilomar Conference on Signals, Systems and Computers, pages 1757–1761. IEEE, 2014

    2014

  9. [17]

    T. Nguyen. Understanding transformers via n-gram statistics.arXiv preprint arXiv:2407.12034, 2024

    2024 arXiv

  10. [18]

    Howtransformerslearncausalstructurewithgradientdescent

    E.Nichani, A.Damian, andJ.D.Lee. Howtransformerslearncausalstructurewithgradientdescent. InForty-first International Conference on Machine Learning, 2024. URLhttps://openreview. net/forum?id=jNM4imlHZv

    2024

  11. [19]

    Olsson, N

    C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022. 15

    2022 arXiv

  12. [20]

    Rajaraman, M

    N. Rajaraman, M. Bondaschi, K. Ramchandran, M. Gastpar, and A. V. Makkuva. Transformers on markov data: Constant depth suffices.arXiv preprint arXiv:2407.17686, 2024

    2024 arXiv

  13. [21]

    Raventós, M

    A. Raventós, M. Paul, F. Chen, and S. Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression.Advances in Neural Information Processing Systems, 36, 2024

    2024

  14. [22]

    Rousseau and K

    J. Rousseau and K. Mengersen. Asymptotic behaviour of the posterior distribution in overfitted mixture models.Journal of the Royal Statistical Society Series B: Statistical Methodology, 73(5): 689–710, 2011

    2011

  15. [23]

    M. E. Sander, R. Giryes, T. Suzuki, M. Blondel, and G. Peyré. How do transformers perform in-context autoregressive learning ? InForty-first International Conference on Machine Learning,

  16. [24]

    Svete and R

    A. Svete and R. Cotterell. Transformers can representn-gram language models.arXiv preprint arXiv:2404.14994, 2024

    2024 arXiv

  17. [25]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polo- sukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processin...

    2017

  18. [26]

    Von Oswald, E

    J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. InInternational Conference on Machine Learning, pages 35151–35174. PMLR, 2023

    2023

  19. [27]

    Von Oswald, E

    J. Von Oswald, E. Niklasson, M. Schlegel, S. Kobayashi, N. Zucchet, N. Scherrer, N. Miller, M. San- dler, M. Vladymyrov, R. Pascanu, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023

    2023 arXiv

  20. [28]

    S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  21. [29]

    Yadlowsky, L

    S. Yadlowsky, L. Doshi, and N. Tripuraneni. Pretraining data mixtures enable narrow model selec- tion capabilities in transformer models.arXiv preprint arXiv:2311.00871, 2023. 16 Organization of the Appendix.The Appendix is organized as follows. App. A extends the statistical ...

    2023 arXiv

  22. [2024]

    URLhttps://openreview.net/forum?id=kZbTkpnafR

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.