Provable Low-Frequency Bias of In-Context Learning of Representations

Hidenori Tanaka; Wei Hu; Yongyi Yang

REVIEW 3 major objections 5 minor 19 references

Provable Low-Frequency Bias of In-Context Learning of Representations

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In-context learning provably favors low-frequency representations

desk verdict The double-convergence framework and Theorem 1 are genuinely useful, but Theorem 2's contraction inequality is inverted, so the paper's central claim is not proven as written. read the letter →

arxiv 2507.13540 v2 pith:4DHQMU3O submitted 2025-07-17 cs.LG

Yongyi Yang , Hidenori Tanaka , Wei Hu This is my paper

classification cs.LG

keywords in-contextlearninghiddenrepresentationslow-frequencybiasdoubleconvergencegraphLaplacianattentionmapsspectralgapnoiserobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the hidden representations of a transformer performing in-context learning are driven, by the architecture itself, into the low-frequency subspace of a graph that encodes the prompt's data-generating process. It proves this through a 'double convergence' argument: within each layer, token representations converge as context grows, and across layers the latent cluster centers converge toward the top eigenspace of the reweighted adjacency matrix $M = D^{-1/2} W D^{-1/2}$. If true, this gives a unified explanation for several observed in-context learning phenomena, including suppression of pretrained word meanings, mildly distorted representation geometry, and decaying but nonvanishing energy. It also predicts that in-context learning is intrinsically robust to high-frequency noise, which the paper verifies empirically.

What carries the argument

The argument rests on three objects. First, 'nice attention maps' (Definition 3): attention matrices that are lower-triangular, row-stochastic, have weights satisfying a soft uniformity condition, and reflect a token-to-set function with weights distributed proportionally to token frequencies. Second, 'great mappings' (Definition 4): neuron-wise nonlinearities that preserve the norms of the latent representation projected onto the low- and high-frequency subspaces up to constants $\gamma_1, \gamma_2$. Third, the spectral gap $\delta_q(\rho_A I + \rho_B M)$ of the reweighted graph matrix, together with the condition $\delta_q \gamma_1/\gamma'_2 \le 1 - \epsilon$, which makes the high-frequency-to-low-frequency energy ratio contract by a factor $1-\epsilon$ at every layer. These ingredients let the proof separate context-wise convergence (Theorem 1) from layer-wise convergence and combine them into Theorem 2.

What would settle it

Measure the ratio $\|V_n^{(\ell)} D^{1/2} Y\| / \|V_n^{(\ell)} D^{1/2} X\|$ across layers of a deep transformer fed random-walk prompts: the proof predicts geometric decay with rate at most $1-\epsilon$ per layer, so a flat or rising ratio would falsify Theorem 2. A head-level check of eq. (5), whether attention weights follow token frequencies within $O(\log n/\sqrt{k})$, would locate the failure precisely.

Watch

Extended reading notes

Core claim

The central claim is Theorem 2: under the paper's assumptions, the ratio $\|V_n^{(L)} D^{1/2} Y\| / \|V_n^{(L)} D^{1/2} X\|$ tends to $0$ as the number of layers and then the context length grow, so the hidden representations converge to the top eigenspace of the symmetrically reweighted graph matrix $M = D^{-1/2} W D^{-1/2}$. Because $M = I - \tilde{L}$ for the symmetrically normalized Laplacian $\tilde{L}$, this top eigenspace is exactly the low-frequency, smooth subspace of the graph underlying the prompt's random walk. The paper presents this as the reason in-context learning of representations emerges: the model ends up encoding global graph structure in its hidden states, and the same mechanism explains why high-frequency components of the input, including corruptions, are progressively suppressed.

Load-bearing premise

The argument collapses if real attention maps do not put weights on the four fixed connection types in proportion to token frequencies (Definition 3), or if some layer's nonlinearity fails to contract high-frequency components relative to low-frequency ones by a uniform factor; the paper's own empirical check verifies only the connection types, not the proportionality condition (eq. (5)).

Editorial extensions

If this is right

The suppression of pretrained word meanings reported by Park et al. is explained as progressive attenuation of high-frequency components, rather than a separate mechanism.
The slightly compressed periphery in PCA plots of learned representations is predicted to come from the reweighting $W = \hat{W}_{x,y} \pi_x \pi_y$, not just uneven visitation frequency.
Energy decay across layers follows because high-frequency components leave low eigenspaces while low-frequency components persist, so total energy decays but does not vanish.
In-context learning should be intrinsically robust to high-frequency input noise, so a 1% uniform-token corruption rate is corrected once the context grows, matching the paper's experiments.
The first eigenvector of $M$ is effectively removed by PCA centralization, which explains why the observed structure appears to start from the second eigenvector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the low-frequency bias is as generic as the authors suggest, the same double-convergence argument should apply to any Markov-generated prompt, implying that the denoising behavior is not specific to planar graph walks.
The framework suggests a testable ordering: tasks whose correct answer lives in the high-frequency subspace of the prompt graph should be harder for deep in-context learning, because those components are exactly the ones being killed.
A natural extension is to prove that gradient-based pretraining produces attention maps satisfying Definition 3, which would turn the structural assumption into a derived property rather than a postulate.
The noise-robustness experiment implies a practical prescription: for noisy inputs, more layers and longer contexts should monotonically improve accuracy, which can be checked inexpensively on existing models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a serious theory paper with a genuinely useful framing, but the central theorem does not hold as stated. The contraction multiplier in Theorem 2 is backwards. Corollary 11 (eq. 54) gives R_ℓ ≤ (γ′_2/(γ_1 δ_q)) R_{ℓ−1}, so contraction requires δ_q γ_1/γ′_2 > 1. Theorem 2 instead assumes δ_q γ_1/γ′_2 ≤ 1−ε, which makes the per-layer factor at least 1/(1−ε) > 1, so the Y/X ratio grows rather than vanishes. Eq. (62) then applies δ_q γ_1/γ′_2 as the multiplier, which is the reciprocal of what Corollary 11 provides. A linear counterexample satisfies the stated assumptions and the ratio diverges. So the advertised provable low-frequency bias is not proven; the fix would need the opposite inequality and a nontrivial condition on the nonlinearities, which the paper neither states nor verifies.

What is new and worth keeping: the double-convergence abstraction—context-wise first, then layer-wise—and the notion of reflect/nice attentions. Theorem 1, the context-wise step, looks internally consistent and is a clean lemma about how attention acts on latent representations once the map is regular enough. The spectral characterization of the layer-wise limit (top eigenspace of ρ_A I + ρ_B D^{−1/2} W D^{−1/2}) is a good target for a mechanistic explanation, and the empirical plots do show a qualitative match with the predicted eigenvectors. The paper is also unusually honest about its assumptions and puts enough detail in the appendix that the proof error was catchable, which is a point in its favor.

Soft spots beyond the theorem: the context-wise conclusion is partly encoded in Definition 3's eq. (5), which already requires attention weights to match empirical token frequencies; Appendix F checks only connectivity-type proportions, not that proportional-weight condition. The noise-robustness experiment is a single run on one model with no error bars, and no code or data are released.

Who this is for: people working on inference dynamics of Transformers and on spectral methods for representation learning. The framework and Theorem 1 may survive a correction; Theorem 2 needs to be repaired and reassessed before the main claims can be taken seriously. I would still send this to peer review rather than desk-reject, because the framework is important, the error is subtle, and a good referee can either kill the central claim or help the authors fix it. But I would not cite Theorem 2 as proven.

Referee Report

3 major / 5 minor

Summary. This paper proposes a theoretical account of In-Context Learning of Representations (ICLR), the phenomenon observed by Park et al. (2024) in which hidden representations of a deep Transformer, fed with a random walk over a graph, converge to a structure aligned with that graph. The authors introduce a 'double convergence' framework. Theorem 1 states that if an attention map is 'nice' (Definition 2) and reflects a token function f with weights close to the empirical token frequencies within f(x) (Definition 3, eq. (5)), then a sequence of hidden states that clusters around token-dependent latent vectors remains clustered, and the attention acts on the latent vectors as a stationary-distribution-weighted average over f(x) (eq. (6)). Theorem 2 asserts that under a four-type decomposition of each attention map (self, neighbor, other, sink; eq. (7)) and layer-wise nonlinearities that are 'great mappings' (Definition 4) with a spectral-gap condition on the reweighted graph operator M = D^{-1/2} W D^{-1/2}, the ratio of high-frequency to low-frequency components of the latent representations tends to zero as L tends to infinity after n tends to infinity (eq. (10)); that is, representations converge to the low-frequency eigenspace of M.

Significance. The paper is ambitious, and its modular design is a genuine methodological contribution: treating attention maps as given, absorbing all neuron-wise nonlinearities into norm-ratio 'great mapping' constants, and separating context-wise from layer-wise convergence makes a deep, nonlinear analysis tractable. The empirical content is concrete and falsifiable: Figure 2 compares the analytically predicted second and third eigenvectors of M against actual PCA coordinates of hidden states, Figure 4 tests the predicted denoising of high-frequency input corruption, and Figure 3 shows the predicted energy redistribution across eigenspaces. If Theorem 2 were correct as stated and eq. (5) verified on real models, this would be the first rigorous explanation of ICLR and a useful foundation for studying inference dynamics. That said, the central theorem currently contains a direction error detailed below, and the paper's own verification covers the attention-type decomposition but not the frequency-proportionality condition (eq. (5)) or the per-layer spectral and norm-ratio conditions. These two gaps separate the current manuscript from its central claim.

major comments (3)

[Theorem 2 and its proof (Appendix B.2, eq. (62))] The per-layer contraction multiplier is inverted, so Theorem 2 is false as stated. Corollary 11 (eq. (54)), together with its proof (eqs. (55)-(56)), bounds the ratio R_ell = ||Z'(ell) D^{1/2} Y|| / ||Z'(ell) D^{1/2} X|| by R_ell <= (gamma'_2 / (gamma_1 * delta_q)) R_{ell-1}: the attention step contracts the high/low-frequency ratio by a factor 1/delta_q = |lambda_{q+1}| / |lambda_q|, and the nonlinearity can expand it by at most gamma'_2 / gamma_1. Contraction therefore requires delta_q * gamma_1 / gamma'_2 > 1, whereas Theorem 2 assumes delta_q * gamma_1 / gamma'_2 <= 1 - epsilon, under which every layer multiplies the ratio by at least 1/(1 - epsilon) > 1 and the ratio diverges. Eq. (62) applies delta_q * gamma_1 / gamma'_2 as the multiplier, which is the reciprocal of the factor that Corollary 11 actually provides. A concrete linear counterexample satisfies the stated assumptions: take M with delta_q = 1.5 and take sigma to be the neuron-wise linear map diag(1, 2), so that gamma_1 = 1, gamma'_2 = 2, and delta_q * gamma_1 / gamma'_2 = 0.75 <= 1 - epsilon; then every layer multiplies the Y/X ratio by 2 / (1 * 1.5) = 4/3, so the ratio diverges and eq. (10) fails. The theorem is repairable by reversing the assumption to delta_q * gamma_1 / gamma'_2 >= 1 + epsilon and replacing the multiplier in eq. (62) by gamma'_2 / (gamma_1 * delta_q), but as written the proof establishes the opposite of the claimed convergence.
[Appendix F vs. Definition 3 / Theorem 2] Assumption verification does not cover the load-bearing conditions. The link to real pre-trained models requires, per layer: (i) the frequency-proportionality condition of Definition 3, eq. (5), which drives the pi-weighted form of the limiting attention update in Theorem 1; and (ii) the great-mapping constants gamma_1, gamma'_2 and the spectral-gap condition of Theorem 2. Appendix F verifies only that self, neighbor, and sink connection types account for over 72 percent of attention weights, which concerns the support (connectivity pattern) of the attention matrices and is silent on eq. (5); nothing in the appendix measures psi of Definition 3 or the per-layer norm-ratio constants in eq. (8) on real models. Consequently, the assertion that the theory explains the ICLR phenomena observed in llama-3.1-8B is not supported by the provided evidence; the empirical sections validate the predicted consequences (eigenvector alignment, noise robustness, energy redistribution), not the assumptions from which those consequences were derived.
[Definition 3, eq. (5), and Theorem 1, eq. (6)] The conclusion of Theorem 1 is partly assumed by Definition 3. Eq. (5) explicitly requires attention weights within f(x) to match the empirical frequencies F_{y,k} / sum_{y' in f(x)} F_{y',k} up to O(psi / sqrt(k)); since these empirical frequencies concentrate around pi_y by Corollary 4, the limit in eq. (6) is essentially the stationary-weighted average that the definition already points to. What Theorem 1 genuinely adds is that the good-sequence (clustering) property propagates through the layer at the stated rate, which is not vacuous. However, the explanation in Section 6.1 that ICL suppresses original semantic meaning is conditional on having assumed that attention is proportional to DGP token frequencies; it is not derived from the graph structure or from the model alone. The paper is transparent about not deriving attention maps (Section 2), so this is a framing issue rather than an inconsistency, but the summary and Section 7 should state clearly that the frequency-proportionality condition is an assumption, not a prediction.

minor comments (5)

[Definition 5 and Theorem 2] Definition 5 states that eigenvalues are arranged in non-decreasing order of absolute values, while Theorem 2 (eq. (9)) uses non-increasing order; since the meaningful regime is delta_q > 1, the non-increasing convention is the intended one and Definition 5 should be corrected.
[Theorem 2, eq. (9)] The eigen-decomposition states that Lambda' contains n - q eigenvalues, but M is c by c; this should be c - q. There is also a notational collision between the sequence length n and the matrix size c in V_n^{(L)}.
[Appendix B.1, proof of Corollary 11 (eq. (55))] The displayed inequality writes ||Z' D^{1/2} X|| on both sides; the second occurrence should be ||Z D^{1/2} X||, since the spectral bound expresses the norm of the attention output in terms of the norm of the input.
[Section 6.4] The sentence 'the projections of the latent representations onto the low eigenspaces converges to 0' is misleading: under Theorem 2 it is the high-frequency eigenspaces (large index i) whose components vanish, while the low-frequency components persist or grow; please rephrase.
[Section 1 and Section 6] There are minor typos: 'the theContext-wise Process' in Section 1 and 'shown in in Figure 2' in Section 6; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's derivation is self-contained and conditional on explicitly posited attention assumptions; the disputed Theorem 2 proof step is a correctness concern, not a circular reduction.

full rationale

The claimed derivation chain is formally self-contained and non-circular. Theorem 1 is a consequence of Definition 3's proportionality condition (eq. 5), the nice-attention tail bound (eq. 3), and the random-walk ergodicity estimate (Corollary 4, from Fan et al. 2021): the attention output is shown to be within O((ψγ + ψN + log n)/√k) of the frequency-weighted average Σ_y (F_{y,k}/R_k) z_y, and that average is shown to approach the stationary-weighted average of eq. (6) because F_{y,k}/k concentrates on π_y. This is a conditional 'if attention has this structure, then representations average accordingly' statement, and the paper is explicit that the attention structure is posited, not derived: 'we adopt a more pragmatic approach: Rather than attempting to derive the structure of attention maps from first principles, we posit a structured form' (Section 2), and deriving it from pretraining is left as 'a key next step' (Section 7). The proportionality assumption is therefore not a hidden restatement of the conclusion; it concerns attention weights, while the theorem adds the ergodicity and error-propagation steps, and the paper does not use the theorem to justify the assumption. Layer-wise convergence (Corollaries 10-11, Theorem 2) is a standard spectral contraction applied to the explicitly assumed dynamics Z' = ρ_A Z + ρ_B D^{-1}W Z + ...; no fitted parameter is renamed as a prediction, and the noise-robustness prediction (Section 6.5) is an out-of-sample empirical test on llama-3.1-8B, not a fit to the theory's constants. Citations to Park et al. (2024) (two overlapping authors) supply the ICLR phenomenon, the DGP, and the empirical benchmark, but the phenomenon is independently reproducible and the theorem proofs are contained in this paper, so the self-citation is not load-bearing. Two non-circularity concerns should nevertheless be flagged per the review rule: (i) Appendix F verifies only the eq. (7) A/B/T connectivity decomposition (about 72% of weights) and does not verify the load-bearing eq. (5) proportionality condition, so the empirical support for the key assumption is incomplete; (ii) the proof of Theorem 2 at eq. (62) appears to apply δ_q γ_1/γ'_2 as the per-layer multiplier of ||Y||/||X||, whereas Corollary 11 (eq.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central claim rests on a set of structural assumptions about attention and nonlinearities that are stated clearly but only partially verified. The reweighted adjacency W and the operator M are constructions from the assumed DGP and attention types, so the ledger is dominated by ad hoc modeling choices rather than fitted parameters.

free parameters (4)

attention-type mixing weights ρ_A, ρ_B, ρ_O, ρ_T per layer
Introduced in eq. (7) as a convex combination; the theorem's operator M = ρ_A I + ρ_B D^{-1/2} W D^{-1/2} and its spectral gap depend on them, and the spectral gap condition must hold, but they are never measured or fitted.
nice-attention parameters ψ (Definitions 2 and 3)
Assumed small; controls the rate in Theorem 1. Not estimated from data.
great-mapping constants γ_1, γ_2, γ'_1, γ'_2 per layer
Definition 4 (eq. 8) assumes nonlinearities preserve norm ratios within the eigenspaces; together with δ_q they must satisfy the contraction condition in Theorem 2. Not measured.
spectral-gap slack ε
Theorem 2 assumes existence of ε > 0 with δ_q(...) γ_1/γ'_2 ≤ 1 - ε; never estimated empirically.

assumptions (7)

ad hoc to paper Attention maps are given, not derived: A^(ℓ) = ρ_A A^(ℓ,A) + ρ_B A^(ℓ,B) + ρ_O A^(ℓ,O) + ρ_T A^(T) (eq. 7).
Load-bearing structural assumption; the paper states it aims to show consequences of such maps, not to derive them (Section 2).
ad hoc to paper Definition 3 (eq. 5): reflecting attentions distribute weights proportionally to token frequencies within f(x_k), with error ψ/√k.
This proportionality essentially preselects the π-weighted averaging limit in Theorem 1; it is only empirically checked as connectivity type, not as a quantitative distribution.
ad hoc to paper Definition 4 and Theorem 2 conditions: each σ^(ℓ) is a 'great mapping' with norm-ratio bounds along X and Y, and the spectral gap condition holds.
Discharges nonlinearities (FFN, norms) without evidence that ReLU or GELU satisfy it; Appendix C only proves linear maps are great.
domain assumption DGP: first c tokens are a fixed traversal, then a random walk on connected undirected graph G, n > 10c (Section 3.1).
Copies Park et al. (2024); the fixed traversal is an edge-case simplification.
standard math Markov-chain Hoeffding inequality (Theorem 3 from Fan et al. 2021).
Used for Corollaries 4 and 5 to control frequency deviations; standard concentration result.
standard math Spectral graph facts: M = I - normalized Laplacian; first eigenvector d^{-1/2}; low eigenvectors smooth (Spielman 2019, cited in Section 6).
Bridges M's top eigenspace to low-frequency graph signals.
standard math PCA centralization removes the component along d^{-1/2} (Section 6.2).
Explains why eigenvectors 2 and 3 match PC1 and PC2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Provable Low-Frequency Bias of In-Context Learning of Representations." pith.science (2026). https://pith.science/paper/4DHQMU3O

@misc{pith2026250713540,
  author       = {Pith},
  title        = {Pith review of: Provable Low-Frequency Bias of In-Context Learning of Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DHQMU3O}},
  note         = {Machine review of arXiv:2507.13540}
}

read the original abstract

In-context learning (ICL) enables large language models (LLMs) to acquire new behaviors from the input sequence alone without any parameter updates. Recent studies have shown that ICL can surpass the original meaning learned in pretraining stage through internalizing the structure the data-generating process (DGP) of the prompt into the hidden representations. However, the mechanisms by which LLMs achieve this ability is left open. In this paper, we present the first rigorous explanation of such phenomena by introducing a unified framework of double convergence, where hidden representations converge both over context and across layers. This double convergence process leads to an implicit bias towards smooth (low-frequency) representations, which we prove analytically and verify empirically. Our theory explains several open empirical observations, including why learned representations exhibit globally structured but locally distorted geometry, and why their total energy decays without vanishing. Moreover, our theory predicts that ICL has an intrinsic robustness towards high-frequency noise, which we empirically confirm. These results provide new insights into the underlying mechanisms of ICL, and a theoretical foundation to study it that hopefully extends to more general data distributions and settings.

Figures

Figures reproduced from arXiv: 2507.13540 by the authors.

**Figure 1.** Overview of the DGP and the Double Convergence Process. The input sequence is generated by a random walk over a graph defined on the vocabulary (top), and then passed into a pre-trained Transformer model. Context-wise convergence occurs within each layer: token representations associated with the same word converge into tight clusters. Layer-wise convergence occurs across layers: the cluster centers gradually evolve… view at source ↗

**Figure 2.** Comparison between empirical observations and theoretical predictions. (a): The 2nd and 3rd eigenvectors of Wf, illustrating the idealized grid structure hypothesized in Park et al. (2024); (b): The 2nd and 3rd eigenvectors of M, as predicted by Theorem 2; (c): The first two principal components of the actual hidden representations from a pre-trained language model (llama3.1-8B), collected at context positions 2360… view at source ↗

**Figure 3.** The Normalized energy evolution across layers. Each “Component” curve shows the energy along a direction defined by the k-th eigenvector of M. We only show the first 5 components as an illustration. The “Overall” curve represents the average energy across all directions. To eliminate scale effects, the matrix Z(ℓ) is normalized to have unit Frobenius norm at each layer. 500 1000 1500 2000 2500 Timepoint 0 2 4 6 8 … view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Proportion of structured attention connections. For each attention head, we sum attention weights that falls into type A,B and T across all layers, and divide them by total attention weights (which is equal to the number of tokens per layer, since the attention weights…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages

[1]

A.1 E VENTS IN A RANDOM WALK SEQUENCE Theorem 3 (Theorem 1 in (Fan et al., 2021))

The proof start by identifying a high-probability event in the random walk sequence that ensures it is “regular” enough. A.1 E VENTS IN A RANDOM WALK SEQUENCE Theorem 3 (Theorem 1 in (Fan et al., 2021)). Given graph G = (V, E) with stationary distribution π, there exists constant C > 0 satisfies the following statement. Let {xi}∞ i=1 ∈ VN be a random walk...

work page 2021
[4]

Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559,

arXiv
[6]

Measuring mathematical problem solving with the math dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

arXiv
[8]

In-context convergence of transformers

Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05249,

arXiv
[9]

Semi-supervised classification with graph convolutional networks

12 Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907,

arXiv
[13]

Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka

https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. arXiv preprint arXiv:2501.00070,

arXiv 2022
[14]

Hopfield networks is all you need

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi´c, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217,

arXiv 2008
[15]

Spectral and algebraic graph theory, incomplete draft, dated december 4, 2019,

D Spielman. Spectral and algebraic graph theory, incomplete draft, dated december 4, 2019,

work page 2019

Show all 19 references

[17]

How transformers implement induction heads: Approximation and optimization analysis

Mingze Wang, Ruoxi Yu, Lei Wu, et al. How transformers implement induction heads: Approximation and optimization analysis. arXiv preprint arXiv:2410.11474,

arXiv
[19]

(γ1, γ2, Z , U ) and σ′ : Rd → Rd be a great mapping w.r.t.(γ′ 1, γ′ 2, Z , U ), then σ1 ◦ σ2 is a great mapping w.r.t.(γ1γ′ 1, γ2γ′ 2, Z , U )

Let σ : Rd → Rd be a great mapping w.r.t. (γ1, γ2, Z , U ) and σ′ : Rd → Rd be a great mapping w.r.t.(γ′ 1, γ′ 2, Z , U ), then σ1 ◦ σ2 is a great mapping w.r.t.(γ1γ′ 1, γ2γ′ 2, Z , U ). The proof of Lemma 13 is obvious. 20 D A G ENERALIZED FRAMEWORK THAT IS INDEPENDENT OF DAT...

2024
[2016]

Transformers meet in-context learning: A universal approximation theory

Gen Li, Yuchen Jiao, Yu Huang, Yuting Wei, and Yuxin Chen. Transformers meet in-context learning: A universal approximation theory. arXiv preprint arXiv:2506.05200,

arXiv
[2018]

Asymptotic theory of in-context learning by linear attention

Yue M Lu, Mary I Letey, Jacob A Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405.11751,
[2019]

Recurrent self-attention dynamics: An energy-agnostic perspective from jacobians

Akiyoshi Tomihari and Ryo Karakida. Recurrent self-attention dynamics: An energy-agnostic perspective from jacobians. arXiv preprint arXiv:2505.19458,
[2020]

Exploring the robustness of in-context learning with noisy labels

Chen Cheng, Xinzhi Yu, Haodong Wen, Jingsong Sun, Guanzhang Yue, Yihao Zhang, and Zeming Wei. Exploring the robustness of in-context learning with noisy labels. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

2025
[2021]

Hyperspherical energy transformer with recurrent depth

Yunzhe Hu, Difan Zou, and Dong Xu. Hyperspherical energy transformer with recurrent depth. arXiv preprint arXiv:2502.11646,

arXiv
[2022]

A mathematical perspec- tive on transformers

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspec- tive on transformers. arXiv preprint arXiv:2312.10794,

arXiv
[2023]

What learning algo- rithm is in-context learning? investigations with linear models

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022a. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What l...

arXiv
[2024]

Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837,

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837,

arXiv
[2025]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

1901

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

A.1 E VENTS IN A RANDOM WALK SEQUENCE Theorem 3 (Theorem 1 in (Fan et al., 2021))

The proof start by identifying a high-probability event in the random walk sequence that ensures it is “regular” enough. A.1 E VENTS IN A RANDOM WALK SEQUENCE Theorem 3 (Theorem 1 in (Fan et al., 2021)). Given graph G = (V, E) with stationary distribution π, there exists constant C > 0 satisfies the following statement. Let {xi}∞ i=1 ∈ VN be a random walk...

work page 2021

[2] [4]

Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559,

arXiv

[3] [6]

Measuring mathematical problem solving with the math dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

arXiv

[4] [8]

In-context convergence of transformers

Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05249,

arXiv

[5] [9]

Semi-supervised classification with graph convolutional networks

12 Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907,

arXiv

[6] [13]

Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka

https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. arXiv preprint arXiv:2501.00070,

arXiv 2022

[7] [14]

Hopfield networks is all you need

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi´c, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217,

arXiv 2008

[8] [15]

Spectral and algebraic graph theory, incomplete draft, dated december 4, 2019,

D Spielman. Spectral and algebraic graph theory, incomplete draft, dated december 4, 2019,

work page 2019