Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Cengiz Pehlevan; Jacob Zavatone-Veth; Mary Letey; Yue M. Lu

REVIEW 2 major objections 4 minor 42 references

Sequential correlations shrink usable context for in-context regression and favor softmax attention when the query is also correlated.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:51 UTC pith:GL7A3YIB

load-bearing objection Clean, usable extension of linear-attention ICL theory to weak sequential correlations, with a simple effective-length formula that matches experiment; softmax gap is empirical only. the 2 major comments →

arxiv 2607.03660 v1 pith:GL7A3YIB submitted 2026-07-04 stat.ML cs.LGstat.CO

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Mary Letey , Yue M. Lu , Cengiz Pehlevan , Jacob Zavatone-Veth This is my paper

classification stat.ML cs.LGstat.CO

keywords in-context learninglinear attentionsequential correlationseffective context lengthsoftmax attentionlinear regressionhigh-dimensional asymptotics

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most theory of in-context learning assumes independent examples inside a prompt. Real sequences are correlated. This paper builds a solvable linear-attention model of in-context linear regression with a correlation kernel among context tokens and an optional correlation between the query and those tokens. When the query is independent of the context, bulk correlations act like a reduction in sample size: performance matches ordinary i.i.d. prompts of a shorter effective length set by a single scalar that measures how anisotropic the correlation matrix is. When the query is itself correlated with the context, the same models gain extra predictive signal and error falls. Softmax attention captures more of that gain than linear attention, revealing an architectural mismatch. The closed-form results hold only for weak correlations that stay order-one as dimension and context length grow together; stronger, longer-range correlations can break concentration.

Core claim

For weakly correlated context tokens and an independent query, the in-context risk of the optimal reduced linear-attention predictor at length ℓ equals the uncorrelated risk evaluated at the strictly smaller effective length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ, where k₂ = tr(K^{2})/ℓ ≥ 1. When the query is also correlated with the context, risk falls further; the reduction is modest for linear attention and substantially larger for softmax attention.

What carries the argument

Effective context length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ (Eq. 1) together with the high-dimensional deterministic equivalent for the optimal reduced linear-attention parameters (Proposition 1), which separates bulk-context summaries (k₂) from query-context summaries (k₀, k₁).

Load-bearing premise

The math requires correlations to stay weak: the three summary statistics of the correlation kernel must remain order-one while dimension and context length both grow, so that the random matrices that define the predictor still concentrate.

What would settle it

Train the same reduced linear-attention model on exponential-kernel data with fixed dimension, increase correlation length until it is comparable to context length, and check whether measured ICL error continues to track the effective-length formula or diverges from it as predicted when concentration fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Clean, usable extension of linear-attention ICL theory to weak sequential correlations, with a simple effective-length formula that matches experiment; softmax gap is empirical only.

read the letter

The one thing worth knowing: for weakly correlated context tokens and an independent query, correlated ICL risk at length ℓ equals uncorrelated risk at ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ. That formula is new, simple, and matches both reduced and full linear attention under the stated regime. When the query is also correlated, risk drops further, more for softmax than for linear attention.

What is actually new is the clean separation of bulk context correlations (which only shrink effective sample size) from query-context correlations (which supply usable signal). The paper builds directly on the Lu et al. 2025 / Letey et al. resolvent setup, derives the population risk, the concentration condition tr(K⁴)/tr(K²)² → 0, and the asymptotic error (Prop. 1), and ships code. Theory curves track the linear-attention experiments; the strong-correlation breakdown is shown rather than papered over (Fig. 2). Citations to the classical correlated-ridge literature and to Markov/dynamical ICL work are appropriate; self-citations supply the uncorrelated baseline, not circular fitting.

Soft spots are real but already flagged by the authors. The closed form requires k₀, k₁, k₂ = Θ(1); once correlation length grows with ℓ the high-d concentration fails and the theory does not apply. The architectural-mismatch claim (softmax advantage) is purely empirical—no matching derivation—so it is an observation, not a theorem. Finite-d experiments use the usual proportional scalings and a single exponential kernel; that is fine for a first paper but limits generality. None of these undermine the claims under their stated hypotheses.

This is for people who already work on solvable ICL regression or who need a principled way to think about prompt structure. It is not a foundational rewrite of ICL, but it is solid, reproducible, and useful. I would send it to referees; the math and the linear-attention experiments deserve a careful look, and the open questions (strong correlations, softmax theory) are clearly marked.

Referee Report

2 major / 4 minor

Summary. The paper studies in-context linear regression with sequentially correlated tokens under a reduced linear-attention model. For weakly correlated context tokens and an independent query, it derives that ICL risk at context length ℓ equals uncorrelated risk at effective length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ with k₂ = tr(K²)/ℓ ≥ 1 (Eq. 1). When the query is also correlated with the context, risk decreases further; the reduction is larger for softmax attention than for linear attention (Prop. 1; Figs. 1, 3). The analysis uses high-dimensional random-matrix methods under the assumption k₀, k₁, k₂ = Θ(1), with an explicit concentration condition (App. D) and matching experiments on reduced/full linear attention and empirical softmax/MLP transformers.

Significance. The work cleanly separates bulk context correlations (effective sample-size reduction) from query–context correlations (extra predictive signal) in a standard ICL regression sandbox. The effective-length formula is simple, parameter-free given k₂, and falsifiable; theory curves match reduced and full linear attention under the stated weak-correlation regime, and code is released. The architectural-mismatch observation, though empirical, is a useful pointer that softmax may better exploit sequential structure. The paper is a solid incremental contribution that extends the Lu et al. / Letey et al. linear-attention RMT line to a basic feature of natural sequences, while honestly delimiting the strong-correlation regime where concentration fails.

major comments (2)

The architectural-mismatch claim (abstract, §4, Fig. 3) is load-bearing for the paper’s second headline result, yet it rests entirely on empirical comparisons of softmax vs. linear attention; there is no matching theory for softmax. The manuscript should either (i) supply a mechanistic or approximate analysis that explains why Δ_query is larger for softmax, or (ii) clearly reframe the claim as an empirical observation that motivates future theory, so that the abstract and conclusion do not overstate what is proven.
Appendix D and Fig. 2 correctly flag that concentration of Γ* and of ICL error can fail when correlation length grows with ℓ (tr(K⁴)/tr(K²)² ↛ 0). The main text (abstract, §3, conclusion) still presents the effective-length law as the primary takeaway without always restating the weak-correlation hypothesis k₀,k₁,k₂=Θ(1). A short, prominent caveat in the main text (e.g., after Eq. 1) would prevent readers from applying the formula outside its regime of validity.

minor comments (4)

Notation for the correlation summaries (k₀, k₁, k₂) is introduced in §2 but used heavily before the exponential-kernel specialisation; a one-line reminder that k₂ ≥ 1 by Jensen would help.
Figure 1 caption refers to “Eq. (1)” for effective context length; ensure the equation number is stable across arXiv and camera-ready versions.
Appendix A training hyperparameters (max_lr, lamb, batch size, seeds) are given, but the precise definition of “best checkpointed model” (early-stopping criterion) could be stated more explicitly for reproducibility.
A few typos: “withtransformers” (related work), occasional missing spaces around citations, and “findings” vs. “findings.” in the abstract.

Circularity Check

0 steps flagged

No significant circularity: effective-length and risk formulas are derived from the resolvent under stated weak-correlation assumptions; self-citations supply only the uncorrelated baseline and RMT setup.

full rationale

The central claim (Eq. 1 / Prop. 1) is obtained by applying random-matrix analysis to the optimal reduced linear-attention parameters Γ* under the explicit high-dimensional regime α=ℓ/d, τ=n/d², κ=k/d = Θ(1) and the weak-correlation restriction k0,k1,k2=Θ(1). The population risk (Lemma 1) and the deterministic equivalent for E_ICL(Γ*) (Prop. 1, Appendices E–G) are computed from the data second moments K and k_test; they are not fitted to the experimental curves. Self-citations to Lu et al. (2025) and Letey et al. (2026b) supply the uncorrelated baseline formula and the cavity/resolvent machinery, both of which are independent of the new correlation summaries k0,k1,k2. The paper itself flags the breakdown of concentration when tr(K⁴)/tr(K²)²↛0 (Appendix D, Fig. 2) and does not claim the closed form outside that regime. Architectural-mismatch statements for softmax attention are presented as empirical (Figs. 1, 3) and correctly lie outside the linear-attention theory. No step reduces a claimed prediction to a fitted input or to a self-definitional identity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Central claims rest on the standard high-d ICL-regression program plus a weak-correlation restriction and a reduced linear-attention predictor. Experimental free parameters fix the finite-d check; they do not enter the asymptotic formulas. No new physical entities are postulated—only summary statistics of a given PSD kernel K and query vector k_test.

free parameters (4)

label noise ρ = 0.01
Fixed at 0.01 in all experiments; appears symbolically in ℓ_eff and risk formulas but numerical value is a design choice.
finite-d scaling tuple (d,n,ℓ,k) = d=32, n=4096, ℓ=128, k=320
Chosen as d=32, n=4d², ℓ=4d, k=10d to match prior asymptotic regime; not fitted to the correlation effect.
training hyperparameters (max_lr, lamb, T, batch size) = max_lr=0.001, lamb=1e-4, T=10000, batch=64
Hand-chosen for convergence of trained models; theory for reduced LA does not use them.
explicit ridge λ (theory)
Appears in the optimal Γ*; often taken to ridgeless limit; effective ridge ˜λ is solved self-consistently, not fit to data.

axioms (5)

domain assumption Tokens are jointly Gaussian with second moments E[x_a x_b^T]=K_ab I_d/d and optional query coupling k_test; labels are linear with i.i.d. noise.
Section 2 / Appendix B; enables Wick expansions and RMT analysis.
domain assumption High-dimensional proportional limit d→∞ with α=ℓ/d, τ=n/d², κ=k/d all Θ(1); task and feature covariances have Θ(d) traces.
Standard ICL-regression scaling from Lu et al. 2025; used throughout Prop. 1.
ad hoc to paper Weak correlations: k₀,k₁,k₂=Θ(1) so that quadratic forms of vec(H) concentrate (tr(K⁴)/tr(K²)²→0).
Appendix D; necessary for deterministic equivalents; violated by strong sequential correlations.
domain assumption Reduced linear-attention predictor ŷ=tr(Γ H^T) with H built from query and context Gram features approximates full linear attention.
Section 2; inherited from prior linear-attention ICL theory; full LA is checked empirically.
standard math Random-matrix leave-one-out / cavity method yields deterministic equivalents for the resolvent of the empirical second-moment matrix of H_μ.
Appendix G; standard technique in this literature.

invented entities (2)

Effective context length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ independent evidence
purpose: Summarizes how bulk context correlations reduce usable ICL sample size for independent queries.
Derived quantity, not a postulated object; independent_evidence is the match to uncorrelated runs at length ℓ_eff (Fig. 1).
Correlation summaries k₀, k₁, k₂ independent evidence
purpose: Compress K and k_test into scalars that enter population risk and asymptotic error.
Definitions from the kernel; not free entities. independent_evidence true as they are computable from any chosen K.

pith-pipeline@v1.1.0-grok45 · 33249 in / 3563 out tokens · 30028 ms · 2026-07-12T00:51:04.663377+00:00 · methodology

0 comments

read the original abstract

Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Figures

Figures reproduced from arXiv: 2607.03660 by Cengiz Pehlevan, Jacob Zavatone-Veth, Mary Letey, Yue M. Lu.

**Figure 2.** Figure 2: Simulations of EICL(Γ∗ ) for several scaling regimes, compared with theory (Prop. 1). We analytically study the case of weak correlations with respect to context length ℓ: for the computation of theory curves we take k2, k0, k1 = Θℓ(1). We do not study stronger correlations in this work, corresponding to cases such as k2 = o(ℓ) or k2 ∝ ℓ, and do not attempt to characterise all of the ways in which the t… view at source ↗

**Figure 3.** Figure 3: Query effects depend on architecture. We now turn to the second effect in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages

[1]

Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

Pith/arXiv arXiv
[2]

Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan

URLhttps://arxiv.org/abs/2401.12973. Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Risk and cross validation in ridge regression with correlated samples. InForty-second International Conference on Machine Learning,

Pith/arXiv arXiv
[3]

Nicholas Barnfield, Subhabrata Sen, and Pragya Sur

URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/b2e63e36c57e153b9015fece2352a9f9-Paper-Conference.pdf. Nicholas Barnfield, Subhabrata Sen, and Pragya Sur. Multi-layer cross-attention is provably optimal for multi-modal in-context learning,

2023
[4]

Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou

URLhttps://arxiv.org/abs/2602.04872. Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a trans- former: A memory viewpoint. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.),Advances in Neural Information Processing Systems, volume 36, pp. 1560–1588. Cur- ran Associates, Inc.,

Pith/arXiv arXiv
[5]

Blake Bordelon, Mary I

URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ 0561738a239a995c8cd2ef0e50cfa4fd-Paper-Conference.pdf. Blake Bordelon, Mary I. Letey, and Cengiz Pehlevan. Theory of scaling laws for in-context regression: Depth, width, context and time,

2023
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

URLhttps://arxiv.org/abs/2510.01098. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

arXiv 1901
[7]

Stephanie C

URLhttps: //arxiv.org/abs/2303.12712. Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers,

Pith/arXiv arXiv
[8]

Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao

URLhttps://arxiv.org/abs/2204.02311. Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao. In-context learning of linear dynamical systems withtransformers: Errorboundsanddepth-separation,

Pith/arXiv arXiv
[9]

Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai

URLhttps://arxiv.org/abs/2502.08136. Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai. In-context operator learning on the space of probability measures,

arXiv
[10]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

URLhttps://arxiv.org/abs/2601.09979. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi De...

arXiv
[11]

Samet Demir and Zafer Dogan

URLhttps://arxiv.org/abs/2501.12948. Samet Demir and Zafer Dogan. How data mixing shapes in-context learning: Asymptotic equivalence for transformers with mlps. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.),Advances in Neural Information Processing Systems, volume 38, pp. 119346–119376. Cur- ran Associates, Inc.,

Pith/arXiv arXiv
[12]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui

URLhttps://proceedings.neurips.cc/paper_files/paper/2025/file/ ad184b4e096284a55cf66e1d4eeb29df-Paper-Conference.pdf. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning,

2025
[13]

Benjamin L

URL https://arxiv.org/abs/2301.00234. Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains,

Pith/arXiv arXiv
[14]

8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie

URLhttps://arxiv.org/abs/ 2402.11004. 8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam’s razor,

Pith/arXiv arXiv
[15]

Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant

URLhttps://arxiv.org/abs/2310.17086. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Advances in Neural Information Processing Systems,

Pith/arXiv arXiv
[16]

Halil Alperen Gozeten, M

URLhttps://arxiv.org/abs/2506.05574. Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-time training provably improves transformers as in-context learners,

arXiv
[17]

Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov

URL https://arxiv.org/abs/2503.11842. Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neu- ral Information Processing Systems, volume 37, pp. ...

arXiv
[18]

URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf

doi: 10.52202/079017-0422. URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers,

work page doi:10.52202/079017-0422 2024
[19]

Mary Letey, Yue M

URLhttps://openreview.net/forum?id=oo5TNikeJl. Mary Letey, Yue M. Lu, Cengiz Pehlevan, and Jacob A Zavatone-Veth. Sequential correlations change in-context learning: Effective context length and architectural mismatch. InHigh-dimensional Learning Dynamics 2026, 2026a. URLhttps://openreview.net/forum?id=AYprbQxAbI. Mary Letey, Jacob A Zavatone-Veth, Yue M....

Pith/arXiv arXiv 2026
[20]

Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond

Yingcong Li, Ankit Singh Rawat, and Samet Oymak. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neural Information Processing Systems, volume 37, pp. 138324–138364. Curran Associates, Inc., 2024b. URLhttps://proce...

2024
[21]

9 Toni J.b

URLhttps://arxiv.org/abs/2505.06475. 9 Toni J.b. Liu, Nicolas Boulle, Raphaël Sarfati, and Christopher Earls. Llms learn governing principles of dynamical systems, revealing an in-context neural scaling law. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15097–15117. Association for Computational Lin- guistics,

Pith/arXiv arXiv 2024
[22]

URLhttp://dx.doi.org/10.18653/v1/2024

doi: 10.18653/v1/2024.emnlp-main.842. URLhttp://dx.doi.org/10.18653/v1/2024. emnlp-main.842. Yue M. Lu, Mary Letey, Jacob A. Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention.Proceedings of the National Academy of Sciences, 122(28): e2502599122,

doi:10.18653/v1/2024.emnlp-main.842 2024
[23]

URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122

doi: 10.1073/pnas.2502599122. URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122. Arvind Mahankali, Tatsunori B. Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention,

doi:10.1073/pnas.2502599122
[24]

Behrad Moniri and Hamed Hassani

URLhttps://arxiv.org/abs/ 2307.03576. Behrad Moniri and Hamed Hassani. Asymptotics of linear regression with linearly dependent data. In Necmiye Ozay, Laura Balzano, Dimitra Panagou, and Alessandro Abate (eds.),Proceedings of the 7th Annual Learning for Dynamics & Control Conference, volume 283 ofProceedings of Machine Learning Research, pp. 72–85. PMLR, ...

Pith/arXiv arXiv
[25]

URLhttps://arxiv.org/abs/2411.02544. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack C...

Pith/arXiv arXiv
[26]

Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka

URLhttps://transformer-circuits.pub/ 2022/in-context-learning-and-induction-heads/index.html. Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning,

2022
[27]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al

URLhttps://arxiv.org/abs/2412.01003. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9,

Pith/arXiv arXiv
[28]

Gautam Reddy

URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/2e10b2c2e1aa4f8083c37dfe269873f8-Paper-Conference.pdf. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context clas- sification task. InThe Twelfth International Conference on Learning Representations,

2023
[29]

Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao

doi: 10.1002/j.1538-7305.1948.tb01338.x. Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao. Understanding in-context learning on structured manifolds: Bridging attention to kernel methods,

doi:10.1002/j.1538-7305.1948.tb01338.x 1948
[30]

Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau

URLhttps://arxiv.org/abs/2502.10390. Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau. In-context algebra,

Pith/arXiv arXiv
[31]

William L

URLhttps: //arxiv.org/abs/2512.16902. William L. Tong and Cengiz Pehlevan. MLPs learn in-context on regression and classification tasks. InThe Thirteenth International Conference on Learning Representations,

arXiv
[32]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

URLhttps://openreview.net/ forum?id=MbX0t1rUlp. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

Pith/arXiv arXiv
[33]

Transformers learn low sensitivity functions: Investigations and implications

Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Transformers learn low sensitivity functions: Investigations and implications. InInternational Conference on Learning Representations, volume 2025, pp. 3565–3594,

2025
[34]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H

URLhttps: //proceedings.mlr.press/v202/von-oswald23a.html. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.Transactions on Ma- chi...

Pith/arXiv arXiv
[35]

Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett

URLhttps://arxiv.org/abs/2303.03846. Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? InThe Twelfth International Conference on Learning Representations,

Pith/arXiv arXiv
[36]

Ruiqi Zhang, Spencer Frei, and Peter L

URLhttps://arxiv.org/abs/2510.19753. Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context.Jour- nal of Machine Learning Research, 25(49):1–55, 2024a. URLhttp://jmlr.org/papers/v25/23-1042. html. Ruiqi Zhang, Jingfeng Wu, and Peter L. Bartlett. In-context learning of a linear transformer block: Benefits of t...

arXiv
[37]

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang

URLhttps://arxiv.org/abs/2501.16265. Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization,

Pith/arXiv arXiv
[38]

12 A Experimental details The data structure follows the Section 2 setup

URLhttps://arxiv.org/ abs/2305.19420. 12 A Experimental details The data structure follows the Section 2 setup. For all experiments we choose a stationary kernel,i.e., distance-dependent correlations, given by Kab := exp(−|a−b|/ξ).(2) Here the correlation lengthξis manifestly part of the definition ofK; whenξ= 0,K=Iℓ. This definition is convenient due to ...

Pith/arXiv arXiv 2025
[39]

correlation length

B Detailed setup Here we reiterate the data and model setup given in the main document with more details. We leave the token design matrix general,i.e., xa∼KN(0,Σ/d) where the sample-sample correlations are given by positive semi-definiteKas EX[XaiXbj] = 1 dKabΣij fora,b∈[ℓ]andi,j∈[d].(4) 13 The label noise we take to be sequentially uncorrelated ε∼N(0,ρI...

2025
[40]

task diversity

In the maximally-correlated caseK=11 ⊤, we see thatA=Z 2 11 1 d ∑d i=2Z2 1i tends in distribution to a χ2 random variable with 1 degree of freedom (as1 d ∑d i=2Z2 1i→1almost surely), which clearly does not concentrate. This gives us an intuitive heuristic for concentration of∥vec(H)∥: Necessary condition for concentration Quadratic forms invec(H)converge ...

2025
[41]

Throughout this entire section we will takeΣ =Id

The methodology in this section is to analyse the random objectΓ∗, which contains randomness through the particular training batch(Hµ,yµ test)n µ=1, as a resolvent of a larger random matrix, and the necessarytr ( ΓA⊤) andtr ( Γ⊤BΓ ) terms as traces of a resolvent against deterministic test matrices. Throughout this entire section we will takeΣ =Id. This s...

2025
[42]

leave-one-out

for the optimal parameters vec(Γ∗) = ( n dλI+ n∑ µ=1 vec(Hµ)vec(Hµ)⊤ )−1n∑ µ=1 yµ testvec(Hµ)(10) forµ∈[n]denoting the index for thentraining sample sequences. Define zµ= [ yµ test/d vec(Hµ)/ √ d ] ∈Rd(d+1)+1 and construct extended resolvent Gext(π) = 1∑ µ∈[n]zµz⊤µ+πBext +τλI.(11) for some test matrixBext∈R(d(d+1)+1)×(d(d+1)+1). It’s important to note tha...

2025

[1] [1]

Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

Pith/arXiv arXiv

[2] [2]

Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan

URLhttps://arxiv.org/abs/2401.12973. Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Risk and cross validation in ridge regression with correlated samples. InForty-second International Conference on Machine Learning,

Pith/arXiv arXiv

[3] [3]

Nicholas Barnfield, Subhabrata Sen, and Pragya Sur

URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/b2e63e36c57e153b9015fece2352a9f9-Paper-Conference.pdf. Nicholas Barnfield, Subhabrata Sen, and Pragya Sur. Multi-layer cross-attention is provably optimal for multi-modal in-context learning,

2023

[4] [4]

Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou

URLhttps://arxiv.org/abs/2602.04872. Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a trans- former: A memory viewpoint. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.),Advances in Neural Information Processing Systems, volume 36, pp. 1560–1588. Cur- ran Associates, Inc.,

Pith/arXiv arXiv

[5] [5]

Blake Bordelon, Mary I

URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ 0561738a239a995c8cd2ef0e50cfa4fd-Paper-Conference.pdf. Blake Bordelon, Mary I. Letey, and Cengiz Pehlevan. Theory of scaling laws for in-context regression: Depth, width, context and time,

2023

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

URLhttps://arxiv.org/abs/2510.01098. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

arXiv 1901

[7] [7]

Stephanie C

URLhttps: //arxiv.org/abs/2303.12712. Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers,

Pith/arXiv arXiv

[8] [8]

Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao

URLhttps://arxiv.org/abs/2204.02311. Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao. In-context learning of linear dynamical systems withtransformers: Errorboundsanddepth-separation,

Pith/arXiv arXiv

[9] [9]

Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai

URLhttps://arxiv.org/abs/2502.08136. Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai. In-context operator learning on the space of probability measures,

arXiv

[10] [10]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

URLhttps://arxiv.org/abs/2601.09979. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi De...

arXiv

[11] [11]

Samet Demir and Zafer Dogan

URLhttps://arxiv.org/abs/2501.12948. Samet Demir and Zafer Dogan. How data mixing shapes in-context learning: Asymptotic equivalence for transformers with mlps. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.),Advances in Neural Information Processing Systems, volume 38, pp. 119346–119376. Cur- ran Associates, Inc.,

Pith/arXiv arXiv

[12] [12]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui

URLhttps://proceedings.neurips.cc/paper_files/paper/2025/file/ ad184b4e096284a55cf66e1d4eeb29df-Paper-Conference.pdf. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning,

2025

[13] [13]

Benjamin L

URL https://arxiv.org/abs/2301.00234. Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains,

Pith/arXiv arXiv

[14] [14]

8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie

URLhttps://arxiv.org/abs/ 2402.11004. 8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam’s razor,

Pith/arXiv arXiv

[15] [15]

Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant

URLhttps://arxiv.org/abs/2310.17086. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Advances in Neural Information Processing Systems,

Pith/arXiv arXiv

[16] [16]

Halil Alperen Gozeten, M

URLhttps://arxiv.org/abs/2506.05574. Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-time training provably improves transformers as in-context learners,

arXiv

[17] [17]

Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov

URL https://arxiv.org/abs/2503.11842. Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neu- ral Information Processing Systems, volume 37, pp. ...

arXiv

[18] [18]

URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf

doi: 10.52202/079017-0422. URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers,

work page doi:10.52202/079017-0422 2024

[19] [19]

Mary Letey, Yue M

URLhttps://openreview.net/forum?id=oo5TNikeJl. Mary Letey, Yue M. Lu, Cengiz Pehlevan, and Jacob A Zavatone-Veth. Sequential correlations change in-context learning: Effective context length and architectural mismatch. InHigh-dimensional Learning Dynamics 2026, 2026a. URLhttps://openreview.net/forum?id=AYprbQxAbI. Mary Letey, Jacob A Zavatone-Veth, Yue M....

Pith/arXiv arXiv 2026

[20] [20]

Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond

Yingcong Li, Ankit Singh Rawat, and Samet Oymak. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neural Information Processing Systems, volume 37, pp. 138324–138364. Curran Associates, Inc., 2024b. URLhttps://proce...

2024

[21] [21]

9 Toni J.b

URLhttps://arxiv.org/abs/2505.06475. 9 Toni J.b. Liu, Nicolas Boulle, Raphaël Sarfati, and Christopher Earls. Llms learn governing principles of dynamical systems, revealing an in-context neural scaling law. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15097–15117. Association for Computational Lin- guistics,

Pith/arXiv arXiv 2024

[22] [22]

URLhttp://dx.doi.org/10.18653/v1/2024

doi: 10.18653/v1/2024.emnlp-main.842. URLhttp://dx.doi.org/10.18653/v1/2024. emnlp-main.842. Yue M. Lu, Mary Letey, Jacob A. Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention.Proceedings of the National Academy of Sciences, 122(28): e2502599122,

doi:10.18653/v1/2024.emnlp-main.842 2024

[23] [23]

URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122

doi: 10.1073/pnas.2502599122. URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122. Arvind Mahankali, Tatsunori B. Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention,

doi:10.1073/pnas.2502599122

[24] [24]

Behrad Moniri and Hamed Hassani

URLhttps://arxiv.org/abs/ 2307.03576. Behrad Moniri and Hamed Hassani. Asymptotics of linear regression with linearly dependent data. In Necmiye Ozay, Laura Balzano, Dimitra Panagou, and Alessandro Abate (eds.),Proceedings of the 7th Annual Learning for Dynamics & Control Conference, volume 283 ofProceedings of Machine Learning Research, pp. 72–85. PMLR, ...

Pith/arXiv arXiv

[25] [25]

URLhttps://arxiv.org/abs/2411.02544. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack C...

Pith/arXiv arXiv

[26] [26]

Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka

URLhttps://transformer-circuits.pub/ 2022/in-context-learning-and-induction-heads/index.html. Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning,

2022

[27] [27]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al

URLhttps://arxiv.org/abs/2412.01003. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9,

Pith/arXiv arXiv

[28] [28]

Gautam Reddy

URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/2e10b2c2e1aa4f8083c37dfe269873f8-Paper-Conference.pdf. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context clas- sification task. InThe Twelfth International Conference on Learning Representations,

2023

[29] [29]

Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao

doi: 10.1002/j.1538-7305.1948.tb01338.x. Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao. Understanding in-context learning on structured manifolds: Bridging attention to kernel methods,

doi:10.1002/j.1538-7305.1948.tb01338.x 1948

[30] [30]

Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau

URLhttps://arxiv.org/abs/2502.10390. Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau. In-context algebra,

Pith/arXiv arXiv

[31] [31]

William L

URLhttps: //arxiv.org/abs/2512.16902. William L. Tong and Cengiz Pehlevan. MLPs learn in-context on regression and classification tasks. InThe Thirteenth International Conference on Learning Representations,

arXiv

[32] [32]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

URLhttps://openreview.net/ forum?id=MbX0t1rUlp. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

Pith/arXiv arXiv

[33] [33]

Transformers learn low sensitivity functions: Investigations and implications

Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Transformers learn low sensitivity functions: Investigations and implications. InInternational Conference on Learning Representations, volume 2025, pp. 3565–3594,

2025

[34] [34]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H

URLhttps: //proceedings.mlr.press/v202/von-oswald23a.html. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.Transactions on Ma- chi...

Pith/arXiv arXiv

[35] [35]

Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett

URLhttps://arxiv.org/abs/2303.03846. Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? InThe Twelfth International Conference on Learning Representations,

Pith/arXiv arXiv

[36] [36]

Ruiqi Zhang, Spencer Frei, and Peter L

URLhttps://arxiv.org/abs/2510.19753. Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context.Jour- nal of Machine Learning Research, 25(49):1–55, 2024a. URLhttp://jmlr.org/papers/v25/23-1042. html. Ruiqi Zhang, Jingfeng Wu, and Peter L. Bartlett. In-context learning of a linear transformer block: Benefits of t...

arXiv

[37] [37]

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang

URLhttps://arxiv.org/abs/2501.16265. Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization,

Pith/arXiv arXiv

[38] [38]

12 A Experimental details The data structure follows the Section 2 setup

URLhttps://arxiv.org/ abs/2305.19420. 12 A Experimental details The data structure follows the Section 2 setup. For all experiments we choose a stationary kernel,i.e., distance-dependent correlations, given by Kab := exp(−|a−b|/ξ).(2) Here the correlation lengthξis manifestly part of the definition ofK; whenξ= 0,K=Iℓ. This definition is convenient due to ...

Pith/arXiv arXiv 2025

[39] [39]

correlation length

B Detailed setup Here we reiterate the data and model setup given in the main document with more details. We leave the token design matrix general,i.e., xa∼KN(0,Σ/d) where the sample-sample correlations are given by positive semi-definiteKas EX[XaiXbj] = 1 dKabΣij fora,b∈[ℓ]andi,j∈[d].(4) 13 The label noise we take to be sequentially uncorrelated ε∼N(0,ρI...

2025

[40] [40]

task diversity

In the maximally-correlated caseK=11 ⊤, we see thatA=Z 2 11 1 d ∑d i=2Z2 1i tends in distribution to a χ2 random variable with 1 degree of freedom (as1 d ∑d i=2Z2 1i→1almost surely), which clearly does not concentrate. This gives us an intuitive heuristic for concentration of∥vec(H)∥: Necessary condition for concentration Quadratic forms invec(H)converge ...

2025

[41] [41]

Throughout this entire section we will takeΣ =Id

The methodology in this section is to analyse the random objectΓ∗, which contains randomness through the particular training batch(Hµ,yµ test)n µ=1, as a resolvent of a larger random matrix, and the necessarytr ( ΓA⊤) andtr ( Γ⊤BΓ ) terms as traces of a resolvent against deterministic test matrices. Throughout this entire section we will takeΣ =Id. This s...

2025

[42] [42]

leave-one-out

for the optimal parameters vec(Γ∗) = ( n dλI+ n∑ µ=1 vec(Hµ)vec(Hµ)⊤ )−1n∑ µ=1 yµ testvec(Hµ)(10) forµ∈[n]denoting the index for thentraining sample sequences. Define zµ= [ yµ test/d vec(Hµ)/ √ d ] ∈Rd(d+1)+1 and construct extended resolvent Gext(π) = 1∑ µ∈[n]zµz⊤µ+πBext +τλI.(11) for some test matrixBext∈R(d(d+1)+1)×(d(d+1)+1). It’s important to note tha...

2025