Pith. sign in

REVIEW 2 major objections 4 minor 42 references

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

T0 review · 2 major / 4 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sequential correlations shrink usable context for in-context regression and favor softmax attention when the query is also correlated.

desk verdict Clean, usable extension of linear-attention ICL theory to weak sequential correlations, with a simple effective-length formula that matches experiment; softmax gap is empirical only. read the letter →

arxiv 2607.03660 v1 pith:GL7A3YIB submitted 2026-07-04 stat.ML cs.LGstat.CO

Mary Letey , Yue M. Lu , Cengiz Pehlevan , Jacob Zavatone-Veth This is my paper
classification stat.MLcs.LGstat.CO
keywords in-contextlearninglinearattentionsequentialcorrelationseffectivecontextlengthsoftmaxregressionhigh-dimensionalasymptotics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most theory of in-context learning assumes independent examples inside a prompt. Real sequences are correlated. This paper builds a solvable linear-attention model of in-context linear regression with a correlation kernel among context tokens and an optional correlation between the query and those tokens. When the query is independent of the context, bulk correlations act like a reduction in sample size: performance matches ordinary i.i.d. prompts of a shorter effective length set by a single scalar that measures how anisotropic the correlation matrix is. When the query is itself correlated with the context, the same models gain extra predictive signal and error falls. Softmax attention captures more of that gain than linear attention, revealing an architectural mismatch. The closed-form results hold only for weak correlations that stay order-one as dimension and context length grow together; stronger, longer-range correlations can break concentration.

What carries the argument

Effective context length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ (Eq. 1) together with the high-dimensional deterministic equivalent for the optimal reduced linear-attention parameters (Proposition 1), which separates bulk-context summaries (k₂) from query-context summaries (k₀, k₁).

What would settle it

Train the same reduced linear-attention model on exponential-kernel data with fixed dimension, increase correlation length until it is comparable to context length, and check whether measured ICL error continues to track the effective-length formula or diverges from it as predicted when concentration fails.

Watch

Extended reading notes

Core claim

For weakly correlated context tokens and an independent query, the in-context risk of the optimal reduced linear-attention predictor at length ℓ equals the uncorrelated risk evaluated at the strictly smaller effective length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ, where k₂ = tr(K^{2})/ℓ ≥ 1. When the query is also correlated with the context, risk falls further; the reduction is modest for linear attention and substantially larger for softmax attention.

Load-bearing premise

The math requires correlations to stay weak: the three summary statistics of the correlation kernel must remain order-one while dimension and context length both grow, so that the random matrices that define the predictor still concentrate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies in-context linear regression with sequentially correlated tokens under a reduced linear-attention model. For weakly correlated context tokens and an independent query, it derives that ICL risk at context length ℓ equals uncorrelated risk at effective length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ with k₂ = tr(K²)/ℓ ≥ 1 (Eq. 1). When the query is also correlated with the context, risk decreases further; the reduction is larger for softmax attention than for linear attention (Prop. 1; Figs. 1, 3). The analysis uses high-dimensional random-matrix methods under the assumption k₀, k₁, k₂ = Θ(1), with an explicit concentration condition (App. D) and matching experiments on reduced/full linear attention and empirical softmax/MLP transformers.

Significance. The work cleanly separates bulk context correlations (effective sample-size reduction) from query–context correlations (extra predictive signal) in a standard ICL regression sandbox. The effective-length formula is simple, parameter-free given k₂, and falsifiable; theory curves match reduced and full linear attention under the stated weak-correlation regime, and code is released. The architectural-mismatch observation, though empirical, is a useful pointer that softmax may better exploit sequential structure. The paper is a solid incremental contribution that extends the Lu et al. / Letey et al. linear-attention RMT line to a basic feature of natural sequences, while honestly delimiting the strong-correlation regime where concentration fails.

major comments (2)
  1. The architectural-mismatch claim (abstract, §4, Fig. 3) is load-bearing for the paper’s second headline result, yet it rests entirely on empirical comparisons of softmax vs. linear attention; there is no matching theory for softmax. The manuscript should either (i) supply a mechanistic or approximate analysis that explains why Δ_query is larger for softmax, or (ii) clearly reframe the claim as an empirical observation that motivates future theory, so that the abstract and conclusion do not overstate what is proven.
  2. Appendix D and Fig. 2 correctly flag that concentration of Γ* and of ICL error can fail when correlation length grows with ℓ (tr(K⁴)/tr(K²)² ↛ 0). The main text (abstract, §3, conclusion) still presents the effective-length law as the primary takeaway without always restating the weak-correlation hypothesis k₀,k₁,k₂=Θ(1). A short, prominent caveat in the main text (e.g., after Eq. 1) would prevent readers from applying the formula outside its regime of validity.
minor comments (4)
  1. Notation for the correlation summaries (k₀, k₁, k₂) is introduced in §2 but used heavily before the exponential-kernel specialisation; a one-line reminder that k₂ ≥ 1 by Jensen would help.
  2. Figure 1 caption refers to “Eq. (1)” for effective context length; ensure the equation number is stable across arXiv and camera-ready versions.
  3. Appendix A training hyperparameters (max_lr, lamb, batch size, seeds) are given, but the precise definition of “best checkpointed model” (early-stopping criterion) could be stated more explicitly for reproducibility.
  4. A few typos: “withtransformers” (related work), occasional missing spaces around citations, and “findings” vs. “findings.” in the abstract.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: effective-length and risk formulas are derived from the resolvent under stated weak-correlation assumptions; self-citations supply only the uncorrelated baseline and RMT setup.

full rationale

The central claim (Eq. 1 / Prop. 1) is obtained by applying random-matrix analysis to the optimal reduced linear-attention parameters Γ* under the explicit high-dimensional regime α=ℓ/d, τ=n/d², κ=k/d = Θ(1) and the weak-correlation restriction k0,k1,k2=Θ(1). The population risk (Lemma 1) and the deterministic equivalent for E_ICL(Γ*) (Prop. 1, Appendices E–G) are computed from the data second moments K and k_test; they are not fitted to the experimental curves. Self-citations to Lu et al. (2025) and Letey et al. (2026b) supply the uncorrelated baseline formula and the cavity/resolvent machinery, both of which are independent of the new correlation summaries k0,k1,k2. The paper itself flags the breakdown of concentration when tr(K⁴)/tr(K²)²↛0 (Appendix D, Fig. 2) and does not claim the closed form outside that regime. Architectural-mismatch statements for softmax attention are presented as empirical (Figs. 1, 3) and correctly lie outside the linear-attention theory. No step reduces a claimed prediction to a fitted input or to a self-definitional identity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

Central claims rest on the standard high-d ICL-regression program plus a weak-correlation restriction and a reduced linear-attention predictor. Experimental free parameters fix the finite-d check; they do not enter the asymptotic formulas. No new physical entities are postulated—only summary statistics of a given PSD kernel K and query vector k_test.

free parameters (4)
  • label noise ρ = 0.01
    Fixed at 0.01 in all experiments; appears symbolically in ℓ_eff and risk formulas but numerical value is a design choice.
  • finite-d scaling tuple (d,n,ℓ,k) = d=32, n=4096, ℓ=128, k=320
    Chosen as d=32, n=4d², ℓ=4d, k=10d to match prior asymptotic regime; not fitted to the correlation effect.
  • training hyperparameters (max_lr, lamb, T, batch size) = max_lr=0.001, lamb=1e-4, T=10000, batch=64
    Hand-chosen for convergence of trained models; theory for reduced LA does not use them.
  • explicit ridge λ (theory)
    Appears in the optimal Γ*; often taken to ridgeless limit; effective ridge ˜λ is solved self-consistently, not fit to data.
assumptions (5)
  • domain assumption Tokens are jointly Gaussian with second moments E[x_a x_b^T]=K_ab I_d/d and optional query coupling k_test; labels are linear with i.i.d. noise.
    Section 2 / Appendix B; enables Wick expansions and RMT analysis.
  • domain assumption High-dimensional proportional limit d→∞ with α=ℓ/d, τ=n/d², κ=k/d all Θ(1); task and feature covariances have Θ(d) traces.
    Standard ICL-regression scaling from Lu et al. 2025; used throughout Prop. 1.
  • ad hoc to paper Weak correlations: k₀,k₁,k₂=Θ(1) so that quadratic forms of vec(H) concentrate (tr(K⁴)/tr(K²)²→0).
    Appendix D; necessary for deterministic equivalents; violated by strong sequential correlations.
  • domain assumption Reduced linear-attention predictor ŷ=tr(Γ H^T) with H built from query and context Gram features approximates full linear attention.
    Section 2; inherited from prior linear-attention ICL theory; full LA is checked empirically.
  • standard math Random-matrix leave-one-out / cavity method yields deterministic equivalents for the resolvent of the empirical second-moment matrix of H_μ.
    Appendix G; standard technique in this literature.
invented entities (2)
  • Effective context length ℓ_eff = ((1+ρ)/(k₂+ρ))ℓ independent evidence
    purpose: Summarizes how bulk context correlations reduce usable ICL sample size for independent queries.
    Derived quantity, not a postulated object; independent_evidence is the match to uncorrelated runs at length ℓ_eff (Fig. 1).
  • Correlation summaries k₀, k₁, k₂ independent evidence
    purpose: Compress K and k_test into scalars that enter population risk and asymptotic error.
    Definitions from the kernel; not free entities. independent_evidence true as they are computable from any chosen K.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch." pith.science (2026). https://pith.science/paper/GL7A3YIB

@misc{pith2026260703660,
  author       = {Pith},
  title        = {Pith review of: Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GL7A3YIB}},
  note         = {Machine review of arXiv:2607.03660}
}
read the original abstract

Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Figures

Figures reproduced from arXiv: 2607.03660 by the authors.

Figure 1
Figure 1. ICL test loss against token correlation strength for three attention architectures. We highlight two [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Simulations of EICL(Γ∗ ) for several scaling regimes, compared with theory (Prop. 1). We analytically study the case of weak cor￾relations with respect to context length ℓ: for the computation of theory curves we take k2, k0, k1 = Θℓ(1). We do not study stronger correlations in this work, corre￾sponding to cases such as k2 = o(ℓ) or k2 ∝ ℓ, and do not attempt to charac￾terise all of the ways in which the the￾ory’s p… view at source ↗
Figure 3
Figure 3. Query effects depend on architecture. We now turn to the second effect in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan

    URLhttps://arxiv.org/abs/2401.12973. Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Risk and cross validation in ridge regression with correlated samples. InForty-second International Conference on Machine Learning,

    arXiv

  3. [3]

    Nicholas Barnfield, Subhabrata Sen, and Pragya Sur

    URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/b2e63e36c57e153b9015fece2352a9f9-Paper-Conference.pdf. Nicholas Barnfield, Subhabrata Sen, and Pragya Sur. Multi-layer cross-attention is provably optimal for multi-modal in-context learning,

    2023

  4. [4]

    Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou

    URLhttps://arxiv.org/abs/2602.04872. Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a trans- former: A memory viewpoint. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.),Advances in Neural Information Processing Systems, volume 36, pp. 1560–1588. Cur- ran Associates, Inc.,

    arXiv

  5. [5]

    Blake Bordelon, Mary I

    URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ 0561738a239a995c8cd2ef0e50cfa4fd-Paper-Conference.pdf. Blake Bordelon, Mary I. Letey, and Cengiz Pehlevan. Theory of scaling laws for in-context regression: Depth, width, context and time,

    2023

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    URLhttps://arxiv.org/abs/2510.01098. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    arXiv 1901

  7. [7]

    Stephanie C

    URLhttps: //arxiv.org/abs/2303.12712. Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers,

    arXiv

  8. [8]

    Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao

    URLhttps://arxiv.org/abs/2204.02311. Frank Cole, Yulong Lu, Tianhao Zhang, and Yuxuan Zhao. In-context learning of linear dynamical systems withtransformers: Errorboundsanddepth-separation,

    arXiv

Show all 42 references
  1. [9]

    Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai

    URLhttps://arxiv.org/abs/2502.08136. Frank Cole, Dixi Wang, Yineng Chen, Yulong Lu, and Rongjie Lai. In-context operator learning on the space of probability measures,

  2. [10]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    URLhttps://arxiv.org/abs/2601.09979. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing...

  3. [11]

    Samet Demir and Zafer Dogan

    URLhttps://arxiv.org/abs/2501.12948. Samet Demir and Zafer Dogan. How data mixing shapes in-context learning: Asymptotic equivalence for transformers with mlps. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.),Advances in Neural Informa...

    arXiv

  4. [12]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui

    URLhttps://proceedings.neurips.cc/paper_files/paper/2025/file/ ad184b4e096284a55cf66e1d4eeb29df-Paper-Conference.pdf. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui....

    2025

  5. [13]

    Benjamin L

    URL https://arxiv.org/abs/2301.00234. Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains,

    arXiv

  6. [14]

    8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie

    URLhttps://arxiv.org/abs/ 2402.11004. 8 Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam’s razor,

    arXiv

  7. [15]

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant

    URLhttps://arxiv.org/abs/2310.17086. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.),Advances in Neural ...

    arXiv

  8. [16]

    Halil Alperen Gozeten, M

    URLhttps://arxiv.org/abs/2506.05574. Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-time training provably improves transformers as in-context learners,

  9. [17]

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov

    URL https://arxiv.org/abs/2503.11842. Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Z...

  10. [18]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf

    doi: 10.52202/079017-0422. URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 17d60fef592086d1a5cb136f1946df59-Paper-Conference.pdf. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers,

    2024 doi

  11. [19]

    Mary Letey, Yue M

    URLhttps://openreview.net/forum?id=oo5TNikeJl. Mary Letey, Yue M. Lu, Cengiz Pehlevan, and Jacob A Zavatone-Veth. Sequential correlations change in-context learning: Effective context length and architectural mismatch. InHigh-dimensional Learning Dynamics 2026, 2026a. URLhttps...

    2026 arXiv

  12. [20]

    Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond

    Yingcong Li, Ankit Singh Rawat, and Samet Oymak. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neural Information Processing Systems...

    2024

  13. [21]

    9 Toni J.b

    URLhttps://arxiv.org/abs/2505.06475. 9 Toni J.b. Liu, Nicolas Boulle, Raphaël Sarfati, and Christopher Earls. Llms learn governing principles of dynamical systems, revealing an in-context neural scaling law. InProceedings of the 2024 Conference on Empirical Methods in Natural ...

    2024 arXiv

  14. [22]

    URLhttp://dx.doi.org/10.18653/v1/2024

    doi: 10.18653/v1/2024.emnlp-main.842. URLhttp://dx.doi.org/10.18653/v1/2024. emnlp-main.842. Yue M. Lu, Mary Letey, Jacob A. Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention.Proceedings of the National Academy of ...

    2024 doi

  15. [23]

    URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122

    doi: 10.1073/pnas.2502599122. URLhttps://www.pnas.org/doi/abs/10.1073/ pnas.2502599122. Arvind Mahankali, Tatsunori B. Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention,

    doi

  16. [24]

    Behrad Moniri and Hamed Hassani

    URLhttps://arxiv.org/abs/ 2307.03576. Behrad Moniri and Hamed Hassani. Asymptotics of linear regression with linearly dependent data. In Necmiye Ozay, Laura Balzano, Dimitra Panagou, and Alessandro Abate (eds.),Proceedings of the 7th Annual Learning for Dynamics & Control Conf...

    arXiv

  17. [25]

    URLhttps://arxiv.org/abs/2411.02544. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jone...

    arXiv

  18. [26]

    Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka

    URLhttps://transformer-circuits.pub/ 2022/in-context-learning-and-induction-heads/index.html. Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning,

    2022

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al

    URLhttps://arxiv.org/abs/2412.01003. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9,

    arXiv

  20. [28]

    Gautam Reddy

    URLhttps://proceedings.neurips.cc/paper_files/ paper/2023/file/2e10b2c2e1aa4f8083c37dfe269873f8-Paper-Conference.pdf. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context clas- sification task. InThe Twelfth International Conference on Le...

    2023

  21. [29]

    Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao

    doi: 10.1002/j.1538-7305.1948.tb01338.x. Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao. Understanding in-context learning on structured manifolds: Bridging attention to kernel methods,

    1948 doi

  22. [30]

    Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau

    URLhttps://arxiv.org/abs/2502.10390. Eric Todd, Jannik Brinkmann, Rohit Gandikota, and David Bau. In-context algebra,

    arXiv

  23. [31]

    William L

    URLhttps: //arxiv.org/abs/2512.16902. William L. Tong and Cengiz Pehlevan. MLPs learn in-context on regression and classification tasks. InThe Thirteenth International Conference on Learning Representations,

  24. [32]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

    URLhttps://openreview.net/ forum?id=MbX0t1rUlp. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv pre...

    arXiv

  25. [33]

    Transformers learn low sensitivity functions: Investigations and implications

    Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Transformers learn low sensitivity functions: Investigations and implications. InInternational Conference on Learning Representations, volume 2025, pp. 3565–3594,

    2025

  26. [34]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H

    URLhttps: //proceedings.mlr.press/v202/von-oswald23a.html. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and Wi...

    arXiv

  27. [35]

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett

    URLhttps://arxiv.org/abs/2303.03846. Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? InThe Twelfth International Conference on Learning Representations,

    arXiv

  28. [36]

    Ruiqi Zhang, Spencer Frei, and Peter L

    URLhttps://arxiv.org/abs/2510.19753. Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context.Jour- nal of Machine Learning Research, 25(49):1–55, 2024a. URLhttp://jmlr.org/papers/v25/23-1042. html. Ruiqi Zhang, Jingfeng Wu, and Pet...

  29. [37]

    Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang

    URLhttps://arxiv.org/abs/2501.16265. Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization,

    arXiv

  30. [38]

    12 A Experimental details The data structure follows the Section 2 setup

    URLhttps://arxiv.org/ abs/2305.19420. 12 A Experimental details The data structure follows the Section 2 setup. For all experiments we choose a stationary kernel,i.e., distance-dependent correlations, given by Kab := exp(−|a−b|/ξ).(2) Here the correlation lengthξis manifestly ...

    2025 arXiv

  31. [39]

    correlation length

    B Detailed setup Here we reiterate the data and model setup given in the main document with more details. We leave the token design matrix general,i.e., xa∼KN(0,Σ/d) where the sample-sample correlations are given by positive semi-definiteKas EX[XaiXbj] = 1 dKabΣij fora,b∈[ℓ]an...

    2025

  32. [40]

    task diversity

    In the maximally-correlated caseK=11 ⊤, we see thatA=Z 2 11 1 d ∑d i=2Z2 1i tends in distribution to a χ2 random variable with 1 degree of freedom (as1 d ∑d i=2Z2 1i→1almost surely), which clearly does not concentrate. This gives us an intuitive heuristic for concentration of∥...

    2025

  33. [41]

    Throughout this entire section we will takeΣ =Id

    The methodology in this section is to analyse the random objectΓ∗, which contains randomness through the particular training batch(Hµ,yµ test)n µ=1, as a resolvent of a larger random matrix, and the necessarytr ( ΓA⊤) andtr ( Γ⊤BΓ ) terms as traces of a resolvent against deter...

    2025

  34. [42]

    leave-one-out

    for the optimal parameters vec(Γ∗) = ( n dλI+ n∑ µ=1 vec(Hµ)vec(Hµ)⊤ )−1n∑ µ=1 yµ testvec(Hµ)(10) forµ∈[n]denoting the index for thentraining sample sequences. Define zµ= [ yµ test/d vec(Hµ)/ √ d ] ∈Rd(d+1)+1 and construct extended resolvent Gext(π) = 1∑ µ∈[n]zµz⊤µ+πBext +τλI....

    2025

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.