Pith. sign in

REVIEW 4 major objections 4 minor 21 references

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Heavy-tailed pretraining distributions improve in-context task selection under distribution shift but worsen ICL generalization, especially in low-data regimes.

desk verdict A substantive, technically real theory of how tail behavior shapes ICL, but the 'fundamental trade-off' is proven for an idealized Bayes-optimal learner, and the bridge to real transformers is an explicit assumption, not a result. read the letter →

arxiv 2510.01163 v2 pith:JLHVFHEK submitted 2025-10-01 cs.LG stat.ML

Waïss Azizian , Ali Hasan This is my paper
classification cs.LGstat.ML
keywords pretrainingdistributionpropertiestaskframeworkfundamentalgeneralizationheavy-tailed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In-context learning (ICL) lets a model handle a new task from just a few examples in its input, without updating its weights. This paper asks how the distribution of tasks used during pretraining shapes this ability. The authors model a pretraining task distribution as a prior over tasks, and assume the trained transformer behaves like the Bayesian optimal predictor—that is, it combines prior knowledge with the in-context examples to infer the task. Under this assumption, they prove two results. First, the posterior over tasks concentrates around the true task as more examples appear, and the speed depends on how much prior probability the true task had. Heavy-tailed priors give more mass to unusual tasks, so the model can identify shifted or rare tasks faster. Second, they prove a generalization bound showing that heavy-tailed priors require more pretraining tasks to achieve the same out-of-context performance, because they have fewer finite moments. Combined, these results predict a fundamental trade-off: heavy tails improve robustness to distribution shift but hurt sample efficiency. The authors test this on linear regression, Ornstein–Uhlenbeck processes, and stochastic Volterra equations by training small transformers on different pretraining distributions and evaluating on shifted tasks. The observed trends match the theoretical predictions: heavy-tailed priors (e.g., small Student-t degrees of freedom) perform better under large shifts, while light-tailed priors perform better in-distribution or with few tasks.
Extended reading notes

Core claim

The central assertion is that heavier-tailed pretraining distributions improve ICL task selection (Theorem 1, with rate governed by log(1/pi(theta*))/T) but degrade ICL generalization (Theorem 2, with bounds worsening as the prior's moment q decreases), producing a fundamental trade-off between robustness to distribution shift and sample efficiency.

Load-bearing premise

The analysis assumes that a transformer trained to minimize next-token prediction loss approximates the Bayesian optimal predictor (Section 3.2, paragraph after Eq. 3). Without this assumption, posterior concentration (Theorem 1) does not directly translate into ICL performance; finite-capacity transformers trained by gradient descent are known to deviate from Bayes-optimality in some regimes (e.g., Raventós et al., 2023).

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops a theoretical framework to relate the pretraining task distribution to in-context learning (ICL), decomposing ICL performance into task selection and generalization. Under the assumption that a trained transformer approximates the Bayesian optimal predictor, Theorem 1 bounds the posterior concentration rate over tasks by log(1/pi(theta*))/T in Rényi divergence, so heavier-tailed priors can aid task selection for out-of-distribution tasks. Theorem 2 derives generalization bounds for empirical risk minimization under heavy-tailed priors and weakly dependent sequences, with bounds that worsen as the prior's moment q decreases, so heavier tails appear harmful for generalization. The paper reports qualitative experimental support on linear regression, Ornstein–Uhlenbeck processes, and Volterra processes with long-range dependence.

Significance. If the main claims hold, the paper provides a useful design principle: pretraining distributions should be chosen by balancing tail robustness against sample efficiency. The theoretical machinery is developed from first principles with no fitted constants; the concentration results for heavy-tailed, non-i.i.d. sequences in Appendix E are of independent interest. The experiments go beyond simple linear regression to SDEs and Volterra processes, which is valuable. However, the central claims are conditional on an unverified Bayes-optimality bridge, and the main-text theorems are stated more broadly than the appendix proofs support. The contribution is original and fixable, but the current presentation overstates what has been shown.

major comments (4)
  1. [§3.2, Theorem 1 (Eq. 4) and Appendix D.4] The main-text Theorem 1 is stated under Assumption 1, which only assumes polynomial tail control, and concludes the rate log(1/pi(theta*))/T + O(log T / T). The actual proof, Theorem D.2 plus Proposition D.2, uses the stronger Assumption 4: exponential tail bounds on the likelihood ratio and on the data (Eqs. D.72–D.73), a second-moment condition on pi, and prior regularity for the Laplace approximation. These hypotheses are not in Assumption 1, so Eq. (4) is not proved under the stated assumptions. This is load-bearing because the rate is the main task-selection result.
  2. [§3.2, after Eq. (3); Appendix A] The whole task-selection claim is for the Bayesian optimal predictor defined in Eq. (3). The manuscript asserts that a transformer trained to optimality 'learns' this predictor, and Appendix A makes explicit that the model is assumed to be sufficiently expressive and trained optimally. This is an unverified assumption, and the authors themselves cite Raventós et al. (2023), who show that trained transformers can be non-Bayesian in linear regression. The experiments do not isolate the Bayesian mechanism, so Theorems 1 and 2 do not currently constrain the ICL behavior of finite, gradient-trained transformers. The central trade-off needs either a validation of this bridge or a substantial weakening of the claims to 'Bayesian-optimal ICL'.
  3. [Theorem 2, Eq. (9); Appendix E.3, Eq. (E.156)] The dependence of Eq. (9) on B_T is inconsistent with the appendix. Under Assumption 2(iii), B_T is a per-pair coefficient, so the cumulative effect over T steps should involve a sum over s>t. The appendix bound in Lemma E.8 and Eq. (E.156) contains sqrt( sum_{t=1}^T (sum_{s>t} B_{t,s})^2 ), which for constant B_{t,s}=B_T scales like B_T T^{3/2}/sqrt(N). Eq. (9) instead gives B_T T/(sqrt(T)sqrt(N)) = B_T sqrt(T)/sqrt(N), a factor of T smaller. This makes the displayed main-text bound substantially more optimistic than what is proved, and it affects the quantitative claims about temporal dependence in Section 4.3.
  4. [§3.3, Eq. (8) and Theorem 2] Theorem 2 bounds the generalization gap between the population next-token loss and the empirical loss for an ERM model. To conclude that a trained transformer 'accurately behaves as the Bayesian optimal predictor,' one would also need an approximation error bound showing that the model class F contains the Bayes predictor, or a misspecification term. No such term appears in Eq. (8) or Theorem 2. The present result controls ERM generalization but not the distance from the Bayes-optimal predictor, so the connection between the generalization bound and ICL remains incomplete.
minor comments (4)
  1. [Takeaway #1, §3.2] The statement 'heavier-tailed priors are beneficial for task identification' overgeneralizes. Theorem 1's rate depends on log(1/pi(theta*)), so heavier tails help only when the true task is in a region where the heavy-tailed prior has larger density. The experiments in Fig. 1 indeed show the normal prior performing best for small shifts. The takeaway should be qualified.
  2. [Appendix E.5, Theorem E.2] The theorem statement begins 'for any δ∈(0,e^-2], for any δ∈(0,N e^{-q}],' with a duplicated quantifier, and the two regimes are not cleanly separated. Please fix the statement.
  3. [§4 and Appendix C.2] The experiments are reported as repeated over three seeds, but the figures do not show error bars or confidence bands. For a claim about qualitative trends across distributions, error bars would materially help the reader assess the robustness of the crossover points.
  4. [Assumption 2 and Theorem 2] The notation O(·) in Theorem 2 hides dependence on q, log N, the scale of pi, and the size of F. This makes the displayed bounds difficult to compare with the appendix. At least in the appendix, state explicitly which quantities are treated as constants and which must grow with T, N, and delta.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the bounds are derived from stated assumptions and validated by external experiments; no fitted parameter is relabeled as a prediction.

full rationale

The derivation chain is self-contained. Theorem 1 (Section 3.2, Eq. 4; Appendix D) is a Bayesian posterior concentration bound whose leading term log(1/pi(theta*))/T follows from the posterior being proportional to the prior times the likelihood; this is a mathematical consequence, not an input disguised as an output. Theorem 2 (Section 3.3, Eqs. 9-10) is a new concentration/generalization bound whose q-dependence is derived from the assumed finite q-th moment (Assumption 2(i)); the heavier-tail-hurts-generalization statement is a corollary, not a restatement. No parameter is fitted to the experimental curves and then called a prediction; the experiments on linear regression, OU, and Volterra processes are external checks of the qualitative signs of the bounds. The paper explicitly flags the Bayes-optimality bridge ('We therefore assume that the model is sufficiently expressive and trained optimally enough to approximate the Bayes optimal predictor', Appendix A) as an assumption, and citing Raventós et al. (2023) for the experimental setup is independent support, not a load-bearing self-citation. The self-citations present (e.g., Azizian et al., 2025) concern optimization dynamics and are not used to justify the central trade-off. No circular step, self-definitional identity, renamed known result, or fitted-input-called-prediction is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The theoretical results rest on the Bayes-optimality assumption and on the stated data-generation and dependence assumptions. No new entities are introduced. The prior moment q is not fitted; it is a property of the assumed pretraining distribution.

assumptions (4)
  • domain assumption Bayes-optimality of the trained model: the model f trained to minimize next-token prediction loss learns the Bayesian optimal predictor.
    Invoked in Section 3.2 (paragraph after Eq. 3) to connect posterior concentration to ICL performance. This is an idealization not verified for finite transformers.
  • domain assumption Assumption 1/3: controlled tails of the data, log-likelihood-ratio moment bound, and local Lipschitz regularity of the log-likelihood for the given task distribution.
    These are the conditions for Theorem 1 (task selection). They are stated as 'mild' but include e.g. sup_theta p_T(X|theta)/p_T(X|theta0) logging square integrable, which can be restrictive for certain non-parametric task spaces.
  • domain assumption Assumption 2/5: finite q-th moment of the prior, Wasserstein-1 weak dependence coefficients A_T and B_T, sub-Gaussian conditional data, and Lipschitz losses/model class.
    These are the conditions for Theorem 2 (generalization). They require the task distribution to have polynomial tails with exponent q and the data sequence to satisfy a specific Wasserstein dependence condition, which may be hard to verify in general transformer settings.
  • standard math Standard mathematical tools: Donsker-Varadhan variational formula, Laplace approximation, Dudley-type entropy integral, convex domination.
    Used throughout Appendices D and E without proof, as standard results in probability and statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off." pith.science (2026). https://pith.science/paper/JLHVFHEK

@misc{pith2026251001163,
  author       = {Pith},
  title        = {Pith review of: How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLHVFHEK}},
  note         = {Machine review of arXiv:2510.01163}
}
read the original abstract

The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples. To clarify and improve these capabilities, we characterize how the statistical properties of the pretraining distribution (e.g., tail behavior, coverage) shape ICL. We develop a theoretical framework that encompasses generalization and task selection and show how distributional properties govern sample efficiency, task retrieval, and robustness. To this end, we generalize existing concentration results to heavy-tailed priors and dependent sequences, better reflecting the structure of LLM pretraining data. Our framework reveals a fundamental design trade-off: heavy-tailed pretraining distributions facilitate robust task selection under distribution shifts but are detrimental to generalization, especially in low-data regimes. We then empirically evaluate our predictions by studying how ICL performance varies with the pretraining distribution on challenging tasks such as stochastic differential equations and stochastic processes with memory. Together, these findings suggest that controlling key statistical properties of the pretraining distribution is essential for building ICL-capable and reliable LLMs.

Figures

Figures reproduced from arXiv: 2510.01163 by the authors.

Figure 1
Figure 1. Influence of the degree of freedom parameter of a Student-𝑡 pretraining distribution on the ICL error for different task shifts with and without importance weighting. Weighted samples given by −★ marker. 0 1 2 3 4 Task Center (Task Shift) 5 0 5 log MSE Best MSE =3.0 =5.0 =10.0 = 0 1 2 3 4 Task Center (Task Shift) 0 2 4 6 Mean MSE 0 1 2 3 4 Task Center (Task Shift) 5 0 5 Full Context Length MSE (a) 𝑛 = 500 0 1 2 3 4 … view at source ↗
Figure 2
Figure 2. Generalization for linear regression with a Student-𝑡 prior of varying 𝜈 as a function of 𝑛. The generalization thus depends critically on the moment 𝑞 of the prior 𝜋: the smaller the moment 𝑞, the heavier the tail of the prior 𝜋 and the worse the generalization error. This provides a counterpoint to the task selection result of Theorem 1 that showed that heavier-tailed priors are beneficial for task identification.… view at source ↗
Figure 3
Figure 3. Influence of the degree of freedom parameter of a Student-𝑡 pretraining distribution on the ICL error for different task shifts with and without importance weighting for predicting the next step in an OU process with context length of 32. Weighted samples indicated by the −★ marker. 0 1 2 3 4 Task Center (Task Shift) 5 4 3 2 1 log MSE Best MSE =1.0 =1.5 =2.0 =2.5 0 1 2 3 4 Task Center (Task Shift) 4 3 2 1 0 Mean MSE… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Influence of the shape of a generalized normal pretraining distribution on the ICL error for different task shifts with and without importance weighting for predicting the next step in an OU process. of the shift magnitude ∥Δ∥ Studying this error as a function of the s…
Figure 5
Figure 5. Figure 5: Generalization of a transformer trained to predict the next step of the Volterra as a function of 𝑛 the number of tasks with context length of 32. strong performance, can we improve the performance of distribution 𝑃 by matching 𝑄 via importance sampling i.e. 𝔼𝑄 [ℓ(𝑋)] …
Figure 6
Figure 6. Figure 6: Linear regression with Student-𝑡 pretraining distributions: MSE as a function of ICL step for different task shift magnitudes. Heavy-tailed priors (𝜈 = 3) show superior robustness to distribution shift, while light-tailed priors (𝜈 = ∞, Gaussian) perform better on unpe…
Figure 7
Figure 7. Figure 7: Linear regression with generalized normal pretraining distributions: MSE as a function of ICL step for different task shift magnitudes. The shape parameter 𝛽 has a more modest impact on performance compared to Student- 𝑡 distributions, with all variants showing similar…
Figure 8
Figure 8. Figure 8: Generalization analysis for linear regression across different numbers of pretraining tasks 𝑛 for a context length of 64. As predicted by Theorem 2, heavy-tailed priors (small 𝜈) require more tasks to achieve performance comparable to light-tailed priors, but eventuall…
Figure 9
Figure 9. Figure 9: Ablation on the effect of variance for Gaussian pretraining distributions in linear regression. Only in￾distribution performance is affected by the variance, with larger variances leading to worse performance. 0 5 10 15 20 25 30 Context Length 10 2 10 1 10 0 MSE ICL MS…
Figure 10
Figure 10. Figure 10: Ornstein–Uhlenbeck processes with Student-𝑡 pretraining distributions: MSE as a function of ICL step for different task shift magnitudes. Unlike linear regression, heavy-tailed priors maintain strong in-distribution performance while providing superior robustness to p…
Figure 11
Figure 11. Figure 11: Ornstein–Uhlenbeck processes with generalized normal pretraining distributions (importance weighted): MSE as a function of ICL step for different task shift magnitudes. The shape parameter 𝛽 shows consistent effects across perturbation levels, with all variants signif…
Figure 12
Figure 12. Figure 12: Stochastic Volterra equations: MSE as a function of ICL step across different kernel exponents 𝛼. Smaller 𝛼 values correspond to stronger long-range dependencies, leading to slower convergence and higher final error. The performance gap between different 𝛼 values demo…
Figure 13
Figure 13. Figure 13: Generalization analysis for Volterra processes across different numbers of pretraining tasks 𝑛. Processes with stronger temporal dependencies (smaller 𝛼) exhibit larger performance gaps at low 𝑛, consistent with Theorem 2. The dependency coefficients in our theory sca…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 7 linked inside Pith

  1. [2]

    23 C Experimental Details We roughly follow the experimental setup used by Raventós et al

    The dependency coefficients in our theory scale with𝛼, explaining why more training tasks are needed to achieve good performance for smaller𝛼values. 23 C Experimental Details We roughly follow the experimental setup used by Raventós et al. (2023). Our code is largely based on their implementation given in‡. C.1 Data Generation In all experiments, task par...

    2023

  2. [8]

    Experiments are run with AdamW optimizer with a weight decay of 0.1 with a cosine learning rate schedule and 50,000 warmup steps

    All models were trained for5×10 5 iterations. Experiments are run with AdamW optimizer with a weight decay of 0.1 with a cosine learning rate schedule and 50,000 warmup steps. All experiments were run on NVIDIA H100 GPUs. We performed a hyperparameter sweep over learning rate where we considered two learning rates and chose the best model. Experiments are...

    2025

  3. [11]

    sparse memory

    A Additional Related Work 16 B Additional Experimental Results 17 B.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B.2 Ornstein–Uhlenbeck Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.3 Volterra Processes . . . . . . . . . . . . . . . . . . . . ....

    2025

  4. [15]

    27 D.4 Task Selection Bound for ICL We begin with a discretization argument and first we generalize the bracketing numbers to the non-i.i.d

    ≤poly(𝑇).(D.31) •Regularity of the likelihood: for any𝑡≥1,𝜃,𝜃 ′∈Θ∩ 𝔹(0,𝑅), sup 𝑥1:𝑡∈𝔹(0,𝑅) 𝑡 log p𝑡(𝑥𝑡|𝑥 1:𝑡−1,𝜃) p𝑡(𝑥𝑡|𝑥 1:𝑡−1,𝜃′)≤poly(𝑅)∥𝜃−𝜃 ′∥.(D.32) For a sequence(𝑥𝑡)𝑡≥1, we denote by𝑥𝑎:𝑏 the subsequence(𝑥𝑎,𝑥𝑎+1,...,𝑥 𝑏)for1≤𝑎≤𝑏with the convention that𝑥 𝑎:𝑏=𝑥 1:𝑡 if𝑎<1. 27 D.4 Task Selection Bound for ICL We begin with a discretization argument and ...

    1999

  5. [17]

    Proof.Fix𝑅>0that will be chosen later and takeX ′=𝔹(0,𝑅)andΘ ′=𝔹(0,𝑅)

    p𝑇(𝑥1:𝑇|𝜃) ≤𝜋(𝜃∉Θ ′)𝜇+O(log(𝑇)),(D.63) where theO(·)hides constants that do not depend on𝜋or𝑇. Proof.Fix𝑅>0that will be chosen later and takeX ′=𝔹(0,𝑅)andΘ ′=𝔹(0,𝑅). Let us consider a𝛿-cover of Θ′ with𝛿>0that will be chosen later: there are𝐾sets𝑈 𝑗,𝑗=1,...,𝐾that coverΘ ′ such that for any𝜃,𝜃′∈𝑈 𝑗, we have∥𝜃−𝜃 ′∥≤𝛿. By e.g., Wainwright (2019, Ex. 5.2), we ...

    2019

  6. [18]

    30 D.5 Laplace Approximation We will make use of the following version of the Laplace approximation, see Wong (2001, Chap

    Otherwise, consider the case the KL will typically be of order𝑇so that this term isO(1): it represents the best ICL error one can hope for when the true task𝜃∗ is not in the support of the prior𝜋. 30 D.5 Laplace Approximation We will make use of the following version of the Laplace approximation, see Wong (2001, Chap. 9, Thm

    2001

  7. [19]

    p𝑇(𝑥1:𝑇|𝜃) (D.85) ≤−log 𝔼𝜃∼𝜋 1{∥𝜃∥≤𝑟 𝑇} exp(−poly(𝑟 𝑇)𝑇∥𝜃−𝜃 0∥+O(1) )+O 1 𝑇𝑑+1 (D.86) ≤−log 𝔼𝜃∼𝜋 1{∥𝜃∥≤∥𝜃 0∥+1 } exp(−poly(log𝑇)𝑇∥𝜃−𝜃 0∥+O(1) )+O 1 𝑇𝑑+1 ,(D.87) where we used that we assumed that𝑟𝑇 =poly(log𝑇)≥∥𝜃 0∥+1. Applying Lemma D.5 with𝜀=1/(poly(log𝑇)𝑇)yields: 𝔼𝜃∼𝜋[1{∥𝜃∥≤∥𝜃 0∥+1 } exp(−poly(log𝑇)𝑇∥𝜃−𝜃 0∥)] =poly(log𝑇)𝑇 −𝑑(𝜋(𝜃 0)𝐶+𝑜(1) ),(D.88) where...

    2007

  8. [20]

    sup 𝑓∈F 𝔼 h b𝐿(𝑓,(𝜃 𝑛,𝑥𝑛 1:𝑇)𝑛≤𝑁) i − b𝐿(𝑓,(𝜃 𝑛,𝑥𝑛 1:𝑇)𝑛≤𝑁) # (E.151) =𝔼

    Assumption 5(Weak dependence).We assume that there are deterministic coefficients(𝐴𝑡)𝑡≥1 and(𝐵 𝑠,𝑡)𝑡≥𝑠≥1 such that, for any𝑡≥𝑠≥1,𝜃,𝜃 ′∈Θ, any𝑥 1:(𝑠−1) ∈X 𝑠−1, and any𝑥𝑡,𝑥𝑡′∈X, 𝑊1(p𝑡(𝑑𝑥𝑡|𝜃),p 𝑡(𝑑𝑥𝑡′|𝜃′))≤𝐴 𝑡∥𝜃−𝜃 ′∥(E.87) 𝑊1(p𝑡(𝑑𝑥𝑡|𝑥 1:𝑠,𝜃),p 𝑡(𝑑𝑥𝑡′|𝑥 1:(𝑠−1),𝑥′ 𝑠,𝜃))≤𝐵 𝑠,𝑡∥𝜃∥.(E.88) In the second assumption, the Wasserstein distance between the conditional...

    2018

Show all 21 references
  1. [21]

    6.1), for𝑇large enough, there is𝑐>0constant such that, with probability at least1−𝑒−𝑐𝑇,∥𝑄Δ∥≥𝑐 √ 𝑇∥Δ∥and∥𝑄 ⊤𝑟0∥≤𝑐 −1√ 𝑇∥𝑟 0∥

    Then ℓ𝑇(𝜃)−ℓ 𝑇(𝜃 0)=− 1 2𝜎 2 𝜖 ∥𝑄Δ∥ 2 2−2Δ⊤𝑄⊤𝑟0 Now, by e.g., Wainwright (2019, Thm. 6.1), for𝑇large enough, there is𝑐>0constant such that, with probability at least1−𝑒−𝑐𝑇,∥𝑄Δ∥≥𝑐 √ 𝑇∥Δ∥and∥𝑄 ⊤𝑟0∥≤𝑐 −1√ 𝑇∥𝑟 0∥. Hence, uniformly over∥𝜃∥≥𝑅(so∥Δ∥≥𝑅−∥𝜃 0∥), ℓ𝑇(𝜃)−ℓ 𝑇(𝜃

    2019

  2. [32]

    C.2 Architecture and Optimization Details For all experiments, we consider the architecture inspired by GPT-2 as used in Raventós et al

    For evaluation, we evaluate the model output against𝔼[𝑋(𝑘+1)Δ𝑡|𝑋 0,𝑋Δ𝑡,...,𝑋 𝑘Δ𝑡]which is computable in closed form. C.2 Architecture and Optimization Details For all experiments, we consider the architecture inspired by GPT-2 as used in Raventós et al. (2023). For linear regr...

    2023

  3. [1961]

    An information-theoretic approach to gen- eralization theory.arXiv preprint arXiv:2408.13275,

    Borja Rodríguez-Gálvez, Ragnar Thobaben, and Mikael Skoglund. An information-theoretic approach to gen- eralization theory.arXiv preprint arXiv:2408.13275,

    arXiv

  4. [1994]

    URLhttp://www.jstor.org/stable/2244496

    ISSN 00911798, 2168894X. URLhttp://www.jstor.org/stable/2244496. OussamaZekri, AmbroiseOdonnat, AbdelhakimBenechehab, LinusBleistein, NicolasBoulle, andIevgenRedko. Large language models as markov chains.arXiv preprint arXiv:2410.02724,

  5. [1999]

    In-context learning with long-context models: An in-depth exploration

    Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com...

    2025

  6. [2005]

    Language models are few-shot learners

    10 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  7. [2006]

    to the non-i.i.d case and the following result generalises the results of Zhang (2006) to the non-i.i.d. case. Definition 2.Given a sequence of random variables(𝑥𝑡)𝑡≤𝑇 on a measurable spaceX, with parametric densities p𝑡(·|𝜃)parameterized by𝜃∈Θ, compact setsΘ ′⊂ΘandX ′⊂X, the𝜀...

    2006

  8. [2008]

    Is in-context universality enough? MLPs are also universal in-context

    Anastasis Kratsios and Takashi Furuya. Is in-context universality enough? MLPs are also universal in-context. arXiv preprint arXiv: 2502.03327,

    arXiv

  9. [2013]

    Transformers meet in-context learning: A universal approximation theory.arXiv preprint arXiv: 2506.05200, 2025a

    Gen Li, Yuchen Jiao, Yu Huang, Yuting Wei, and Yuxin Chen. Transformers meet in-context learning: A universal approximation theory.arXiv preprint arXiv: 2506.05200, 2025a. Shaojie Li and Yong Liu. Concentration and moment inequalities for general functions of independent rando...

    arXiv

  10. [2021]

    Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv: 2311.00871,

    Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models.arXiv preprint arXiv: 2311.00871,

    arXiv

  11. [2023]

    A survey of in-context reinforcement learning.arXiv preprint arXiv:2502.07978,

    Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. A survey of in-context reinforcement learning.arXiv preprint arXiv:2502.07978,

    arXiv

  12. [2024]

    Generalization for slowly mixing processes.arXiv preprint arXiv:2305.00977,

    Andreas Maurer. Generalization for slowly mixing processes.arXiv preprint arXiv:2305.00977,

    arXiv

  13. [2025]

    Out-of-distribution generalization of in-context learning: A low-dimensional subspace perspective.arXiv preprint arXiv:2505.14808,

    Soo Min Kwon, Alec S Xu, Can Yaras, Laura Balzano, and Qing Qu. Out-of-distribution generalization of in-context learning: A low-dimensional subspace perspective.arXiv preprint arXiv:2505.14808,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.