Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

In-Context Learning Strategies Emerge Rationally

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Transformers' in-context strategies emerge from a Bayes-optimal tradeoff between data fit and implementation complexity, and a three-parameter formula predicts next-token behavior almost perfectly throughout training.

desk verdict A genuinely unifying three-parameter model of ICL that deserves referee time, but the unvalidated complexity proxy and a post hoc 5x fix in classification mean the 'perfect prediction' claim needs the appendix. read the letter →

arxiv 2506.17859 v2 pith:GWTNS7TC submitted 2025-06-21 cs.LG cs.AI

Daniel Wurgaft , Ekdeep Singh Lubana , Core Francisco Park , Hidenori Tanaka , Gautam Reddy , Noah D. Goodman This is my paper · ORCID
classification cs.LGcs.AI
keywords in-contextlearninghierarchicalBayesianmodelrationalanalysissimplicitybiasscalinglawstaskdiversitytransientgeneralizationloss-complexitytradeoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why a Transformer trained on a mixture of tasks switches between qualitatively different in-context-learning strategies, and to show these switches are rational: they follow a Bayes-optimal tradeoff between how well a strategy fits the training data and how complex it is to implement. The authors claim that a model's next-token predictions throughout training are a weighted average of just two Bayesian strategies—a memorizing predictor that assumes a discrete prior over the tasks seen in training, and a generalizing predictor that matches the true task distribution—with the weight set by a three-parameter formula. Fitted on part of the training grid, the formula is reported to predict Transformer outputs on held-out conditions almost perfectly across Balls & Urns, in-context linear regression, and in-context classification, using a normative lens from cognitive science that treats behavior as optimal adaptation to data under computational constraints. If the account is right, two widely studied phenomena—the task-diversity threshold beyond which models generalize out-of-distribution, and transient generalization, which later collapses into memorization—are the same loss-complexity tradeoff seen from different angles, and the regime in which each strategy dominates can be predicted from data statistics alone.

What carries the argument

The load-bearing object is the log-posterior-odds identity of equation (4), $\eta(N,D) = \gamma N^{1-\alpha} \Delta L(D) - \Delta K(D)^\beta$, where $\Delta L(D)$ is the difference in training loss between the memorizing and generalizing predictors on the task mixture and $\Delta K(D)^\beta$ is the difference in their exponentiated Kolmogorov complexities, estimated by losslessly compressing cleaned Python source code and numpy arrays for each predictor. This identity converts two assumed computational constraints—power-law scaling of loss with sample count, and a simplicity bias with Transformer-specific implementation cost $K(Q)^\beta$—into a single number that, passed through a sigmoid, weights the two Bayesian predictors. Task diversity enters only through the complexity gap and the loss gap, training time enters only through $N^{1-\alpha}$, and the crossover line $\eta(N,D) = 0$ is where the two strategies are equally probable.

What would settle it

Measure the time to transience $N^*(D)$ at task diversities beyond those reported, e.g., $D = 2^{10}$ and $2^{11}$ in Balls & Urns, where the loss gap $\Delta L(D)$ is small; the framework predicts superlinear growth and, as $\Delta L(D)$ approaches zero, a divergence of $N^*$, so observing transience at finite training time there, or a linear rather than superlinear scaling, would falsify the complexity term's role.

Watch

Extended reading notes

Core claim

The paper's central claim is that in-context learning on a finite mixture of tasks is captured by a hierarchical Bayesian model in which pretraining updates the posterior probability of two hypotheses—the memorizing predictor M, which assumes a discrete prior over the seen tasks, and the generalizing predictor G, which assumes a prior over the true task distribution—and inference-time behavior is their posterior-weighted average: $h_{\text{pred}}(s) = \sigma(\eta(N,D)) M(s) + (1 - \sigma(\eta(N,D))) G(s)$. Assuming power-law loss scaling with dataset size and a simplicity prior under which the probability of implementing predictor Q is proportional to $2^{-K(Q)^\beta}$, with K the Kolmogorov complexity estimated by losslessly compressing hand-written code, the log posterior odds take the closed form $\eta(N,D) = \gamma N^{1-\alpha} \Delta L(D) - \Delta K(D)^\beta$. The loss term grows sublinearly with training steps and favors the memorizing predictor, while the complexity term favors the simpler generalizing predictor, and their balance produces the task-diversity threshold, the transient-generalization crossover, and a boundary where the two predictors are equally probable. Fitting only the three parameters $(\alpha, \beta, \gamma)$ to next-token predictions on part of the $(N,D)$ grid, the authors report near-perfect prediction of Transformer outputs on the rest—mean $R^2$ of 0.97 in linear regression, 0.92 agreement in classification, 0.97 Spearman correlation in Balls & Urns across 72 condition maps—and derive the new prediction that the time to transience grows superlinearly with task diversity.

Load-bearing premise

The load-bearing premise is that a Transformer's true cost of implementing a strategy is tracked by the compressed size of hand-written Python code and arrays describing that strategy, so the complexity gap $\Delta K(D)^\beta$ determines where the diversity threshold and transience crossover fall; the paper itself flags this code-compression proxy as a crucial limitation.

Editorial extensions

If this is right

  • Task-diversity thresholds and transient generalization are one phenomenon: any intervention that raises the memorizing predictor's implementation cost, or shrinks its loss advantage, delays the crossover to memorization.
  • The time to transience is $N^*(D) = [\Delta K(D)^\beta / (\gamma\, \Delta L(D))]^{1/(1-\alpha)}$; it grows superlinearly with task diversity, and as the loss gap approaches zero, generalization should persist for essentially unbounded training time.
  • Preference for the memorizing predictor grows sublinearly with training steps and sigmoidally with $N^{1-\alpha}$, so even past the task-diversity threshold the model keeps drifting toward memorization—contradicting an earlier claim that high diversity locks in generalization.
  • Larger models memorize more: the fitted complexity penalty $\beta$ decays exponentially with MLP width, so increased capacity enters the framework as a reduced prior penalty on complex solutions.
  • Whether a model's ICL counts as Bayesian depends on the assumptions; once simplicity bias and sublinear sample efficiency are included, the generalizing strategy is Bayes-optimal in some regimes even though it does not minimize training loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the compression proxy truly tracks implementation cost, the framework implies that the complexity gap, not the loss gap, sets the sharpness of the diversity threshold; a stress test would be to change how tasks are encoded (altering only $\Delta K(D)^\beta$) and verify that the boundary shifts as the formula predicts.
  • The same posterior-weighted averaging should extend beyond two hypotheses: in settings with more candidate strategies, such as mixtures of Markov chains, the framework's logic predicts additional crossover boundaries as each strategy's complexity and loss gap come into play.
  • Because the transience timescale is governed by $\gamma N^{1-\alpha}\Delta L(D)$, the framework suggests that scheduling interventions (such as learning-rate annealing, which the paper finds improves adherence to the predicted trajectories) could be used deliberately to extend or shorten the window of out-of-distribution generalization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies in-context learning in Transformers trained on mixtures of tasks in three settings: Balls & Urns, in-context linear regression, and binary classification. It first observes that model behavior can be described as a transition between two Bayesian predictors: a memorizing predictor with a discrete prior over seen tasks and a generalizing predictor with the true task distribution as prior. It then proposes a hierarchical Bayesian account in which the log-posterior odds of the two predictors take the form η(N,D) = γN^{1−α}ΔL(D) − ΔK(D)^β, with three fitted parameters (α, β, γ), and predictions are a sigmoid-weighted mixture of the two predictors. The paper reports that this model reproduces task-diversity thresholds, transient generalization, and held-out next-token predictions on a reserved 20% of the (N,D) grid, and it draws novel predictions, including superlinear scaling of the transience time N*(D) with task diversity.

Significance. If the central mechanism were fully supported, the paper would provide an unusually compact and unifying normative account of when and why Transformers switch between memorizing and generalizing in-context strategies. Its strengths are substantial: the empirical replication spans three quite different task families and 72 maps; the held-out 20% of the (N,D) grid provides a genuine interpolation check; and the functional-form ablations in App. I show that removing each of α, β, or γ degrades the account. The paper also makes concrete, falsifiable predictions, such as the superlinear growth of N*(D), with some out-of-sample support in App. K. The central limitation, acknowledged by the authors in Sec. 5, is that the complexity term ΔK(D)^β is estimated by losslessly compressing hand-written predictor code and numpy arrays; this proxy is not independently shown to track a Transformer's implementation cost, and App. H.4.3 reports a post hoc 5x multiplier on the generalizing predictor's code in three classification conditions. Because this term drives the task-diversity boundary and the transience crossover, the rational-mechanism interpretation currently rests on an unvalidated link.

major comments (4)
  1. [Section 4, Eq. 4; App. H.4.3.] The complexity term ΔK(D)^β in Eq. 4 is the only D-dependent quantity opposing the loss term and sets both the task-diversity boundary and the transience timescale N*(D), but the paper does not validate that lossless compression of hand-written Python code and numpy arrays tracks a Transformer's implementation cost. App. H.4.3 reports that for three classification maps with task dimensionality 16 the model fits only after 'heuristically multiply[ing] the bit size of the code for the generalizing predictor by 5'; that is a post hoc rescaling of the very term that generates the prior odds. Because β is a fitted exponent, a flexible sigmoid σ(γN^{1−α}ΔL − ΔK^β) could accommodate the observed maps even if K were unrelated to true implementation complexity, so the reported correlations do not by themselves validate the rational mechanism. The limitation stated in Sec. 5 concedes exactly this point and should be treated as a load-bearing gap rather than a routine caveat.
  2. [App. E; Sec. 4; abstract.] The abstract and Sec. 4 state that the model 'almost perfectly predicts Transformer next-token predictions throughout training,' but App. E restricts the evaluation to checkpoints after a heuristically chosen two-hypotheses threshold, defined as interpolation loss 10–20% above its minimum. The model is therefore not a model of the full training trajectory, and no aggregate prediction quality is reported for the excluded early-training regime. The wording 'throughout training' overstates the empirical scope and should be qualified.
  3. [App. D.1, Eq. 4.] In the derivation of Eq. 4, Neff is defined as γN^{1−α} with γ = A/L_Q, where L_Q is the loss of predictor Q. Since L_M and L_G differ, and their difference is precisely ΔL(D), the single coefficient γ in Eq. 4 does not follow from that definition: either γ is predictor-dependent and should appear separately in the two posterior terms, or γ is a common constant and its identification with A/L_Q is inconsistent. This step is load-bearing because the crossover condition and the expression N*(D) = [ΔK(D)^β/(γΔL(D))]^{1/(1−α)} are derived from Eq. 4.
  4. [Sec. 4, Figs. 4–5.] The match between the model's posterior probability σ(η(N,D)) and the Transformer's relative-distance maps is presented as if it did not use the same data ('without fitting to the relative distance maps'), but drel is computed from the same Transformer next-token predictions h(·) to which η is fitted. The held-out 20% of the (N,D) grid is a genuine interpolation check and is evidence for the predictive form, but the correlation with drel is not independent evidence for the loss-complexity mechanism.
minor comments (4)
  1. [Sec. 4.1; App. K.] The superlinear-transience prediction is tested only for D ∈ {2^8, 2^9} with learning-rate annealing, and App. K reports that the last two observed transience points deviate from the model's prediction; the main-text claim that the predictions 'hold well' should quantify the size and conditions of this deviation.
  2. [Sec. 4.1, Fig. 6(a).] The sigmoidal curves in N^{1−α} are obtained by fitting a separate parameterized logistic with three free parameters per training run; the figure and text should clarify that these curves are not generated directly from Eq. 5 without additional per-condition fits.
  3. [Sec. 4, validation paragraph.] The reported 'mean agreement of 0.92 in Classification' is not defined in the main text; please specify how agreement is computed and report confidence intervals or per-map variability for all three aggregate metrics.
  4. [App. F.2.] The median-of-means estimator for average log-likelihood is described briefly; adding the number of buckets or the variance-reduction diagnostic would allow readers to judge convergence, especially for the long-tailed per-token losses mentioned there.

Circularity Check

2 steps flagged · score 6.0 of 10

Relative-distance agreement is built into the interpolation metric, and the complexity term is post hoc rescaled in three classification maps, so the loss-complexity 'prediction' is only partly independent.

  1. self definitional [App. F.2 ('Computing relative distance'); Sec. 4, Eq. 5]
    "This metric implicitly makes the assumption that in some function space, the model h(.) lies on a line between the predictors M and G."

    Eq. 5 defines h_pred(s) = σ(η)M(s)+(1−σ(η))G(s), i.e., exactly an interpolation on the M–G line, and the free parameters are fit by minimizing the distance between h_pred and h. The paper's relative-distance metric d_rel is defined from d(h,G), d(h,M), and d(G,M) under the same line assumption; for Euclidean distance, d_rel is algebraically equal to the interpolation coefficient. Thus after fitting σ(η) to h, the reported 'almost perfect match' between posterior probabilities and relative-distance maps is largely a restatement of the metric's definition plus the fit, not an independent confirmation of the rational framework. The held-out 20% of the grid gives some genuine extrapolation, but the coordinate system itself forces the agreement.

  2. fitted input called prediction [App. H.4.3 (Classification model predictions)]
    "However, this is not captured by our complexity measure, since the compressed size of the code for the memorizing and generalizing predictors is roughly the same, thus we are unable to capture the bias toward the memorizing predictor in low task diversity settings. To overcome this, in these 3 conditions only, we heuristically multiply the bit size of the code for the generalizing predictor by 5, and with that fix, we find good performance (though as can be seen, the model still under-weights the memorizing solution for some low task diversity conditions)."

    In Eq. 4, ΔK(D)^β is the only D-dependent term opposing the loss term and is therefore the quantity that sets the task-diversity boundary and the transience crossover. Rather than measuring this complexity difference independently, the paper recalibrates it after seeing the failure: it multiplies the generalizing predictor's compressed code size by 5 in exactly the conditions where the model would otherwise not match the Transformer. This is a post hoc fit of the input that generates the prior odds to the phenomenon the framework is supposed to predict; the subsequent agreement in those classification maps is therefore not independent evidence for the loss-complexity tradeoff.

full rationale

The paper is not entirely circular: fitting three parameters (α, β, γ) to an 80% subset of the (N,D) grid and validating next-token predictions on the remaining 20% is a legitimate held-out extrapolation, and the coding-theorem prior is an independent assumption rather than a self-citation. However, two load-bearing validations reduce partly to construction. First, the relative-distance metric is defined by assuming h lies on the M–G segment, and Eq. 5 is the same interpolation; once σ(η) is fit to h, the correlation between posterior probabilities and d_rel is substantially forced by the metric's definition. Second, the central complexity term is not independently anchored: it is estimated from compressed hand-written code, admitted in Sec. 5 to be only 'the simple relation we assume between algorithmic complexity and complexity of implementation by a Transformer,' and in three classification maps it is heuristically rescaled by a factor of 5 to make the model work. This does not destroy the paper's value as a compact descriptive model, but it means the rational 'explanation' and the strong correlations are only partially independent of the data they are said to predict.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central model rests on two computational constraints (power-law scaling, simplicity prior), a two-hypotheses assumption with a post hoc threshold, and a compression-based proxy for Kolmogorov complexity. The three fitted parameters (α, β, γ) and the heuristic complexity multiplier are the main free choices. No new physical entities are introduced.

free parameters (5)
  • α (power-law exponent) = not reported in main text
    Exponent in A1 loss scaling L(N)≈L(∞)+A/N^α; fitted to Transformer next-token predictions (Sec. 4).
  • β (complexity exponent) = not reported in main text
    Exponent on Kolmogorov complexity in A2; fitted; ablations show without it the simplicity prior over-penalizes complexity (App. I).
  • γ (loss-term coefficient) = not reported in main text
    Constant multiplying N^{1-α}ΔL(D) in Eq. 4; subsumes A/L_Q; fitted.
  • Two-hypotheses threshold = 20% above min loss (Balls & Urns, Classification), 10% (Linear Regression)
    Heuristic cutoff chosen post hoc to define where the two-predictor model applies (App. E); excludes early training.
  • Complexity multiplier for G = 5 for 3 classification conditions with task dimensionality 16
    Ad hoc multiplier on compressed size of generalizing predictor code to force a memorizing bias in low diversity (App. H.4.3).
assumptions (5)
  • domain assumption A1: power-law scaling of loss: L(N)≈L(∞)+A/N^α
    Empirical scaling law from Kaplan et al., Hoffmann et al.; used to define effective sample size N_eff=γN^{1-α}.
  • ad hoc to paper A2: simplicity prior p(Q) ∝ 2^{-K(Q)^β} with K estimated by lossless compression
    The β exponent and compression proxy are specific to this paper and not derived; load-bearing for diversity effects.
  • domain assumption Two-hypotheses assumption: after a threshold, only M and G need be considered
    Assumes other predictors such as the constant solution are negligible; threshold chosen post hoc (App. E).
  • ad hoc to paper Effective sample size N_eff = γ N^{1-α} across all D
    Derived from A1 assuming a constant γ independent of task diversity; central to the N^{1-α} dependence.
  • standard math Predictors M and G have closed forms given T_train and T_true
    Bayesian posterior predictives with discrete and continuous priors; derivations in App. G.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Learning Strategies Emerge Rationally." pith.science (2026). https://pith.science/paper/GWTNS7TC

@misc{pith2026250617859,
  author       = {Pith},
  title        = {Pith review of: In-Context Learning Strategies Emerge Rationally},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWTNS7TC}},
  note         = {Machine review of arXiv:2506.17859}
}
read the original abstract

Recent work analyzing in-context learning (ICL) has identified a broad set of strategies that describe model behavior in different experimental conditions. We aim to unify these findings by asking why a model learns these disparate strategies in the first place. Specifically, we start with the observation that when trained to learn a mixture of tasks, as is popular in the literature, the strategies learned by a model for performing ICL can be captured by a family of Bayesian predictors: a memorizing predictor, which assumes a discrete prior on the set of seen tasks, and a generalizing predictor, where the prior matches the underlying task distribution. Adopting the normative lens of rational analysis, where a learner's behavior is explained as an optimal adaptation to data given computational constraints, we develop a hierarchical Bayesian framework that almost perfectly predicts Transformer next-token predictions throughout training -- without assuming access to its weights. Under this framework, pretraining is viewed as a process of updating the posterior probability of different strategies, and inference-time behavior as a posterior-weighted average over these strategies' predictions. Our framework draws on common assumptions about neural network learning dynamics, which make explicit a tradeoff between loss and complexity among candidate strategies: beyond how well it explains the data, a model's preference towards implementing a strategy is dictated by its complexity. This helps explain well-known ICL phenomena, while offering novel predictions: e.g., we show a superlinear trend in the timescale for transitioning from generalization to memorization as task diversity increases. Overall, our work advances an explanatory and predictive account of ICL grounded in tradeoffs between strategy loss and complexity.

Figures

Figures reproduced from arXiv: 2506.17859 by the authors.

Figure 1
Figure 1. Why Does a Model Learn Different Strategies for Performing ICL? To answer this question, we analyze three distinct settings where a model is trained to learn a mixture of tasks. (a) Model Behavior Transitions Between Memorizing and Generalizing Predictors. We first make the observation that across settings, as diversity of data distribution and amount of training are increased, model behavior transitions between two… view at source ↗
Figure 2
Figure 2. Experimental Settings: Learning a Finite Mixture of Tasks. (a) General Formulation. Popularly studied experimental settings in the literature on ICL can be seen as training a model to learn a distribution defined using a mixture of tasks (denoted Ttrain), where each task is a parameterized latent function whose parameters are sampled from a distribution Ttrue. (b) Considered Settings. We analyze three distinct insta… view at source ↗
Figure 3
Figure 3. Predictors in Different Experimental Settings. (a) Memorizing and Generalizing Predictors. We compare model behavior to two idealized Bayesian predictors: (i) Memorizing predictor (M), which assumes a discrete prior over the mixture distribution Ttrain, and (ii) Generalizing predictor (G), which assumes a prior over Ttrue, the distribution from which tasks are sampled. (b) Task-Specific Instantiations. These predict… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Relative Distance Captures Transi￾tions in Model Behavior. We show the relative distance between model outputs and the two predic￾tors. Marginals report the absolute distance values (e.g., symmetrized KL between model and predic￾tor outputs for the Balls & Urns setting…
Figure 5
Figure 5. Figure 5: Our Bayesian Model Captures Transitions Between Strategies Explaining Model Behavior. We plot the posterior probability of the memorizing predictor given by our theoretical model (Eq. 4). Across three broad experimental settings—(a) Balls & Urns, (b) Linear Regression,…
Figure 6
Figure 6. Figure 6: Novel predictions from our framework. (a) Our framework predicts that the posterior probability of the memorized predictor (and hence the relative distance, which can be thought of as an empirical estimate of this quantity) will show sub-linear scaling with respect to …
Figure 7
Figure 7. Figure 7: Intuition Elicited by The Bayesian Model. (a) Our framework suggests Transform￾ers have a prior preference for learning simpler solutions, which often generalize better. However, throughout training, preference is updated towards solutions that better explain the data …
Figure 8
Figure 8. Figure 8: Increased memorization with MLP width is captured by Bayesian model as re￾duced complexity penalty. (a) We find relative distance to memorizing predictor decreases with MLP width. (b) This transition is closely captured by our Bayesian model, in which the ‘complexity p…
Figure 9
Figure 9. Figure 9: Two-Hypotheses Threshold. Defining an ‘optimal’ interpolation between the memorizing and generalizing predictors towards minimizing the Euclidean distance to the trained Transformer’s predictions, we report the loss between this optimal interpolation and the Transforme…
Figure 10
Figure 10. Figure 10: General Abstraction Capturing our Experimental Settings and their Predictors. Each setting involves a mixture of parameterized functions (called a “task”), with D functions (the “task diversity"). Task consist of predicting the next element in a sequence, and vary bas…
Figure 11
Figure 11. Figure 11: Visualizing the setup for Balls and Urns. Each task involves an “urn” that outputs a “ball” of a specific type every time it is sampled from. The task then involves seeing samples from an urn, concatenated to form a sequence. A memorizing predictor for this setting in…
Figure 12
Figure 12. Figure 12: Visualizing the setup for Linear Regression. Each task involves a linear regression problem, defined by parameters w, that outputs a pair (x, y), where y = w⊺x + ϵ is a noisy linear transformation of the vector x. The task then involves seeing a sequence of such pairs…
Figure 13
Figure 13. Figure 13: Visualizing the setup for Classification. Each task involves noisy item-label pairs w˜ ⊕ l, and ends with a noisy query item w˜query which comes from the same true item w as one of the items in the sequence. Items are noised via w˜ = √w+σϵ 1+σ2 , with ϵ ∈ N (0, Im/m) …
Figure 14
Figure 14. Figure 14: Task Diversity Effects Across Balls & Urns Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps). 34 [PITH_FULL_…
Figure 15
Figure 15. Figure 15: Task Diversity Effects Across Linear Regression Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps). 35 [PITH_…
Figure 16
Figure 16. Figure 16: Task Diversity Effects Across Classification Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps). IWL evaluatio…
Figure 17
Figure 17. Figure 17: Transience Across Balls & Urns Conditions. OOD performance presented. Blue Dashed line indicates OOD performance of generalizing solution G. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_17.png]
Figure 18
Figure 18. Figure 18: Transience Across Linear Regression Conditions. OOD performance presented. Blue Dashed line indicates OOD performance of generalizing solution G. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_18.png]
Figure 19
Figure 19. Figure 19: Transience Across Classification Conditions. OOD performance presented. Blue Dashed line indicates OOD performance of generalizing solution G. Note that in the case of classification, it is often the case that only one or two task diversity conditions show transient g…
Figure 20
Figure 20. Figure 20: Absolute and Relative Distance from Predictors Across Balls & Urns Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M is shown in the solid line. KL indicates symmetrized KL divergence (a…
Figure 21
Figure 21. Figure 21: Absolute and Relative Distance from Predictors Across Linear Regression Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M is shown in the solid line. 43 [PITH_FULL_IMAGE:figures/full_fi…
Figure 22
Figure 22. Figure 22: Absolute and Relative Distance from Predictors Across Classification Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M is shown in the solid line. KL indicates symmetrized KL divergence …
Figure 23
Figure 23. Figure 23: Bayesian Model Predictions Across Balls & Urns Conditions. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the memorizing solution M given by our …
Figure 24
Figure 24. Figure 24: Bayesian Model Predictions Across Balls & Urns Conditions with Varying MLP Expansion Factors. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the …
Figure 25
Figure 25. Figure 25: Bayesian Model Predictions Across Linear Regression Conditions. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the memorizing solution M given by…
Figure 26
Figure 26. Figure 26: Bayesian Model Predictions Across Classification Conditions. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the memorizing solution M given by ou…
Figure 27
Figure 27. Figure 27: Power laws over complexity measure and sample efficiency are necessarily for explaining ICL phenomenology. By ablating our functional form, we see that the free parameters α, β, γ are required for the performance of the model. In particular, the simplicity bias derive…
Figure 28
Figure 28. Figure 28: Relative distance continues to rise throughout training, even after the task diversity threshold. The figure displays relative distance, as well as absolute distance from the memorizing and generalizing solutions, for the linear regression setting with context length …
Figure 29
Figure 29. Figure 29: (a) top right D = 256 for a clear example). Indeed, fitting parameterized logistic curves to these trajectories yields plateau values different from 1. To explain this, we turn to foundational work from Geman and Geman [98], who showed that a slow (logarithmic) temper…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mechanistic Foundations of Goal-Directed Control

    cs.LG 2026-03 conditional novelty 7.0 of 10

    Context window k is the critical parameter for arbitration-gate formation in an embodied control architecture: no circuit below k≤4, resolved phase structure above k≥8, with EMA-like commitment dynamics.

Reference graph

Works this paper leans on

100 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    arXiv 2024

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Palm-e: An embodied multimodal language model, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model, 2023. URL https://arxiv.org/abs/2303.03378

    arXiv 2023

  4. [4]

    Foundational 11 challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational 11 challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024

    arXiv 2024

  5. [5]

    Openai o3 and o4-mini system card, 2025

    OpenAI. Openai o3 and o4-mini system card, 2025. URL https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf

    2025

  6. [6]

    Gemini: a family of highly capable multimodal models

    Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  7. [7]

    Claude 3.7 sonnet system card, 2025

    AnthropicAI. Claude 3.7 sonnet system card, 2025. URL https://www.anthropic.com/ news/claude-3-7-sonnet

    2025

  8. [8]

    Eliciting in-context learning in vision-language models for videos through curated data distributional properties

    Keunwoo Peter Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data distributional properties. arXiv preprint arXiv:2311.17041, 2023

    arXiv 2023

Show all 100 references
  1. [9]

    In- context learning enables robot action prediction in llms

    Yida Yin, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig. In- context learning enables robot action prediction in llms. arXiv preprint arXiv:2410.12782, 2024

    2024 arXiv

  2. [10]

    Iclr: In-context learning of representations

    Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  3. [11]

    Sparse autoencoders reveal temporal difference learning in large language models

    Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning in large language models. arXiv preprint arXiv:2410.01280, 2024

    2024 arXiv

  4. [12]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  5. [14]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022

    2022

  6. [15]

    Dual process learning: Controlling use of in-context vs

    Suraj Anand, Michael A Lepori, Jack Merullo, and Ellie Pavlick. Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting. arXiv preprint arXiv:2406.00053, 2024

    2024 arXiv

  7. [16]

    In-context language learning: Architectures and algorithms, 2024

    Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms, 2024. URL https://arxiv.org/abs/2401.12973

    2024 arXiv

  8. [17]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection.Advances in neural information processing systems, 36, 2024

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection.Advances in neural information processing systems, 36, 2024

    2024

  9. [18]

    Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    2022 arXiv

  10. [19]

    Larger language models do in-context learning differently, 2023

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023. URL https://arxiv.org/abs/2303.03846. 12

    2023 arXiv

  11. [20]

    Data distributional properties drive emer- gent in-context learning in transformers

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878–18891, 2022

    2022

  12. [21]

    Dual operating modes of in-context learning, 2024

    Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning, 2024. URL https://arxiv.org/abs/2402.18819

    2024 arXiv

  13. [22]

    Transformers learn in-context by gradient descent

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151–35174. PMLR, 2023

    2023

  14. [23]

    Trans- formers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. In International conference on machine learning, pages 19565–19594. PMLR, 2023

    2023

  15. [24]

    Singh, Stephanie C

    Aaditya K. Singh, Stephanie C. Y . Chan, Ted Moskovitz, Erin Grant, Andrew M. Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers, 2023. URL https://arxiv.org/abs/2311.08360

    2023 arXiv

  16. [25]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression

    Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [26]

    Competition dynamics shape algorithmic phases of in-context learning, 2024

    Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning, 2024. URL https://arxiv.org/ abs/2412.01003

    2024 arXiv

  18. [27]

    Toward understanding in-context vs

    Bryan Chan, Xinyi Chen, András György, and Dale Schuurmans. Toward understanding in-context vs. in-weight learning. arXiv preprint arXiv:2410.23042, 2024

    2024 arXiv

  19. [28]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023

    Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023. URL https://arxiv.org/abs/2312.03002

    2023 arXiv

  20. [29]

    Differential learning kinetics govern the transition from memorization to generalization during in-context learning, 2024

    Alex Nguyen and Gautam Reddy. Differential learning kinetics govern the transition from memorization to generalization during in-context learning, 2024. URL https://arxiv.org/ abs/2412.00104

    2024 arXiv

  21. [30]

    Dynamics of transient structure in in-context linear regression transformers.arXiv preprint arXiv:2501.17745, 2025

    Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts, and Daniel Murfet. Dynamics of transient structure in in-context linear regression transformers.arXiv preprint arXiv:2501.17745, 2025

    2025 arXiv

  22. [31]

    Which attention heads matter for in-context learning? arXiv preprint arXiv:2502.14010, 2025

    Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? arXiv preprint arXiv:2502.14010, 2025

    2025 arXiv

  23. [32]

    Singh, Ted Moskovitz, Felix Hill, Stephanie C

    Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y . Chan, and Andrew M. Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation, 2024. URL https://arxiv.org/abs/2404.07129

    2024 arXiv

  24. [33]

    Strategy coopetition explains the emergence and transience of in-context learning

    Aaditya K Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. Strategy coopetition explains the emergence and transience of in-context learning. arXiv preprint arXiv:2503.05631, 2025

    2025 arXiv

  25. [34]

    Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis

    Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains, 2024. URL https: //arxiv.org/abs/2402.11004

    2024 arXiv

  26. [35]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  27. [36]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559, 2022. 13

    2022 arXiv

  28. [37]

    The broader spectrum of in-context learning

    Andrew Kyle Lampinen, Stephanie CY Chan, Aaditya K Singh, and Murray Shanahan. The broader spectrum of in-context learning. arXiv preprint arXiv:2412.03782, 2024

    2024 arXiv

  29. [38]

    The adaptive character of thought

    John R Anderson. The adaptive character of thought. Psychology Press, 2013

    2013

  30. [39]

    Ten years of the rational analysis of cognition

    Nick Chater, Mike Oaksford, Nick Chater, and Mike Oaksford. Ten years of the rational analysis of cognition. Trends in cognitive sciences, 3(2):57–65, 1999

    1999

  31. [40]

    Rational use of cognitive resources: Levels of analysis between the computational and the algorithmic

    Thomas L Griffiths, Falk Lieder, and Noah D Goodman. Rational use of cognitive resources: Levels of analysis between the computational and the algorithmic. Topics in cognitive science, 7(2):217–229, 2015

    2015

  32. [41]

    A rational analysis of rule-based concept learning

    Noah D Goodman, Joshua B Tenenbaum, Jacob Feldman, and Thomas L Griffiths. A rational analysis of rule-based concept learning. Cognitive science, 32(1):108–154, 2008

    2008

  33. [42]

    Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources

    Falk Lieder and Thomas L Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43:e1, 2020

    2020

  34. [43]

    Sgd on neural networks learns functions of increasing complexity

    Preetum Nakkiran, Dimitris Kalimeris, Gal Kaplun, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Adv. in Neural Information Processing Systems (NeurIPS), 2019

    2019

  35. [44]

    Deep learning generalizes because the parameter-function map is biased towards simple functions

    Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint. arXiv:1805.08522, 2018

    2018 arXiv

  36. [45]

    Deep neural networks have an inbuilt occam’s razor

    Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A Louis. Deep neural networks have an inbuilt occam’s razor. Nature Communications, 16(1):220, 2025

    2025

  37. [46]

    Stochastic collapse: How gra- dient noise attracts sgd dynamics towards simpler subnetworks.Advances in Neural Information Processing Systems, 36:35027–35063, 2023

    Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gra- dient noise attracts sgd dynamics towards simpler subnetworks.Advances in Neural Information Processing Systems, 36:35027–35063, 2023

    2023

  38. [47]

    The surprising simplicity of the early-time learning dynamics of neural networks

    Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. Adv. in Neural Information Processing Systems (NeurIPS), 2020

    2020

  39. [48]

    Simplicity bias in trans- formers and their ability to learn sparse boolean functions

    Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in trans- formers and their ability to learn sparse boolean functions. arXiv preprint arXiv:2211.12316, 2022

    2022 arXiv

  40. [49]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  41. [50]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  42. [51]

    What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization

    Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420, 2023

    2023 arXiv

  43. [52]

    Inductive biases and vari- able creation in self-attention mechanisms

    Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and vari- able creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793–5831. PMLR, 2022

    2022

  44. [53]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022

    2022 arXiv

  45. [54]

    In-context learning dynamics with random binary sequences

    Eric J Bigelow, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, and Tomer D Ullman. In-context learning dynamics with random binary sequences. arXiv preprint arXiv:2310.17639, 2023. 14

    2023 arXiv

  46. [55]

    In-context learning agents are asymmetric belief updaters

    Johannes A Schubert, Akshay K Jagadish, Marcel Binz, and Eric Schulz. In-context learning agents are asymmetric belief updaters. arXiv preprint arXiv:2402.03969, 2024

    2024 arXiv

  47. [56]

    Gpt-neox, huggingface., 2025

    GPT-NeoX. Gpt-neox, huggingface., 2025. URL https://huggingface.co/docs/ transformers/en/model_doc/gpt_neox

    2025

  48. [57]

    Gpt- neox-20b: An open-source autoregressive language model, 2022

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt- neox-20b: An op...

    2022 arXiv

  49. [58]

    In Wikipedia, December 2024

    Urn problem. In Wikipedia, December 2024. URL https://en.wikipedia.org/wiki/ Urn_problem

    2024

  50. [59]

    Language modeling is compression

    Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023

    2023 arXiv

  51. [60]

    Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. arXiv preprint arXiv:2406.02550, 2024

    2024 arXiv

  52. [61]

    General-purpose in- context learning by meta-learning transformers

    Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in- context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022

    2022 arXiv

  53. [62]

    In-context learning through the bayesian prism,

    Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism,

  54. [63]

    Bauer, Andrew Saxe, and Christopher Summerfield

    Jirko Rubruck, Jan P. Bauer, Andrew Saxe, and Christopher Summerfield. Early learning of the optimal constant solution in neural networks and humans, 2024. URL https://arxiv.org/ abs/2406.17467

    2024 arXiv

  55. [65]

    Complexity-based induction systems: comparisons and convergence theorems

    Ray Solomonoff. Complexity-based induction systems: comparisons and convergence theorems. IEEE transactions on Information Theory, 24(4):422–432, 1978

    1978

  56. [66]

    Laws of information conservation (nongrowth) and aspects of the foundation of probability theory

    Leonid Anatolevich Levin. Laws of information conservation (nongrowth) and aspects of the foundation of probability theory. Problemy Peredachi Informatsii, 10(3):30–35, 1974

    1974

  57. [67]

    An introduction to Kolmogorov complexity and its applications, volume 3

    Ming Li, Paul Vitányi, et al. An introduction to Kolmogorov complexity and its applications, volume 3. Springer, 2008

    2008

  58. [68]

    A review of methods for estimating algorithmic complexity: Options, challenges, and new directions

    Hector Zenil. A review of methods for estimating algorithmic complexity: Options, challenges, and new directions. Entropy, 22(6):612, 2020

    2020

  59. [69]

    Shannon information and kolmogorov complexity

    Peter Grunwald and Paul Vitányi. Shannon information and kolmogorov complexity. arXiv preprint cs/0410002, 2004

    2004 arXiv

  60. [70]

    Compression complexity.arXiv preprint arXiv:1702.04779, 2017

    Stephen Fenner and Lance Fortnow. Compression complexity.arXiv preprint arXiv:1702.04779, 2017

    2017 arXiv

  61. [71]

    Input–output maps are strongly biased towards simple outputs

    Kamaludin Dingle, Chico Q Camargo, and Ard A Louis. Input–output maps are strongly biased towards simple outputs. Nature communications, 9(1):761, 2018

    2018

  62. [72]

    In-context learning and occam’s razor, 2025

    Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam’s razor, 2025. URL https: //arxiv.org/abs/2410.14086

    2025 arXiv

  63. [73]

    Bayesian scaling laws for in-context learning

    Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D Goodman. Bayesian scaling laws for in-context learning. arXiv preprint arXiv:2410.16531, 2024. 15

    2024

  64. [74]

    Explaining emergent in-context learning as kernel regression

    Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305.12766, 2023

    2023 arXiv

  65. [75]

    A widely applicable bayesian information criterion

    Sumio Watanabe. A widely applicable bayesian information criterion. The Journal of Machine Learning Research, 14(1):867–897, 2013

    2013

  66. [76]

    The local learning coefficient: A singularity-aware complexity measure

    Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure. arXiv preprint arXiv:2308.12108, 2023

    2023 arXiv

  67. [77]

    The developmental landscape of in-context learning, 2024

    Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning, 2024. URL https: //arxiv.org/abs/2402.02364

    2024 arXiv

  68. [78]

    Is in-context learning in large language models bayesian? a martingale perspective

    Fabian Falck, Ziyu Wang, and Chris Holmes. Is in-context learning in large language models bayesian? a martingale perspective. arXiv preprint arXiv:2406.00793, 2024

    2024 arXiv

  69. [79]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  70. [80]

    The rational analysis of memory

    Samuel J Gershman. The rational analysis of memory. Oxford handbook of human memory., 2021

    2021

  71. [81]

    Resource-rational decision making

    Rahul Bhui, Lucy Lai, and Samuel J Gershman. Resource-rational decision making. Current Opinion in Behavioral Sciences, 41:15–21, 2021

    2021

  72. [82]

    Asymptotic theory of in-context learning by linear attention

    Yue M Lu, Mary I Letey, Jacob A Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405.11751, 2024

    2024

  73. [83]

    Learning to learn causal models

    Charles Kemp, Noah D Goodman, and Joshua B Tenenbaum. Learning to learn causal models. Cognitive science, 34(7):1185–1243, 2010

    2010

  74. [84]

    Learning to learn functions

    Michael Y Li, Fred Callaway, William D Thompson, Ryan P Adams, and Thomas L Griffiths. Learning to learn functions. Cognitive science, 47(4):e13262, 2023

    2023

  75. [85]

    Learning the form of causal relationships using hierarchical bayesian models

    Christopher G Lucas and Thomas L Griffiths. Learning the form of causal relationships using hierarchical bayesian models. Cognitive Science, 34(1):113–147, 2010

    2010

  76. [86]

    Learning how to generalize

    Joseph L Austerweil, Sophia Sanborn, and Thomas L Griffiths. Learning how to generalize. Cognitive science, 43(8):e12777, 2019

    2019

  77. [87]

    Learning overhypotheses with hierarchical bayesian models

    Charles Kemp, Andrew Perfors, and Joshua B Tenenbaum. Learning overhypotheses with hierarchical bayesian models. Developmental science, 10(3):307–321, 2007

    2007

  78. [88]

    Learning to learn categories

    Amy Perfors and Joshua Tenenbaum. Learning to learn categories. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31, 2009

    2009

  79. [89]

    Recasting gradient-based meta-learning as hierarchical bayes

    Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.08930, 2018

    2018 arXiv

  80. [90]

    Model-agnostic meta-learning for fast adapta- tion of deep networks, 2017

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks, 2017. URL https://arxiv.org/abs/1703.03400

    2017 arXiv

  81. [91]

    An information-theoretic analysis of in-context learning

    Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information-theoretic analysis of in-context learning. arXiv preprint arXiv:2401.15530, 2024

    2024 arXiv

  82. [92]

    How capable can a transformer become? a study on synthetic, interpretable tasks.arXiv preprint arXiv:2311.12997, 2023

    Rahul Ramesh, Mikail Khona, Robert P Dick, Hidenori Tanaka, and Ekdeep Singh Lubana. How capable can a transformer become? a study on synthetic, interpretable tasks.arXiv preprint arXiv:2311.12997, 2023

    2023 arXiv

  83. [93]

    Function vectors in large language models

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023. 16

    2023 arXiv

  84. [94]

    Transformers learn to imple- ment preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614–45650, 2023

    2023

  85. [95]

    Many-shot jailbreaking

    Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Advances in Neural Information Processing Systems, 37:129696–129742, 2024

    2024

  86. [96]

    Sometimes i am a tree: Data drives unstable hierarchical generalization

    Tian Qin, Naomi Saphra, and David Alvarez-Melis. Sometimes i am a tree: Data drives unstable hierarchical generalization. arXiv preprint arXiv:2412.04619, 2024

    2024

  87. [97]

    Lectures on phase transitions and the renormalization group

    Nigel Goldenfeld. Lectures on phase transitions and the renormalization group. CRC Press, 2018

    2018

  88. [98]

    Stochastic relaxation, gibbs distributions, and the bayesian restoration of images

    Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, PAMI-6 (6):721–741, 1984. 17 Appendix Table of Contents A Glossary of Useful Terms 19 B Related Wo...

    1984

  89. [100]

    while investigating the effects of data-centric properties on ICL, the term was popularized by Singh et al. [24]. Specifically, focusing on an in-context classification task, Singh et al

  90. [101]

    Linear Interpolation of Algorithms

    showed that a model’s ability to perform the generalizing ICL solution (employing a copy mechanism via the induction head) goes away when trained long enough. This phenomenon was recently generalized to a Markov modeling task by Park et al. [26] and to simplified variants of t...

  91. [102]

    " "" "

    focused only on the absolute distance between the Transformer and the generalizing solution. However, when we considering a relative distance measure, we can show this claim to be false (see right side of Fig. 28): Even in conditions in which the task diversity threshold was r...

  92. [2024]

    URL https://arxiv.org/abs/2306.04891

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.