Pith. sign in

REVIEW 3 major objections 4 minor 38 references

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims transformers trained on task hierarchies pick the simplest hypothesis that fits the in-context examples, and explains the behavior as an in-context Bayesian Occam's razor.

desk verdict Clean empirical demonstration of simplicity preference in ICL with nested testbeds, but the Bayesian theory leans on an unproven optimality assumption and the regression appendix has a sign error. read the letter →

arxiv 2506.19351 v1 pith:5ES5UFTR submitted 2025-06-24 cs.LG cs.AIcs.CLstat.ML

Puneesh Deora , Bhavya Vasudeva , Tina Behnia , Christos Thrampoulidis This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML
keywords in-contextlearningOccam'srazorBayesianinferencetransformersMarkovchainslinearregressionmodelcomplexityhypothesisselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what a transformer does when the in-context examples it is given could have been generated by either a simple rule or a more complicated rule that contains the simple one as a special case. Training transformers on mixtures of Markov chains of different orders, and on linear-regression tasks whose weight vectors live in spaces of different dimensionality, the authors find the models consistently act as if they pick the least complex hypothesis that still explains the context: a sequence from an order-1 chain is answered using the frequencies of the immediately preceding symbol, and a sparse regressor is answered with the restricted least-squares solution, even when a higher-order or higher-dimensional explanation fits the data just as well. The paper explains the behavior as in-context Bayesian Occam's razor: the model's output is a posterior-weighted mixture of per-category predictors, and a BIC-style complexity penalty concentrates the posterior on the simplest sufficient class. A case study on GPT-4 with Boolean functions suggests the same inductive bias may be present in large pretrained models.

What carries the argument

The argument turns on two identities. First, the Bayes-optimal mixture formula $p(x_{T+1}=v|X) = \sum_{s\in\mathrm{ord}} p(s|X)\,p(x_{T+1}=v|X,s)$, which writes the model's output as a convex combination of the single-order $s$-gram statistics, weighted by the posterior over complexity classes; the model's in-context job reduces to computing those posterior weights. Second, the BIC-style marginal-likelihood approximation $\log p(X|s) \approx \sum_t \log \hat p_X(x_t | x_{t-1},\dots,x_{t-s}) - \frac{V^s(V-1)}{2}\log T$, which decomposes the evidence for a class into empirical fit minus a complexity penalty proportional to $V^s$ and to $\log T$. Because the fit terms coincide for every $s \geq s^*$ when the simple class generated the data, the penalty alone drives the posterior to the smallest order; for genuinely complex data the fit advantage grows linearly in $T$ and wins the trade-off. The appendix shows a two-layer attention-only transformer can in principle compute the empirical conditional probabilities that feed these quantities, and the same logic runs through the linear-regression case, where the relevant likelihood ratio is governed by Wishart log-determinants scaling as $\Theta(d\log d)$.

What would settle it

Feed a trained transformer (trained on an equal mixture of order-1 and order-3 chains) order-1 prompts engineered so that their empirical tetragram likelihood beats the bigram likelihood by more than the BIC penalty $V^3(V-1)\log T / 2$; the Bayesian account predicts the output should shift toward tetragram statistics once that threshold is crossed. A model that keeps emitting bigram statistics no matter how large the complex class's fit advantage grows would falsify the claimed mechanism, though not the raw simplicity preference.

Watch

Extended reading notes

Core claim

The central claim is that transformers trained on task hierarchies in which the complex class strictly contains the simple class perform in-context Occam's razor: given a prompt compatible with both a simple and a more complex generative hypothesis, the model acts as if it selected the least complex sufficient explanation and predicts accordingly. In the Markov-chain testbed, a model trained on a mixture of order-1 and order-3 chains outputs bigram statistics on order-1 prompts and tetragram statistics on order-3 prompts, rather than always using the most expressive statistics it was trained on. In the linear-regression testbed, in the regime $d > T \geq d/2$ where both the full-dimensional and the restricted least-squares solutions perfectly interpolate the context, the model's predictions track the restricted $d/2$-dimensional solution when the true regressor is sparse. The theoretical account is that the trained model implements the Bayes-optimal predictor for the mixture, whose output is the posterior-weighted combination $p(x_{T+1}=v|X) = \sum_s p(s|X)\,p(x_{T+1}=v|X,s)$, and a BIC-style approximation to the marginal likelihood $p(X|s)$ produces a complexity penalty $V^s(V-1)\log T / 2$ that concentrates the posterior on the simplest class when the simple class generated the data. When the complex class generated the data, the better empirical fit of the complex class accumulates linearly in $T$ and outweighs the penalty, saturating the posterior to the true higher order.

Load-bearing premise

The load-bearing assumption, stated in Section 3.1.3, is that the trained transformer has enough training data to implement the Bayes-optimal predictor over the task mixture; the paper proves that the optimal predictor prefers the simplest sufficient class and shows a two-layer transformer could represent the needed statistics, but it does not prove gradient training reaches that predictor, so if trained transformers settle elsewhere the Bayesian explanation does not follow even if the observed simplicity preference is genuine.

Editorial extensions

If this is right

  • A transformer trained on a mixture of nested task classes performs hidden model selection in context: it identifies which complexity class generated the prompt and predicts with that class's sufficient statistics, rather than defaulting to the most expressive class.
  • Exposure to both classes is required for the razor: a transformer trained only on the complex class keeps using the complex statistics even on simple prompts, so the simplicity preference is a learned property of the training mixture, not a built-in bias.
  • The behavior reproduces in probabilistic context-free grammars and in a pretrained GPT-4 on ambiguous Boolean-function prompts, indicating the inductive bias may be a general property of transformers trained on diverse task distributions.
  • The ablations show that larger models acquire the correct per-class statistics faster, and that the composition of the training mixture matters: a high fraction of simple sequences delays learning the complex statistics, an effect offset by larger batches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The BIC-style penalty grows with the alphabet size $V^s$ and with $\log T$, which suggests a quantitative prediction the paper does not test: the simplicity preference should sharpen as the vocabulary grows and the context lengthens, and the context length at which the posterior switches to the complex class should trace the penalty curve.
  • The theory characterizes the Bayes-optimal predictor rather than the training dynamics, so the empirical mixture-proportion results in the paper suggest a direct test of the mechanism: extract the category posterior from a trained model's output mixture and compare it with the exact Dirichlet-multinomial posterior over held-out prompts.
  • If the bias survives scaling, one could probe it adversarially: prompts engineered so that a complex rule fits the demonstrations overwhelmingly better than a simple one should reveal whether the razor is a genuinely Bayesian trade-off or a hard prior toward simplicity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies in-context learning when the training task distribution is a mixture of two hierarchically related complexity classes, where the complex class is a strict superset of the simple class. In three synthetic testbeds—Markov chains of order 1 vs. order k, linear regression with d/2- vs. d-dimensional regressors, and a PCFG whose complex grammars contain the simple grammars—the authors report that transformers infer the task category from the in-context sequence and predict with the statistics of the simpler class when the sequence was generated by the simple class, while using the more expressive class when the context requires it. They explain this as 'in-context Bayesian Occam's razor': the Bayes-optimal predictor over the mixture is a posterior-weighted combination of per-category predictors, and the posterior concentrates on the simplest sufficient hypothesis (Eqs. (6)-(8) and Section 3.2.3). The paper also presents ablations on training mixture, model scale, context length, and architecture, compares with LSTMs, and reports a GPT-4 Boolean-function case study.

Significance. The core empirical phenomenon—transformers trained on a hierarchy prefer the least complex sufficient explanation in these controlled settings—is convincingly demonstrated by the Markov-chain, PCFG, and qualitatively by the linear-regression experiments, and the ablations (mixture proportion, model scale, LSTM comparison) strengthen the empirical contribution. The Bayesian framework is a natural and potentially useful unifying explanation, and Appendix D's explicit construction shows that a two-layer attention-only transformer can represent the conditional probabilities needed to compute the posterior. However, the paper's theoretical claim is substantially stronger than what is established: the Bayes-optimality premise is assumed rather than derived, and the linear-regression posterior derivation contains an inequality that is false in the regime where it is used. The empirical comparisons also do not directly test the predicted posterior mixture. If the requested corrections are made, this would be a solid contribution to the in-context-learning and algorithm-selection literature; as written, the central theoretical claim is conditional on an unverified assumption.

major comments (3)
  1. [Section 3.1.3, Eqs. (6)-(8)] The central theoretical claim that transformers implement in-context Bayesian Occam's razor rests entirely on the premise stated in Section 3.1.3: 'we assume sufficient training data such that the transformer implements the Bayes optimal minimizer.' No argument or evidence is given that gradient-trained transformers reach this predictor for the hierarchical mixtures of Eq. (4); Appendix D proves only representability by a two-layer attention-only transformer, not convergence under training, and Section 5 explicitly defers training dynamics to future work. The empirical sections do not close this gap: Figs. 2, 3, and 8 compare model outputs to pure n-gram statistics, whereas Eq. (6) predicts a posterior-weighted mixture, and the posterior weights p(s|X) are never measured. A non-Bayesian hard-switch rule (bigram when the context looks order-1, tetragram when it looks order-3) would produce the same qualitative KL pattern on the two trained orders, so the current evidence does not distinguish the proposed mechanism from a simpler selection rule. The order-2 results in Section 4.2 and Fig. 8, where the model neither commits to bigram nor tetragram, are exactly the regime that could discriminate the mixture from a hard switch, but no comparison to the Bayes posterior of Eq. (6) is made. The theoretical explanation should be reframed as conditional, or the posterior mixture should be tested quantitatively.
  2. [Appendix B.3, final paragraph] The claim that ||wLS_d||^2 - ||wLS_{d/2}||^2 > 0 is false in the regime d > T >= d/2 in which it is used. Since wLS_d is the minimum-norm interpolant over all d-dimensional interpolants and the zero-padded wLS_{d/2} is also an interpolant, the opposite inequality ||wLS_d||^2 <= ||wLS_{d/2}||^2 holds by the definition of the minimum-norm solution; the difference is generically negative. This invalidates the derivation as written in the final paragraph of Appendix B.3. The conclusion that Ld/2/Ld is large may still be salvageable because the determinant ratio is exponentially large and the norm term enters with the opposite sign, but the argument must be corrected before the linear-regression theory can be accepted.
  3. [Section 3.2.2 and Fig. 4] The decisive interpolating regime d > T >= d/2 is not actually realized in the settings reported in Fig. 4 as described. The caption states that the transformer was trained on T=39-long sequences with d=10 and d=20. For d=10 the condition d > T cannot hold, and for d=20 it holds only if the evaluation context length is strictly between 10 and 20. The main text and Fig. 4 do not state the evaluation prompt length, so the comparison between wLS_d and wLS_{d/2} in the interpolating regime is unverifiable. Appendix A.1 uses Ttest=15 for the d=20 setting, which is in the claimed regime, but this must be stated in the main text or figure caption; otherwise the central Occam's-razor claim for linear regression is not properly supported.
minor comments (4)
  1. [Section 2.1, Eq. (2)] There is a typo: 'birgram statistics' should be 'bigram statistics'.
  2. [Section 2.1 and Figs. 2, 3] The direction of the KL divergence is not consistently specified: the text says 'KL divergence between the model's output distribution and n-gram statistics' while later captions write 'KL(bigram∥ model)'. Since KL is asymmetric, the paper should define the direction used and keep it consistent.
  3. [Appendix B.3, Eq. (14)] Equation (14) contains an unbalanced parenthesis and the surrounding text includes an incomplete sentence ('InWe will first calculate the difference'), indicating that the appendix needs a careful copyedit.
  4. [Fig. 4 caption] The caption says 'w refers to the two benchmark least-squares solutions wLS_{d/2} or wLS_d' but does not state the evaluation context length or clarify which panel corresponds to which benchmark; please state these details in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical KL comparisons are independent; the Bayes-optimality premise is an explicit idealization, not a fitted input.

full rationale

The paper's central empirical claim—that transformers trained on order-1/order-3 Markov mixtures use bigram statistics on order-1 prompts and tetragram statistics on order-3 prompts—is established by direct KL-divergence comparisons to n-gram statistics computed from the prompt (Figs. 2, 3); these baselines are neither fitted parameters nor derived from the model, so the behavior is not manufactured by the theory. The Bayesian account (Sec. 3.1.3) is a conditional explanation: it explicitly states 'we assume sufficient training data such that the transformer implements the Bayes optimal minimizer' and then derives properties of the Bayes-optimal predictor (Eqs. 6-8). The model output is identified with the Bayes mixture only under this stated premise; the premise is not itself obtained by fitting the data, and the empirical tests would have disproved the account had the model defaulted to tetragram statistics on order-1 prompts. The construction in Appendix D shows representability by a two-layer attention-only transformer, not trainability, and the paper itself lists training dynamics as future work; these are gaps in support, not circular reductions. The regression derivation in Appendix B.3 contains a false inequality (the zero-padded d/2-dimensional interpolant has norm at least that of the minimum-norm d-dimensional interpolant, so the claimed positive difference cannot hold), but this is a correctness error independent of any input-output circularity. No load-bearing self-citation chain is present; cited Bayesian and induction-head results are external works. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted to the reported results; the priors (Dir(1), N(0,I), uniform category prior) are standard modeling choices that define the tasks. The Bayes optimality assumption is the main unproven premise. No new physical or mathematical entities are introduced; 'in-context Occam's razor' is a descriptive label for observed behavior, not a new mechanism.

assumptions (5)
  • domain assumption Bayes optimality of trained transformers in the mixture setting (Section 3.1.3: 'we assume sufficient training data such that the transformer implements the Bayes optimal minimizer')
    Bridges the empirical behavior to the Bayesian theory; unproven for the hierarchical mixture, only shown representable by a constructed 2-layer transformer (Appendix D).
  • standard math BIC-style Laplace approximation to the marginal likelihood for Markov chains (Eq. (7))
    Standard asymptotic result for nested model selection; used to claim the posterior concentrates on the simpler order when the simple process generated the data.
  • standard math Per-sample log-likelihood improvement of the true higher-order model over the lower-order model grows linearly in T for T >> k (Section 3.1.3)
    Standard property of likelihood ratios for Markov chains when the true process has order k > 1; cited to Ghosal and van der Vaart and Csiszar and Talata.
  • domain assumption Gaussian and Dirichlet priors in the training distributions (Eqs. (1), (3), (4), (10)), including a uniform prior over task categories
    Define the task distributions; the Bayesian interpretation uses these priors as the model's implicit prior.
  • standard math Marginal likelihood formulas L_d/2 and L_d and the Wishart/CLT approximations for log-determinants (Section B.3)
    Used to derive that the posterior concentrates on the d/2-dimensional regressor in the interpolating regime; relies on Gaussian feature assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly." pith.science (2026). https://pith.science/paper/5ES5UFTR

@misc{pith2026250619351,
  author       = {Pith},
  title        = {Pith review of: In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ES5UFTR}},
  note         = {Machine review of arXiv:2506.19351}
}
read the original abstract

In-context learning (ICL) enables transformers to adapt to new tasks through contextual examples without parameter updates. While existing research has typically studied ICL in fixed-complexity environments, practical language models encounter tasks spanning diverse complexity levels. This paper investigates how transformers navigate hierarchical task structures where higher-complexity categories can perfectly represent any pattern generated by simpler ones. We design well-controlled testbeds based on Markov chains and linear regression that reveal transformers not only identify the appropriate complexity level for each task but also accurately infer the corresponding parameters--even when the in-context examples are compatible with multiple complexity hypotheses. Notably, when presented with data generated by simpler processes, transformers consistently favor the least complex sufficient explanation. We theoretically explain this behavior through a Bayesian framework, demonstrating that transformers effectively implement an in-context Bayesian Occam's razor by balancing model fit against complexity penalties. We further ablate on the roles of model size, training mixture distribution, inference context length, and architecture. Finally, we validate this Occam's razor-like inductive bias on a pretrained GPT-4 model with Boolean-function tasks as case study, suggesting it may be inherent to transformers trained on diverse task distributions.

Figures

Figures reproduced from arXiv: 2506.19351 by the authors.

Figure 1
Figure 1. (Left) We train transformers on a hierarchy of task families in which the complex class strictly contains the simpler one (the simple category). (Middle) After training, the models use the in-context examples to infer both the appropriate complexity level and the task parameters that best fit the context, on the fly. (Right) We probe the same inductive bias in off-the-shelf LLMs: prompted with ambiguous examples tha… view at source ↗
Figure 2
Figure 2. We train a transformer for next-token prediction on sequences generated by random order-1 and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A transformer trained on sequences generated by random order-1 and order-2 Markov chains can [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: A transformer trained on T = 39-long sequences generated using random (a) d = 10 or (b) d = 20 dimensional regressors. We plot (Mθ([X, xtest]) − w⊤xtest) 2/d where w refers to the two benchmark least-squares solutions wLS d/2 or wLS d in Section 3. When prompted with u…
Figure 5
Figure 5. Figure 5: A transformer trained on sequences from ran [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Results with GPT-4 for in-context learning Boolean functions with two types of prompts. We [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Transformer trained on only the complex task does not exhibit Occam’s razor-like inductive bias. A transformer trained on random order-3 (left column) or order-2 (right column) Markov chains can only predict based on order-3 (left column) or order-2 (right column) stat…
Figure 8
Figure 8. Figure 8: Effect of context length on inference behaviour. Figure shows a trained transformer’s performance when prompted with sequences from different-order chains as a function of the sequence length. This transformer was trained on sequences from order-1 and order-3 chains wi…
Figure 9
Figure 9. Figure 9: Influence of training mixture proportion on inference behaviour. Each heat-map shows the KL divergence between the model’s output distribution and the indicated n-gram baseline as training progresses (horizontal axis), for different fractions of order-1 sequences in th…
Figure 10
Figure 10. Figure 10: Effect of model scale on inference behaviour. Each heatmap shows the KL divergence between the model’s output distribution and the indicated n-gram baseline as training progresses (horizontal axis), across different embedding dimensions (vertical axis). We report resu…
Figure 11
Figure 11. Figure 11: Comparison with LSTMs. Comparison between LSTMs and transformers of different capaci￾ties, trained on a mixture of order-1 and order-3 chains, and tested on unseen sequences at the end of training. We observe that LSTM models require significantly larger capacity than…
Figure 12
Figure 12. Figure 12: Normalized MSE (normalized by d) between the transformer’s predictions and the two benchmark solutions—d/2-LS (simple) and d-LS (complex)—over training steps. Models are trained with context length T = 40, with d = 20 (same setting as [PITH_FULL_IMAGE:figures/full_fi…
Figure 13
Figure 13. Figure 13: A transformer trained on X = [x1, y1, x2, y2, ..., xT , yT ] sequences generated using fixed complex￾ity, random d-dimensional regressors wd (see Section 3 for details). We plot (Mθ([X, xtest])−w⊤ benchxtest) 2/d where wbench refers to two benchmark least-squares solu…
Figure 14
Figure 14. Figure 14: Figure illustrates how varying the context length affects a trained transformer’s performance when [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Figure illustrates the performance of a trained transformer when prompted with sequences of [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Effect of model scale on inference behaviour. Each heatmap shows the KL divergence between the model’s output distribution and the indicated n-gram baseline as training progresses (horizontal axis), across different embedding dimensions (vertical axis). We report resu…
Figure 17
Figure 17. Figure 17: Repetition of the experiment in Fig [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Repetition of the experiment in Fig [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: The plots show the KL divergence between the model’s output and various [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 31 canonical work pages

  1. [1]

    Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. (2023). Transformers learn to implement preconditioned gradient descent for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

  2. [2]

    Aky \"u rek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. (2023). What learning algorithm is in-context learning? I nvestigations with linear models. In Int. Conference on Learning Representations

    work page 2023

  3. [3]

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. (2023). Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

  4. [4]

    Bhattamishra, S., Patel, A., Blunsom, P., and Kanade, V. (2024). Understanding in-context learning in transformers and LLM s by learning to learn discrete functions. In The Twelfth International Conference on Learning Representations

    work page 2024

  5. [5]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    2020

  6. [6]

    T., Liang, T., and Zhou, H

    Cai, T. T., Liang, T., and Zhou, H. H. (2015). Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional gaussian distributions. Journal of Multivariate Analysis , 137:161--172

    work page 2015

  7. [7]

    Chen, S., Sheen, H., Wang, T., and Yang, Z. (2024a). Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality

    work page 2024

  8. [8]

    Chen, S., Sheen, H., Wang, T., and Yang, Z. (2024b). Unveiling induction heads: Provable training dynamics and feature learning in transformers. arXiv preprint arXiv:2409.10559

    arXiv 2024

Show all 38 references
  1. [9]

    Collins, M. (2013). Probabilistic context\-/free grammars (pcfgs). Lecture Notes

    2013

  2. [10]

    and Talata, Z

    Csisz \'a r, I. and Talata, Z. (2006). Context tree estimation for not necessarily finite memory processes, via bic and mdl. IEEE Transactions on Information Theory , 52(3):1007--1016

    2006

  3. [11]

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., Sun, X., Li, L., and Sui, Z. (2024). A survey on in-context learning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the 2024 Conference on Empirical Methods in N...

    2024

  4. [12]

    L., Malach, E., and Goel, S

    Edelman, E., Tsilivis, N., Edelman, B. L., Malach, E., and Goel, S. (2024). The evolution of statistical induction heads: In-context learning markov chains. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [13]

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

    2021

  6. [14]

    Elmoznino, E., Marty, T., Kasetty, T., Gagnon, L., Mittal, S., Sridhar, D., and Lajoie, G. (2025). In-context learning and occam's razor

    2025

  7. [15]

    Fu, D., qi Chen, T., Jia, R., and Sharan, V. (2024). Transformers learn to achieve second-order convergence rates for in-context linear regression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  8. [16]

    Garg, S., Tsipras, D., Liang, P., and Valiant, G. (2022). What can transformers learn in-context? a case study of simple function classes. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems

    2022

  9. [17]

    and van der Vaart, A

    Ghosal, S. and van der Vaart, A. (2017). Fundamentals of Nonparametric Bayesian Inference , volume 44 of Cambridge Series in Statistical and Probabilistic Mathematics . Cambridge University Press

    2017

  10. [18]

    Karpathy, A. (2023). Mingpt. https://github.com/karpathy/minGPT/tree/master

    2023

  11. [19]

    E., Papailiopoulos, D., and Oymak, S

    Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. (2023). Transformers as algorithms: generalization and stability in in-context learning. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org

    2023

  12. [20]

    S., and Oymak, S

    Li, Y., Rawat, A. S., and Oymak, S. (2024). Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  13. [21]

    and Lee, K

    Lin, Z. and Lee, K. (2024). Dual operating modes of in-context learning. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org

    2024

  14. [22]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization

    2019

  15. [23]

    M., Letey, M

    Lu, Y. M., Letey, M. I., Zavatone-Veth, J. A., Maiti, A., and Pehlevan, C. (2025). Asymptotic theory of in-context learning by linear attention

    2025

  16. [24]

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirica...

    2022

  17. [25]

    Pan, J., Gao, T., Chen, H., and Chen, D. (2023). What in-context learning learns in-context: Disentangling task recognition and task learning. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages ...

    2023

  18. [26]

    Panwar, M., Ahuja, K., and Goyal, N. (2024). In-context learning through the bayesian prism

    2024

  19. [27]

    F., Lubana, E

    Park, C. F., Lubana, E. S., and Tanaka, H. (2025). Algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations

    2025

  20. [28]

    V., Ramchandran, K., and Gastpar, M

    Rajaraman, N., Bondaschi, M., Makkuva, A. V., Ramchandran, K., and Gastpar, M. (2024). Transformers on markov data: Constant depth suffices. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  21. [29]

    Raventos, A., Paul, M., Chen, F., and Ganguli, S. (2023). Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  22. [30]

    Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics , 6(2):461--464

    1978

  23. [31]

    von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. (2022). Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677

    2022 arXiv

  24. [32]

    Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. (2023). Label words are anchors: An information flow perspective for understanding in-context learning. In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empiric...

    2023

  25. [33]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. (2022). An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations

    2022

  26. [34]

    G., Oymak, S., Lee, K., and Papailiopoulos, D

    Xiong, Z., Cai, Z., Cooper, J., Ge, A., Papageorgiou, V., Sifakis, Z., Giannou, A., Lin, Z., Yang, L., Agarwal, S., Chrysos, G. G., Oymak, S., Lee, K., and Papailiopoulos, D. (2024). Everything everywhere all at once: Llms can in-context learn multiple tasks in superposition

    2024

  27. [35]

    Zhang, R., Frei, S., and Bartlett, P. L. (2023). Trained transformers learn linear models in-context

    2023

  28. [36]

    Zhang, R., Wu, J., and Bartlett, P. (2024). In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization. Advances in Neural Information Processing Systems , 37:18310--18361

    2024

  29. [37]

    K., Latham, P

    Zhang, Y., Singh, A. K., Latham, P. E., and Saxe, A. (2025). Training dynamics of in-context learning in linear attention

    2025

  30. [38]

    Zwiernik, P., Uhler, C., and Richards, D. (2016). Maximum likelihood estimation for linear gaussian covariance models

    2016

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.