Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Next-token pretraining implies in-context learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model trained to predict the next token must learn from context: its per-token uncertainty falls as context grows.

desk verdict Good conditional theory of in-distribution ICL; the title's 'implies' is doing more work than the math supports. read the letter →

arxiv 2505.18373 v2 pith:XRII22C4 submitted 2025-05-23 cs.LG cs.AI

Paul M. Riechers , Henry R. Bigelow , Eric A. Alt , Adam Shai This is my paper
classification cs.LGcs.AI
keywords in-contextlearningnext-tokenpredictionmyopicentropyratemixed-statepresentationnon-ergodicprocesseshiddenMarkovmodelsinductionheadspower-lawscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that in-context learning—a model improving its predictions as it reads more context—is a mathematical consequence of minimizing ordinary next-token prediction loss, not an exotic emergent property. For any data distribution $Q$, the minimal expected loss equals the joint entropy, and the loss at each position is bounded below by the myopic entropy rate $h^Q_\ell$; a model that reaches that bound is reproducing the true conditional distribution at every context length. Because $h^Q_{\ell+1} \leq h^Q_\ell$ for stationary processes, a well-pretrained model's next-token uncertainty must decline as it observes more of a sequence. The paper shows how to compute the exact shape of that decline from the data's correlational structure, and verifies the predictions with transformers and RNNs trained on synthetic hidden Markov processes, reproducing phase transitions in training loss and power-law in-context loss. It also argues that a model's in-context performance on any specific task is coupled to the entire ensemble of tasks present in pretraining.

What carries the argument

The load-bearing mechanism is the mixed-state presentation (MSP): the Markov chain over predictive states induced by observing a process, where two histories are identified when they assign the same conditional distribution to all futures. For a hidden Markov model with token-dependent substochastic transition matrices $T(x)$, the mixed states are the normalized belief vectors $\eta_\varnothing T(x_1\cdots x_\ell)/(\eta_\varnothing T(x_1\cdots x_\ell)\mathbf{1})$, and the MSP gives a linear stochastic transition operator $W$ on these states. The central identity is $h^Q_\ell = \langle\delta_{\varepsilon(\varnothing)} | W^{\ell-1} | H(X|\mathcal{H})\rangle$, expressing the myopic entropy rate—the exact Bayes-optimal in-context loss—as a matrix element of this metadynamic. That identity converts in-context learning curves into linear algebra, allowing exact predictions of when the loss drops, whether it decays exponentially or as a power law, and why induction heads appear during training on non-ergodic mixtures.

What would settle it

Train a transformer on a stationary ergodic process with infinite Markov order (say the Even process described in the paper) and record per-position validation loss: the claim requires that a well-trained model's loss track $h^Q_\ell$ and never increase with context, so flat, rising, or above-rate per-position loss would falsify it. A sharper quantitative target is the non-ergodic continuum prediction $h^{Q_\infty}_\ell - h^{Q_\infty} \approx \tfrac12 \ell^{-1}$, which a many-coin experiment can check directly by measuring the log-log slope of the in-context loss curve.

Watch

Extended reading notes

Core claim

The central discovery is that a perfect next-token predictor is necessarily an in-context learner. If some parameter setting realizes the true distribution $Q(X_{1:L})$, then minimizing expected loss forces each per-position loss $c^{(\mathrm{Pr}_\theta,Q)}_\ell$ to equal the myopic entropy rate $h^Q_\ell$, and any mismatch with $Q$ at a given context strictly raises that position's loss. For stationary data, $h^Q_{\ell+1} \leq h^Q_\ell$, so the next-token entropy of a well-trained model must go down with context length—the signature of in-context learning—with no assumption about architecture. The paper further shows that the detailed trajectory of this entropy reduction is exactly computable from the generator of the data via the induced Markov chain over predictive (belief) states, and it confirms the predicted curves for a variety of stochastic processes. In the non-ergodic case, where each sequence is drawn from one of several components, the same argument predicts that induction heads are useful, that training loss shows a phase transition while these heads form, and that in-context loss decays as a power law, with exponent $-1$ in the limit of a continuum of components.

Load-bearing premise

The load-bearing premise is that a trained model can actually reach the Bayes-optimal conditional distribution at every context length; the entropy-rate theorem describes a perfect next-token predictor, and finite-capacity networks trained with stochastic gradient descent only approach it, so the claim rests on an empirical convergence property that the paper verifies on small synthetic processes.

Editorial extensions

If this is right

  • Any model that is well pretrained on a stationary correlated process must show in-context loss reduction on in-distribution sequences; if no such reduction appears, the model has not reached the entropy-rate bound.
  • The shape of in-context learning is set by the data process: finite-Markov-order processes stop improving after their Markov order, infinite-order processes improve exponentially, and context-free or non-ergodic sources improve as a power law.
  • Induction heads are a natural consequence of non-ergodic pretraining data: they disambiguate which ergodic component generated the current sequence, and their formation shows up as a phase transition in per-position training loss.
  • A model's in-context loss on one task is not determined by that task alone—it is mathematically coupled to the full mixture of tasks seen in pretraining.
  • On any context within the support of the training distribution, no model can beat the target process's myopic entropy rate; architectural differences matter mainly for out-of-support generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension of the entropy-rate argument, measuring the gap between a model's per-position loss and $h^Q_\ell$ on a known benchmark process gives a direct diagnostic of how close the model is to the data-generating distribution, with no downstream task evaluation needed.
  • Because the power-law result emerges from a continuum of hidden source parameters, a testable extension for real corpora is that the in-context loss exponent is set by the effective density of latent sources (topics, authors, styles); estimating that density from a corpus could predict ICL scaling before training.
  • A related editorial inference is that rare or statistically indistinguishable tasks in pretraining will show weak in-context learning even in large models, so rebalancing pretraining data should change the shape of the in-context loss curve, not just average task accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes an information-theoretic account of in-context learning (ICL). The authors show that for a model whose predictive distribution matches the data-generating process Q, the per-position next-token cross-entropy equals the myopic entropy rate h^Q_ℓ, which for stationary Q is nonincreasing in context length ℓ; hence context-dependent loss reduction is a necessary property of a Bayes-optimal next-token predictor. They construct mixed-state presentations to compute h^Q_ℓ exactly for HMM generators and verify the predictions with transformer and RNN training on synthetic processes, including infinite-order and nonergodic sources. For a nonergodic ensemble of N biased coins, they derive an asymptotic power-law decay of excess myopic entropy, h^{Q∞}_ℓ - h^{Q∞} ≈ (1/2)ℓ^{-1}, and connect induction heads to disambiguation of ergodic components.

Significance. The paper's exact, parameter-free predictions are a genuine strength: the theoretical loss curves are computed from the data-generating process rather than fitted to the trained models, and the reported agreement of transformers and RNNs with these curves on several synthetic processes is convincing. The mixed-state presentation framework gives a useful architecture-agnostic benchmark for in-distribution ICL, and the nonergodic power-law analysis cleanly demonstrates how ensemble structure can yield slow context-dependent loss decay. The main reservation concerns the scope of the 'implies' claim: the formal result is conditional on the model attaining Bayes-optimal conditional distributions, and the paper's definition of ICL as context-dependent loss reduction is narrower than the task-learning notion often used in the LLM literature. With those caveats stated explicitly, the contribution is a solid theoretical and experimental analysis.

major comments (3)
  1. [Section 2, Eq. (2) and following paragraph] The equality min_θ ⟨L_θ⟩ = H[Q] and the subsequent conclusion that c_ℓ = h^Q_ℓ at every ℓ require the existence of parameters θ* with Pr_{θ*}(X_{1:L}) = Q(X_{1:L}) for all contexts in the training window; this is a representability and optimization premise, not a consequence of minimizing next-token loss. Without this premise, only the lower bound c_ℓ ≥ h^Q_ℓ in Eq. (3) holds, and a finite-capacity model can allocate excess loss unevenly across positions, so the per-position loss need not equal the myopic entropy rate and need not be monotone decreasing. The paper's own examples include the Simple Nonunifilar source and Parentheses Matching, which have no finite-state or finite-context presentation, so no finite model can satisfy the premise exactly. The sentence 'For stationary processes, h^Q_{ℓ+1} ≤ h^Q_ℓ, so the next-token entropy of predictions from any model well pretrained ... must go down as it sees more context' and the title's 'implies' therefore overstate the theorem. I recommend stating the theorem explicitly as conditional on achieving the Bayes-optimal predictive distribution, and treating convergence to h^Q_ℓ as an empirically verified property rather than a mathematical consequence of next-token pretraining alone.
  2. [Section 2; Sections 4–5] The claim that the framework applies to 'any architecture' is not supported by the derivation. A fixed-context or finite-state architecture cannot represent Q(X_ℓ | X_{1:ℓ-1}) for ℓ beyond its memory, so Eq. (2)'s equality cannot hold for all ℓ; the architecture-independence conclusion requires an implicit unbounded-context or sufficient-capacity assumption. The experiments use transformers and RNNs with context length 100 and report loss mainly for the first ten positions, which does not test long-context representability for infinite-order processes. The paper should explicitly state the architectural requirements for the equality to hold and qualify statements such as 'Transformers, RNNs, or any other architecture optimized for next-token prediction should display these key features.'
  3. [Title, Abstract, and Section 1] The manuscript's formal result concerns in-distribution loss reduction for a model that already matches the training distribution Q; it does not address out-of-distribution task inference, which is the sense in which 'in-context learning' is most commonly used in the LLM literature. The abstract's phrase 'in-distribution ICL' is a step in the right direction, but the title and sentences such as 'the general ability of AI models to home in on their users and novel requests' imply a stronger claim than the theorems establish. I recommend either retitling the paper to refer to in-distribution context-dependent loss reduction, or explicitly defining the paper's ICL notion at the outset and stating that the out-of-distribution sense remains open.
minor comments (4)
  1. [Section 2, Eq. (2)] The phrase 'very well approximates the true distributions such that Pr_θ(X_{1:L}) = Q(X_{1:L})' mixes approximation with exact equality; please phrase as 'if some setting of the parameters realizes the true joint distribution exactly.'
  2. [Section 3.2, Eq. (4)] The expression for Q(X_{ℓ+1:ℓ+ℓ'}|X_{1:ℓ}=w) is easy to misread because the division is not parenthesized; please write (η_∅ T(w) T(X_{ℓ+1:ℓ+ℓ'}) 1) / (η_∅ T(w) 1), or introduce the belief-state vector before writing the conditional probability.
  3. [Section 5.1 and Appendix E.1, Eq. (9)] The power-law result is a double asymptotic statement: N → ∞ followed by ℓ → ∞. For finite N and small ℓ, the expected loss shown in Fig. 4 is not exactly a power law, and the 'halving' sentence in the text is only valid in that asymptotic regime. Please make the limit order explicit in the main text and avoid implying that every finite nonergodic ensemble exhibits a pure ℓ^{-1} decay.
  4. [Section 1 and References] The statement 'the general ability of AI models to home in on their users and novel requests' is presented as part of the explanation, but the formal results only cover in-distribution contexts; please mark clearly which claims are conjectured extensions rather than theorems. Also, reference [23] is a non-archival blog post; a peer-reviewed source would be more appropriate.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor semantic circularity: ICL is defined as context-dependent loss reduction, which is exactly what the central theorem then derives; the empirical per-position predictions remain falsifiable.

  1. self definitional [Section 1 ('is the essence of ICL') and Section 2 (paragraph after Eq. (3))]
    "This context-dependent reduction in loss, which we can precisely anticipate for certain correlated stochastic processes, is the essence of ICL. ... For stationary processes, hQℓ+1 ≤ hQℓ, so the next-token entropy of predictions from any model well pretrained on an approximately stationary process must go down as it sees more context."

    ICL is defined in the paper as 'context-dependent loss reduction,' and the central implication is that, at the Bayes-optimal solution of next-token loss, per-position loss equals the myopic entropy rate h^Q_ℓ, which is nonincreasing for stationary processes. Thus the conclusion 'ICL arises' is, to a large extent, a restatement of the definition of ICL together with the standard monotonicity of conditional entropy; the min-loss equality itself is Gibbs' inequality (cross-entropy is minimized by the true distribution). The non-definitional content is the exact computation of h^Q_ℓ for each toy process, which is validated by convergence of trained models, so the circularity is limited to labeling and framing rather than to the fitted predictions.

full rationale

The paper's quantitative predictions are computed from the generative process Q (myopic entropy rates from the mixed-state presentation of each HMM, and the N → ∞ beta-integral power-law calculation), then checked against independently trained transformers and RNNs. These theory curves are not fitted to the trained models, and a poorly trained model would fail to track them, so the empirical core is genuinely falsifiable. The equality min_θ ⟨L_θ⟩ = H[Q(X_{1:L})] is the definitional fact that cross-entropy is minimized by the true conditional distribution, and the paper states the representability premise explicitly ('If some setting of the parameters very well approximates the true distributions such that Prθ(X_{1:L}) = Q(X_{1:L})'); whether real networks satisfy it is an empirical convergence question, not a circular step. The cited prior work [15,16] supplies standard computational-mechanics machinery for computing entropy convergence and is not load-bearing for the inequality itself, and no imported uniqueness theorem is used. The only defensible circularity concern is semantic: the paper identifies ICL with context-dependent loss reduction, so the theorem's conclusion is partly contained in its definition of the target phenomenon. This warrants a low score (2) rather than a higher one, because the specific loss curves and scaling exponents are derived from the data source and are not equivalent to their inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The theoretical predictions contain no fitted parameters. In-context loss curves are computed directly from known HMM generators, and the power-law exponent -1 is derived from the Beta-function integral. The main assumptions are stationarity of the data process, the trained model approaching the Bayes-optimal conditional distribution at every context length, the standard MSP construction, and the specific continuum limit for the many-coins calculation.

assumptions (4)
  • domain assumption The training data is generated by a stationary stochastic process for the main monotonicity result.
    Section 2 uses h^Q_{ℓ+1} ≤ h^Q_ℓ, which holds for stationary processes; real text is only approximately stationary.
  • domain assumption The model family is expressive enough and training converges to the global minimum of expected next-token cross-entropy, so Prθ(X_{1:L}) = Q(X_{1:L}) is achievable at every context length.
    Eq. (2) sets the minimum equal to the joint entropy of the data; finite-capacity networks only approximate this.
  • standard math The mixed-state presentation of an HMM exactly captures the predictive states of the process.
    Standard computational mechanics result (Shalizi and Crutchfield); used in Sections 3.1 and 3.2 to compute h^Q_ℓ.
  • domain assumption For the N-coin power law, biases are the uniform grid n/(N+1) with equal component weights, and the N→∞ limit is exchanged with the ℓ→∞ limit.
    Appendix E.1; needed for the Beta-function integral that yields the closed form and the exponent -1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Next-token pretraining implies in-context learning." pith.science (2026). https://pith.science/paper/XRII22C4

@misc{pith2026250518373,
  author       = {Pith},
  title        = {Pith review of: Next-token pretraining implies in-context learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRII22C4}},
  note         = {Machine review of arXiv:2505.18373}
}
read the original abstract

We argue that in-context learning (ICL) predictably arises from standard self-supervised next-token pretraining, rather than being an exotic emergent property. This work establishes the foundational principles of this emergence by focusing on in-distribution ICL, demonstrating how models necessarily adapt to context when trained on token sequences, especially from non-ergodic sources. Our information-theoretic framework precisely predicts these in-distribution ICL dynamics (i.e., context-dependent loss reduction). We verify this with experiments using synthetic datasets of differing types of correlational structure, reproducing characteristic phenomena like phase transitions in training loss for induction head formation and power-law scaling of in-context loss. We further show that a model's in-context performance on any task is mathematically coupled to the ensemble of tasks seen in pretraining, offering a fundamental explanation, grounded in architecture- and modality-independent principles, for such inference-time learning.

Figures

Figures reproduced from arXiv: 2505.18373 by the authors.

Figure 1
Figure 1. Simple processes producing tokens with correlations of increasing complexity. In-context [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Simple processes beyond the paradigm of finite ergodic generators. As with the processes [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The need for induction heads can easily be understood via nonergodic processes. In this [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Experimental validation of in-context loss scaling behavior given by our framework. Left [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Same as Fig. 3, but trained on a GRU RNN. Interestingly, the same phase transition [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Final Checkpoint In-Context Loss for RNN and Transformer Models on [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Expected in-context loss (averaged over all coins) scales as a power-law with increasing [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Neural networks leverage nominally quantum and post-quantum representations

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Pretrained transformers and RNNs linearly encode the Bayesian-updated belief geometry of the minimal classical, quantum, or post-quantum generator of their training data.

Reference graph

Works this paper leans on

32 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  3. [3]

    Sparks of artificial general intelligence: Early experiments with GPT-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  4. [4]

    An explanation of in-context learning as implicit Bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. ICLR, 2022

    work page 2022

  5. [5]

    Bayesian scaling laws for in-context learning

    Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D Goodman. Bayesian scaling laws for in-context learning. arXiv preprint arXiv:2410.16531, 2024

    arXiv 2024

  6. [6]

    Transformers learn in-context by gradient descent

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Mac...

    work page 2023

  7. [7]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 45614–45650. Curran Associates, Inc., 2023. 10

    work page 2023

  8. [8]

    The broader spectrum of in-context learning

    Andrew Kyle Lampinen, Stephanie CY Chan, Aaditya K Singh, and Murray Shanahan. The broader spectrum of in-context learning. arXiv preprint arXiv:2412.03782, 2024

    arXiv 2024

Show all 32 references
  1. [9]

    Transformers represent belief state geometry in their residual stream

    Adam S Shai, Sarah E Marzen, Lucas Teixeira, Alexander Gietelink Oldenziel, and Paul M Riechers. Transformers represent belief state geometry in their residual stream. NeurIPS, 2024

    2024

  2. [10]

    Constrained belief updates explain geometric structures in transformer representations

    Mateusz Piotrowski, Paul M Riechers, Daniel Filan, and Adam S Shai. Constrained belief updates explain geometric structures in transformer representations. ICML, 2025

    2025

  3. [11]

    RNNs represent belief state geometry in their hidden states

    Keenan Pepper. RNNs represent belief state geometry in their hidden states. https://apartresearch.com, June 2024. Research submission to the Computational Mechan- ics Hackathon research sprint co-hosted by Apart, PIBBSS, and Simplex

    2024

  4. [12]

    C. R. Shalizi and J. P. Crutchfield. Computational mechanics: Pattern and prediction, structure and simplicity. J. Stat. Phys., 104:817–879, 2001

    2001

  5. [13]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

  6. [14]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  7. [15]

    P. M. Riechers and J. P. Crutchfield. Spectral simplicity of apparent complexity, Part I: The nondiagonalizable metadynamics of prediction. Chaos, 28:033115, 2018

    2018

  8. [16]

    P. M. Riechers and J. P. Crutchfield. Spectral simplicity of apparent complexity, Part II: Exact complexities and complexity spectra. Chaos, 28:033116, 2018

    2018

  9. [17]

    D. R. Upper. Theory and Algorithms for Hidden Markov Models and Generalized Hidden Markov Models. PhD thesis, University of California, Berkeley, 1997. Published by University Microfilms Intl, Ann Arbor, Michigan

    1997

  10. [18]

    J. P. Crutchfield. Between order and chaos. Nature Physics, 8(January):17–24, 2012

    2012

  11. [19]

    J. P. Crutchfield, C. J. Ellison, and P. M. Riechers. Exact complexity: Spectral decomposition of intrinsic computation. Phys. Lett. A, 380(9-10):998–1002, 2016

    2016

  12. [20]

    Shannon entropy rate of hidden markov processes

    Alexandra M Jurgens and James P Crutchfield. Shannon entropy rate of hidden markov processes. Journal of Statistical Physics, 183(2):32, 2021

    2021

  13. [21]

    Critical behavior in physics and probabilistic formal languages

    Henry W Lin and Max Tegmark. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299, 2017

    2017

  14. [22]

    J. P. Crutchfield and S. Marzen. Signatures of infinity: Nonergodicity and resource scaling in prediction, complexity, and learning. Phys. Rev. E, 91(5):050106(R), 2015. SFI Working Paper 15-04-010; arxiv.org:1504.00386 [cond-mat.stat-mech]

    2015 arXiv

  15. [23]

    Language models model us

    Egg Syntax. Language models model us. AI Alignment F orum, 2024

    2024

  16. [24]

    Predictability, complexity, and learning

    William Bialek, Ilya Nemenman, and Naftali Tishby. Predictability, complexity, and learning. Neural Computation, 13(11):2409–2463, 2001

    2001

  17. [25]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 11

    2001 arXiv

  18. [26]

    Meta-learning of sequential strategies

    Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030, 2019

    1905 arXiv

  19. [27]

    Data distributional properties drive emer- gent in-context learning in transformers

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. Advances in neural information processing systems , 35:18878–18891, 2022

    2022

  20. [28]

    Predictive information

    William Bialek and Naftali Tishby. Predictive information. arXiv preprint cond-mat/9902341, 1999

    1999 arXiv

  21. [29]

    J. P. Crutchfield and D. P. Feldman. Regularities unseen, randomness observed: Levels of entropy convergence. CHAOS, 13(1):25–54, 2003

    2003

  22. [30]

    Grassberger

    P. Grassberger. Toward a quantitative theory of self-generated complexity.Intl. J. Theo. Phys. , 25:907, 1986

    1986

  23. [31]

    g(k + 1, ℓ− k) log k + 1 ℓ + 2 + g(k, ℓ+ 1− k) log ℓ + 1− k ℓ + 2 # (34) = − ℓX k=0 ℓ! k! (ℓ − k)!

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. A Information theoretic interpretation of in-context loss For st...

    2017 arXiv

  24. [2021]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.