Pith. sign in

REVIEW 3 major objections 3 minor 89 references

Decomposing Prediction Mechanisms for In-Context Recall

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a single in-context learning task can be performed by two distinct mechanisms: label-based recall for the first token after a sequence is resumed, and an observation-based Bayesian-style mechanism for the tokens…

desk verdict The OOD experiments cleanly separate initiation from continuation in a genuinely new toy, but the 0% edge-overlap circuit result is on a different model instance and should not be cited as mechanistic evidence until it is reproduced on the main checkpoint. read the letter →

arxiv 2507.01414 v2 pith:MGFJC5W7 submitted 2025-07-02 cs.LG

Sultan Daniels , Dylan Davis , Dhruv Gautam , Wentinn Liao , Gireeja Ranade , Anant Sahai This is my paper · ORCID
classification cs.LG
keywords in-contextlearningassociativerecallmechanisticinterpretabilityedgepruningemergencephasetransitionstransformerslineardynamicalsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a toy task in which transformers see interleaved segments of different linear dynamical systems, each tagged with symbolic open and close labels, and must predict the next observation when a previously seen system's label reappears. It argues that completing this one task requires at least two mechanisms that emerge at different times: a label-based recall mechanism that predicts the first token after a resumed sequence, and an observation-based mechanism that predicts later tokens without relying on the labels. The evidence comes from out-of-distribution manipulations and from edge pruning, which finds zero overlap between the sparse circuits for the first-token and later-token predictions. The authors elevate this into Conjecture C3: transformers use distinct mechanisms to initiate a new episode of an ICL-specified task versus continuing that task. They report a parallel first-token versus second-token performance gap in OLMo-2 7B checkpoints on an English-to-Spanish translation task, supporting the conjecture beyond the toy setting.

What carries the argument

The central object is the needle-in-a-haystack test trace: a context of N interleaved labeled segments, a query open label, and a test segment, with performance measured by median squared error at 1, 2, 3, 7, and 8 tokens after the query. The load-bearing structure is the noiseless orthogonal evolution $x_{i+1} = U x_i$, which makes each system identifiable from six observations and provides an optimal pseudoinverse predictor as a baseline. Edge pruning over continuous masks on a disentangled transformer is the mechanistic tool that isolates the two circuits, and the out-of-distribution misdirection and synchronization experiments are what separate the label-based and observation-based hypotheses behaviorally.

What would settle it

Run the synchronized-haystack test on the exact main-model checkpoint and then prune its 2-after circuit: if the model can use the final label to predict the second token when the first observation is ambiguous, or if the pruned 1-after and 2-after circuits share any edges, the paper's central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that for next-token prediction on interleaved, symbolically labeled time series from random orthogonal linear systems, the first token after a system is resumed is produced by recalling the system through its discrete symbolic label, while the second and later tokens are produced by a label-agnostic approximate Bayesian mechanism that uses the latest observation to infer which system is being continued. The key behavioral evidence is that misdirecting the final label to another haystack system flips only the first-token prediction, synchronizing all haystack sequences to the same state destroys later-token accuracy, and an unseen label triggers a late-training transition in later-token behavior. Edge pruning then finds that the 1-after and 2-after tasks are carried by disjoint sparse circuits, with 0% edge overlap. The authors conclude that a single ICL task is performed by multiple mechanisms with separate emergence times, and that the sharp phase transition in recall accuracy marks the arrival of the initiation mechanism, not the continuation mechanism.

Load-bearing premise

The load-bearing premise is that the two sparse circuits found by edge pruning, which came from an earlier training run using non-uniform orthogonal matrices and a single haystack configuration, are the same mechanisms that drive the behavior of the main model on which the out-of-distribution experiments were run.

Editorial extensions

If this is right

  • In the toy, the ability to continue a resumed sequence develops long before the ability to identify the sequence from its symbolic label, so the sharp phase transition in recall accuracy reports the arrival of the initiation mechanism, not of continuation ability.
  • In OLMo-2 7B, first-token English-to-Spanish translation accuracy improves abruptly and late while second-token accuracy improves gradually, and replacing meaningless 'X:' and 'Y:' labels with 'Spanish:' and 'English:' erases the gap, showing the gap is about in-context task recognition rather than the ability to translate.
  • Because the two mechanisms are separate, gradient pressure from most tokens can keep improving the observation-based continuation mechanism without improving label-based initiation, which can remain poor until a late phase transition.
  • Edge pruning finds 200-edge and 40-edge circuits for the 1-after and 2-after tasks with 0% edge overlap, so the two mechanisms are not merely behaviorally distinguishable but use disjoint computation paths.
  • The first token after an unseen label is treated as the start of a new sequence, and this treatment itself emerges abruptly late in training, meaning even the continuation mechanism is eventually modulated by whether the label is recognized as old or new.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 1-after and 2-after circuits share no edges, interventions that strengthen one mechanism, such as providing more demonstrations of a label-sequence association, should leave the other mechanism's performance unchanged; this is a direct and testable consequence of the paper's mechanistic claim.
  • The same first-token versus later-token gap should appear in any multi-token ICL task whose first token is underdetermined by local context and requires global retrieval, so one could test the conjecture by re-running the OLMo translation experiment with other synthetic labels or with a cloze task where the answer key appears early in the prompt.
  • The oscillation of first-token accuracy under the unseen 'Z:' label in Appendix D suggests that initiation circuits may be brittle under distribution shift, and tracking this across model scales could reveal whether initiation mechanisms are systematically harder to make robust than continuation mechanisms.
  • If the mechanism split is caused by gradient starvation, then upweighting first-token errors early in training should make the label-based recall circuit emerge earlier; this is a concrete training intervention that the paper's discussion implies but does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper introduces a toy task in which transformers are pretrained on interleaved, label-punctuated traces of random orthogonal linear systems, and asks how the model performs in-context associative recall. The authors identify two behaviors: predicting the first observation after a query label (initiation) and predicting subsequent observations (continuation). Training curves show that initiation emerges later and more abruptly than continuation. Four out-of-distribution interventions (label swap, synchronizing haystack rotations, unseen label, and seen-label misdirection) dissociate the two behaviors: the first token tracks the label, while later tokens depend on observations and are largely label-agnostic. Edge pruning on a small transformer yields disjoint sparse circuits for the two token positions (Table 1). The authors conjecture (C3) that transformers use separate mechanisms for initiating versus continuing an ICL-specified task, and they present OLMo-2 7B translation-checkpoint results as supporting evidence in a natural-language setting.

Significance. If the result holds, C3 is a useful refinement of the ICL literature: it predicts that a single multi-token task can be served by different mechanisms with different emergence times. The paper's main strengths are the carefully designed toy problem, the clean out-of-distribution behavioral battery, and the unusually candid reporting of limitations in the footnotes. The label-swap, synchronization, and unseen-label experiments are mutually consistent and provide strong behavioral evidence that the first token is computed differently from later tokens. The OLMo experiment with symbolic X:/Y: labels versus semantically meaningful labels is a nice falsifiable check. However, the mechanistic leg of C3 rests on edge-pruning results that the footnotes acknowledge were obtained on an earlier model instance with non-uniform orthogonal matrices and a single haystack configuration; until that is reproduced on the main checkpoint with a held-out split, the circuit-level claim is not established. Overall the central behavioral claim is defensible, but the paper currently overstates the mechanistic support.

major comments (3)
  1. [Section 4.2, Table 1 and Footnotes 10-11] The 0% edge overlap between the '1-after' and '2-after' circuits is load-bearing for the mechanistic half of Conjecture C3, but it is not measured on the model whose behavior was decomposed in Section 4.1. Footnote 11 states that the pruned model 'is from an earlier training run than the orthogonal medium model that is throughout the rest of this paper' and used a non-uniform distribution over orthogonal matrices, while Footnote 10 states that the pruning dataset was a single haystack configuration with no held-out split. Because edge pruning optimizes continuous masks, a single configuration with no validation can yield circuits that overfit that trace, and a different training run can have different circuits; in either case Table 1 does not establish that the main model uses disjoint computation graphs for the two token positions. Please reproduce the pruning on the main checkpoint with a held-out split, or soften the mechanistic claim in Section 4.2 and in the abstract to a behavioral dissociation.
  2. [Section 4.1.5] The summary of the out-of-distribution experiments concludes that 'the conjecture C3 ... holds true', but Experiments 1-4 are behavioral interventions; they demonstrate that the first-token prediction tracks the label while later-token predictions track observations, which is a functional dissociation. They do not by themselves establish that the two behaviors are implemented in separate computation graphs, which is what the edge-pruning analysis is meant to show. The conclusion should be staged so that the behavioral dissociation is stated first, and the circuit-level claim is conditional on the pruning evidence.
  3. [Section 4.2 and Appendix B] The reported 0% edge overlap is a single operating point: the circuits are pruned to a target sparsity of 0.98 using a binary-search threshold, and no sensitivity analysis is given for the overlap as a function of target sparsity or quantization threshold. Since the claim is that the mechanisms are 'completely distinct', the authors should show that the 0% overlap persists across a range of sparsity targets and thresholds; otherwise the result could reflect the particular pruning budget rather than a structural property of the model.
minor comments (3)
  1. [Appendix E, Table 2] In Table 2, the dhead and learning-rate columns appear visually concatenated (for example, '121.7×10−4' and '164.5×10−5'), making the hyperparameters unreadable; please insert separators or use explicit column boundaries.
  2. [Section 4.1.2] There is a grammatical typo in the sentence 'the solid blue curves ... are have a significantly higher squared-error than their counterpart curves in Fig. 11'; the word 'are' should be removed.
  3. [Section 4.2] The description of the pruning procedure switches between 'optimize continuous gates' and 'set all pruned edges to have weights of 0' without explaining how the quantized masks from Appendix B relate to zeroing weights; please clarify the exact inference-time operation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the behavioral and mechanistic claims are supported by falsifiable experiments whose outcomes were not engineered into the training distribution.

full rationale

The paper's central claims are the emergence-time split between 1-after and 2+-after prediction and the conjecture C3 that distinct mechanisms initiate versus continue a task. These claims are supported by four out-of-distribution inference-time interventions (label swap, synchronized rotations, unseen label, seen label) whose outcomes were not built into the training distribution. The behavioral dissociation is a measured result, not an assumption: label misdirection affects only the first token, while synchronizing rotations collapses later-token performance even though the label carries all necessary information. The mechanistic evidence is the edge-pruning result with 0% edge overlap between the 1-after and 2-after circuits; that overlap is a post-hoc finding, not a fitted parameter relabeled as a prediction. The pseudoinverse predictor is a baseline and does not enter the model's loss or the pruning objective as a fitted input. The paper contains no load-bearing self-citation chain; the cited prior work (e.g., Edge Pruning, orthogonal-matrix sampling, GPT-2-style training code) provides tools or baselines rather than the paper's conclusions. The footnotes about the pruned model coming from an earlier training run and the pruning dataset using a single haystack configuration identify robustness limitations, but they do not make the derivation circular: the pruning result is not equivalent to the claim by construction, and the behavioral half of the argument stands independently of the circuit transfer concern. Overall, no step in the claimed derivation reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the training-data construction (random labels, orthogonal dynamics), the optimality of the pseudoinverse baseline, and the validity of the out-of-distribution intervention logic. The listed free parameters are data-generation or analysis choices that shape the measurements; none is fit so as to produce the central mechanism decomposition, which comes from interventions on trained models. The pruning sparsity (0.98) and emergence cutoffs (0.4/0.5) affect derived quantities such as circuit edge counts and emergence timings, not the qualitative phase-transition finding. No new physical or architectural entities are postulated: H1 and H2 are hypotheses tested by intervention, not entities asserted to exist.

free parameters (4)
  • Zipf exponent for number of systems per trace = 1.5
    Data-generation choice in Section 2.2 step 2: N sampled from Zipf(1.5, 25). Motivated by Zipf's ubiquity, not by the mechanism claim; it shapes how often recall is needed during training.
  • Poisson rate for number of cuts = 2N
    Data-generation choice in Section 2.2 step 5: C ~ Poisson(2N). Sets segment statistics during training.
  • Edge pruning target sparsity = 0.98
    Section 4.2 and Appendix B: binary search finds the quantization threshold so that edge sparsity is close to 0.98. This method choice defines the reported circuit sizes (200 and 40 edges) and the 0% overlap claim.
  • Emergence error cutoffs = 0.4 (identity), 0.5 (orthogonal)
    Appendix E.1: 'These cutoff values were chosen by visual inspection.' Used to compute 'training examples to emergence' in Figs. 26-28. Transparently disclosed; affects only the derived emergence-time metric, not the qualitative phase-transition observation.
assumptions (5)
  • standard math Given 6 consecutive observations of an orthogonal system, the matrix U is uniquely recoverable (Eq. 3), making positions 7+ of a sequence perfectly predictable in principle.
    Section 2: this supports the pseudoinverse predictor baseline and the 'perfect prediction possible once enough context is seen' claim. Standard linear algebra.
  • domain assumption The Moore-Penrose pseudoinverse predictor (Eq. 4) is the appropriate optimal baseline for MSE on this noiseless task.
    Section 2.1: all trained-model comparisons are against this baseline; optimality for least-squares recovery of U is assumed.
  • domain assumption Symbolic labels are redrawn per training example, so label-to-system associations can only be formed in-context.
    Section 2.2 step 4: 'the same system can have different symbolic open and close labels when it appears in different training examples.' This construction is what makes the recall task in-context by definition.
  • domain assumption In the synchronized-rotations experiment, sharing x10 across all haystack systems removes disambiguating information from the first test-segment observation.
    Section 4.1.2: the conclusion that the model cannot use the label for index-2 predictions rests on this construction making x10 non-informative.
  • ad hoc to paper The OLMo English-to-Spanish task with X:/Y: labels is a faithful instance of in-context task recognition whose first-token success measures task initiation.
    Section 5 and Appendix C: a single translation direction and label scheme chosen to parallel the toy; the Spanish:/English: control motivates the interpretation, but the mapping from token position to mechanism is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decomposing Prediction Mechanisms for In-Context Recall." pith.science (2026). https://pith.science/paper/MGFJC5W7

@misc{pith2026250701414,
  author       = {Pith},
  title        = {Pith review of: Decomposing Prediction Mechanisms for In-Context Recall},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGFJC5W7}},
  note         = {Machine review of arXiv:2507.01414}
}
read the original abstract

We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall. We pretrain transformer models on sample traces from this toy, specifically symbolically-labeled interleaved state observations from randomly drawn linear deterministic dynamical systems. We study if the transformer models can recall the state of a sequence previously seen in its context when prompted to do so with the corresponding in-context label. Taking a closer look at this task, it becomes clear that the model must perform two functions: (1) identify which system's state should be recalled and apply that system to its last seen state, and (2) continuing to apply the correct system to predict the subsequent states. Training dynamics reveal that the first capability emerges well into a model's training. Surprisingly, the second capability, of continuing the prediction of a resumed sequence, develops much earlier. Via out-of-distribution experiments, and a mechanistic analysis on model weights via edge pruning, we find that next-token prediction for this toy problem involves at least two separate mechanisms. One mechanism uses the discrete symbolic labels to do the associative recall required to predict the start of a resumption of a previously seen sequence. The second mechanism, which is largely agnostic to the discrete symbolic labels, performs a "Bayesian-style" prediction based on the previous token and the context. These two mechanisms have different learning dynamics. To confirm that this multi-mechanism (manifesting as separate phase transitions) phenomenon is not just an artifact of our toy setting, we used OLMo training checkpoints on an ICL translation task to see a similar phenomenon: a decisive gap in the emergence of first-task-token performance vs second-task-token performance.

Figures

Figures reproduced from arXiv: 2507.01414 by the authors.

Figure 1
Figure 1. The generation of a train or test library of sequences. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of a 251-element-long interleaved training example. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distributions and complementary cumulative distribution functions (CCDFs) used in data [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (44 more)
Figure 4
Figure 4. Figure 4: The one-hot encoding of an open symbolic label. In this example, the system corresponding [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Generating a training example — Notice in this example the continuation from the first [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Needle-in-a-haystack test example. (A two system haystack.) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: When testing on 50 needle-in-a-haystack trace configurations, the order of system indices [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Performance on a long uninterleaved trace — [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Performance on new subsequent segments. 9a is the squared-error of predictions on steps 1 through 8 into the first and third system segments, where each segment is seen for the first time in context. 9b is the squared-error for steps 1 through 8 into the third system s…
Figure 10
Figure 10. Figure 10: Restarting for a new system at the early-stopping checkpoint after seeing [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Training dynamics for recall — The 25th , 50th, and 75th quartiles of the squared-error of the model’s predictions vs the number of training examples seen during training so far are plotted on log-log plots for N = 1 in Fig. 11a, N = 2 in Fig. 11b, and N = 5 in Fig. 1…
Figure 12
Figure 12. Figure 12: The 25th , 50th, and 75th quartiles of the squared-error after 6.25 × 107 training examples as the number of systems in the haystack N increases. Notice that predicting 1-after the open symbol is largely unaffected by the value of N, as the black markers stay steady a…
Figure 13
Figure 13. Figure 13: Misdirecting the model towards the incorrect sequence in the haystack. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Misdirection towards incorrect sequence — The median squared-error of the model’s [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Synchronizing previous systems in the haystack, so the first observation in the test segment [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Synchronizing rotations — The median squared-error of the model’s predictions on the test [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 17
Figure 17. Figure 17: Misdirecting the model with an unseen symbolic label indicating a new sequence. [PITH_FULL_IMAGE:figures/full_fig_p015_17.png]
Figure 18
Figure 18. Figure 18: Misdirection towards an unseen system — The median squared-error of the model’s [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: Misdirecting the model with a previously seen symbolic label. [PITH_FULL_IMAGE:figures/full_fig_p016_19.png]
Figure 20
Figure 20. Figure 20: Misdirection towards a seen system — The median squared-error of the model’s predictions [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]
Figure 21
Figure 21. Figure 21: Comparative example of in-weights associative recall (left) and in-context associative [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: Pretraining loss — The squared-error of each transformer model’s predictions on traces [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]
Figure 23
Figure 23. Figure 23: Recall and restart performance vs pretraining loss on held-out data. The red vertical line [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: 1-after final open symbol circuit in the orthogonal model. The output of the residual [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Comparison of the original in-context associative recall task (left) vs the in-context [PITH_FULL_IMAGE:figures/full_fig_p031_25.png]
Figure 26
Figure 26. Figure 26: Emergence of associative recall on different haystack lengths across model sizes — The [PITH_FULL_IMAGE:figures/full_fig_p032_26.png]
Figure 27
Figure 27. Figure 27: Emergence of associative recall in varying model sizes across haystack lengths for [PITH_FULL_IMAGE:figures/full_fig_p033_27.png]
Figure 28
Figure 28. Figure 28: Emergence of associative recall in varying model sizes across haystack lengths for Identity [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]
Figure 29
Figure 29. Figure 29: Performance of tiny orthogonal model (212K params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p034_29.png]
Figure 30
Figure 30. Figure 30: Performance of tiny orthogonal model (212K params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p034_30.png]
Figure 31
Figure 31. Figure 31: Performance of small orthogonal model (701K params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p035_31.png]
Figure 32
Figure 32. Figure 32: Performance of small orthogonal model (701K params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p035_32.png]
Figure 33
Figure 33. Figure 33: Performance of medium orthogonal model (2.42M params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p036_33.png]
Figure 34
Figure 34. Figure 34: Performance of medium orthogonal model (2.42M params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p036_34.png]
Figure 35
Figure 35. Figure 35: Performance of big orthogonal model (10.7M params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p037_35.png]
Figure 36
Figure 36. Figure 36: Performance of big orthogonal model (10.7M params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p037_36.png]
Figure 37
Figure 37. Figure 37: Performance of tiny identity model (212K params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p038_37.png]
Figure 38
Figure 38. Figure 38: Performance of tiny identity model (212K params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p038_38.png]
Figure 39
Figure 39. Figure 39: Performance of small identity model (701K params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p039_39.png]
Figure 40
Figure 40. Figure 40: Performance of small identity model (701K params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p039_40.png]
Figure 41
Figure 41. Figure 41: Performance of medium identity model (2.42M params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p040_41.png]
Figure 42
Figure 42. Figure 42: Performance of medium identity model (2.42M params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p040_42.png]
Figure 43
Figure 43. Figure 43: Performance of big identity model (10.7M params) across training — linear-scale. [PITH_FULL_IMAGE:figures/full_fig_p041_43.png]
Figure 44
Figure 44. Figure 44: Performance of big identity model (10.7M params) across training — log-scale. [PITH_FULL_IMAGE:figures/full_fig_p041_44.png]
Figure 45
Figure 45. Figure 45: Restarting for a new system — Prediction error vs. the position of a previously unseen [PITH_FULL_IMAGE:figures/full_fig_p042_45.png]
Figure 46
Figure 46. Figure 46: Comparison of recall performance for orthogonal and identity systems. Figs. [PITH_FULL_IMAGE:figures/full_fig_p043_46.png]
Figure 47
Figure 47. Figure 47: The effect of the needle position throughout training — The median-squared error vs. [PITH_FULL_IMAGE:figures/full_fig_p044_47.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 57 canonical work pages

  1. [1]

    In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973, 2024

    Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973, 2024

    arXiv 2024

  2. [2]

    Lepori, Jack Merullo, and Ellie Pavlick

    Suraj Anand, Michael A. Lepori, Jack Merullo, and Ellie Pavlick. Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting, 2025

    2025

  3. [3]

    Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    arXiv 2023

  4. [4]

    Copernicus, New York, NY , USA, 1996

    Per Bak.How nature works : the science of self-organized criticality. Copernicus, New York, NY , USA, 1996

    1996

  5. [5]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    2023

  6. [6]

    Finding transformer circuits with edge pruning

    Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  7. [7]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

Show all 89 references
  1. [9]

    Toward understanding in-context vs

    Bryan Chan, Xinyi Chen, András György, and Dale Schuurmans. Toward understanding in-context vs. in-weight learning, 2025

    2025

  2. [10]

    Stephanie C. Y . Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers, 2022

    2022

  3. [11]

    Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs

    Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  4. [12]

    Quantifying semantic emergence in language models, 2024

    Hang Chen, Xinyu Yang, Jiaying Zhu, and Wenya Wang. Quantifying semantic emergence in language models, 2024

    2024

  5. [13]

    Dynamical versus bayesian phase transitions in a toy model of superposition, 2023

    Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian phase transitions in a toy model of superposition, 2023

    2023

  6. [14]

    Unifying grokking and double descent

    Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. arXiv preprint arXiv:2303.06173, 2023

    2023 arXiv

  7. [15]

    Can transformers learn optimal filtering for unknown systems?IEEE Control Systems Letters, 7:3525–3530, 2023

    Zhe Du, Haldun Balim, Samet Oymak, and Necmiye Ozay. Can transformers learn optimal filtering for unknown systems?IEEE Control Systems Letters, 7:3525–3530, 2023. 21

    2023

  8. [16]

    Understanding emergent abilities of language models from the loss perspective, 2025

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2025

    2025

  9. [17]

    The evolution of statistical induction heads: In-context learning markov chains.arXiv preprint arXiv:2402.11004, 2024

    Benjamin L Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains.arXiv preprint arXiv:2402.11004, 2024

    2024 arXiv

  10. [18]

    A mathematical framework for transformer circuits.Transformer Circuits Thread,

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

  11. [19]

    Predictability and surprise in large generative models

    Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Tr...

    2022

  12. [20]

    What can transformers learn in-context? a case study of simple function classes.Advances in Neural Information Processing Systems, 35:30583–30598, 2022

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes.Advances in Neural Information Processing Systems, 35:30583–30598, 2022

    2022

  13. [21]

    Olmo: Accelerating the science of language models.arXiv preprint arXiv:2402.00838, 2024

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models.arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  14. [22]

    Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    Andrey Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    2023 arXiv

  15. [23]

    Loss landscape degeneracy drives stagewise development in transformers, 2025

    Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape degeneracy drives stagewise development in transformers, 2025

    2025

  16. [24]

    Neural networks and physical systems with emergent collective computational abilities.Proceedings of the national academy of sciences, 79(8):2554–2558, 1982

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities.Proceedings of the national academy of sciences, 79(8):2554–2558, 1982

    1982

  17. [25]

    Task descriptors help transformers learn linear models in-context

    Ruomin Huang and Rong Ge. Task descriptors help transformers learn linear models in-context. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  18. [26]

    Deep networks always grok and here is why

    Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why. InForty-first International Conference on Machine Learning, 2024

    2024

  19. [27]

    G. Kamradt. Needle in a haystack — pressure testing llms.https://github.com/gkamradt/ LLMTest_NeedleInAHaystack, 2023. GitHub repository

    2023

  20. [28]

    Grokking as the transition from lazy to rich training dynamics

    Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  21. [29]

    Andrew Kyle Lampinen, Stephanie C. Y . Chan, Aaditya K. Singh, and Murray Shanahan. The broader spectrum of in-context learning, 2024

    2024

  22. [30]

    The local learning coefficient: A singularity-aware complexity measure, 2024

    Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure, 2024

    2024

  23. [31]

    Surge phenomenon in optimal learning rate and batch size scaling.arXiv preprint arXiv:2405.14578, 2024

    Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. Surge phenomenon in optimal learning rate and batch size scaling.arXiv preprint arXiv:2405.14578, 2024

    2024 arXiv

  24. [32]

    Trans- formers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. InInternational Conference on Machine Learning, pages 19565–19594. PMLR, 2023

    2023

  25. [33]

    Dual operating modes of in-context learning, 2024

    Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning, 2024. 22

    2024

  26. [34]

    Can transformers solve least squares to high precision? InICML 2024 Workshop on In-Context Learning, 2024

    Jerry Weihong Liu, Jessica Grogan, Owen M Dugan, Simran Arora, Atri Rudra, and Christopher Re. Can transformers solve least squares to high precision? InICML 2024 Workshop on In-Context Learning, 2024

    2024

  27. [35]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  28. [36]

    Omnigrok: Grokking beyond algorithmic data

    Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  29. [37]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  30. [38]

    Are emergent abilities in large language models just in-context learning?arXiv preprint arXiv:2309.01809, 2023

    Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning?arXiv preprint arXiv:2309.01809, 2023

    2023 arXiv

  31. [39]

    Dick, and Hidenori Tanaka

    Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, and Hidenori Tanaka. A percolation model of emergence: Analyzing transformers trained on a formal language, 2024

    2024

  32. [40]

    Dichotomy of early and late phase implicit biases can provably induce grokking.arXiv preprint arXiv:2311.18817, 2023

    Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking.arXiv preprint arXiv:2311.18817, 2023

    2023 arXiv

  33. [41]

    A practical bayesian framework for backpropagation networks.Neural computation, 4(3):448–472, 1992

    David JC MacKay. A practical bayesian framework for backpropagation networks.Neural computation, 4(3):448–472, 1992

    1992

  34. [42]

    Exact learning dynamics of in-context learning in linear transformers and its application to non-linear transformers, 2025

    Nischal Mainali and Lucas Teixeira. Exact learning dynamics of in-context learning in linear transformers and its application to non-linear transformers, 2025

    2025

  35. [43]

    Emergence in non-neural models: grokking modular arithmetic via average gradient outer product

    Neil Rohit Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product. InNeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024

    2024

  36. [44]

    Hoffman, and David M

    Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference.Journal of machine learning research, 18, 2017

    2017

  37. [45]

    How to generate random matrices from the classical compact groups

    Francesco Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050, 2006

    2006 arXiv

  38. [46]

    The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36, 2023

    Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36, 2023

    2023

  39. [47]

    Rethinking the role of demonstrations: What makes in-context learning work?, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022

    2022

  40. [48]

    Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, and Ard A. Louis. Is sgd a bayesian sampler? well, almost.Journal of Machine Learning Research, 22(79):1–64, 2021

    2021

  41. [49]

    Grokking mod- ular arithmetic can be explained by margin maximization

    Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica Sutherland. Grokking mod- ular arithmetic can be explained by margin maximization. InNeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  42. [50]

    Transformers can do bayesian inference, 2024

    Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference, 2024

    2024

  43. [51]

    Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, and Ard A. Louis. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem, 2024

    2024

  44. [52]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InThe Eleventh International Conference on Learning Representations, 2023. 23

    2023

  45. [53]

    Differential learning kinetics govern the transition from memorization to generalization during in-context learning, 2024

    Alex Nguyen and Gautam Reddy. Differential learning kinetics govern the transition from memorization to generalization during in-context learning, 2024

    2024

  46. [54]

    Lee, and Alberto Bietti

    Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories, 2024

    2024

  47. [55]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2024

  48. [56]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  49. [57]

    What in-context learning "learns" in-context: Disentangling task recognition and task learning, 2023

    Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning, 2023

    2023

  50. [58]

    In-context learning through the bayesian prism, 2024

    Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism, 2024

    2024

  51. [59]

    Competition dynamics shape algorithmic phases of in-context learning, 2025

    Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning, 2025

    2025

  52. [60]

    Gradient starvation: A learning proclivity in neural networks.Advances in Neural Information Processing Systems, 34:1256–1272, 2021

    Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks.Advances in Neural Information Processing Systems, 34:1256–1272, 2021

    2021

  53. [61]

    Multi-scale feature learning dynamics: Insights for double descent

    Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio, and Guillaume Lajoie. Multi-scale feature learning dynamics: Insights for double descent. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th Internatio...

    2022

  54. [62]

    Grokking: Gen- eralization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    2022 arXiv

  55. [63]

    Grokking at the edge of numerical stability.arXiv preprint arXiv:2501.04697, 2025

    Lucas Prieto, Melih Barsbey, Pedro AM Mediano, and Tolga Birdal. Grokking at the edge of numerical stability.arXiv preprint arXiv:2501.04697, 2025

    2025 arXiv

  56. [64]

    Transformers on markov data: Constant depth suffices

    Nived Rajaraman, Marco Bondaschi, Ashok Vardhan Makkuva, Kannan Ramchandran, and Michael Gastpar. Transformers on markov data: Constant depth suffices. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  57. [65]

    Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression.Advances in Neural Information Processing Systems, 36, 2024

    Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression.Advances in Neural Information Processing Systems, 36, 2024

    2024

  58. [66]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023

    Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023

    2023

  59. [67]

    Michael Eli Sander, Raja Giryes, Taiji Suzuki, Mathieu Blondel, and Gabriel Peyré. How do transformers perform in-context autoregressive learning ? In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, edi...

    2024

  60. [68]

    Are emergent abilities of large language models a mirage?Advances in Neural Information Processing Systems, 36, 2024

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?Advances in Neural Information Processing Systems, 36, 2024. 24

    2024

  61. [69]

    I preliminaries

    Hinrich Schutze and Christopher Manning. I preliminaries. InFoundations of Statistical Natural Language Processing. MIT Press, United States, 1999

    1999

  62. [70]

    The pitfalls of simplicity bias in neural networks.Advances in Neural Information Processing Systems, 33:9573–9585, 2020

    Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks.Advances in Neural Information Processing Systems, 33:9573–9585, 2020

    2020

  63. [71]

    Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie C

    Aaditya K. Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie C. Y . Chan, and Andrew M. Saxe. Strategy coopetition explains the emergence and transience of in-context learning, 2025

    2025

  64. [72]

    Singh, Ted Moskovitz, Felix Hill, Stephanie C

    Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y . Chan, and Andrew M. Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation, 2024

    2024

  65. [73]

    The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19:1–57, 2018

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19:1–57, 2018

    2018

  66. [74]

    Transformers learn in-context by gradient descent

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. InInternational Conference on Machine Learning, pages 35151–35174. PMLR, 2023

    2023

  67. [75]

    Label words are anchors: An information flow perspective for understanding in-context learning, 2023

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning, 2023

    2023

  68. [76]

    Investigating the pre-training dynamics of in-context learning: Task recognition vs

    Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, and Ji-Rong Wen. Investigating the pre-training dynamics of in-context learning: Task recognition vs. task learning, 2024

    2024

  69. [77]

    Cambridge monographs on applied and computational mathematics ; 25

    Sumio Watanabe.Algebraic geometry and statistical learning theory / Sumio Watanabe. Cambridge monographs on applied and computational mathematics ; 25. Cambridge University Press, Cambridge ;, 2009

    2009

  70. [78]

    Emergent abilities of large language models.Transactions on Machine Learning Research, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.Transactions on Machine Learning Research, 2022

    2022

  71. [79]

    Symbol tuning improves in-context learning in language models

    Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al. Symbol tuning improves in-context learning in language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, ...

    2023

  72. [80]

    Larger language models do in-context learning differently, 2023

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023

    2023

  73. [81]

    The learnability of in-context learning, 2023

    Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning, 2023

    2023

  74. [82]

    Bartlett

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression?, 2024

    2024

  75. [83]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  76. [84]

    Which attention heads matter for in-context learning?, 2025

    Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning?, 2025

    2025

  77. [85]

    Learning to update auto-associative memory in recurrent neural networks for improving sequence memorization.arXiv preprint arXiv:1709.06493, 2017

    Wei Zhang and Bowen Zhou. Learning to update auto-associative memory in recurrent neural networks for improving sequence memorization.arXiv preprint arXiv:1709.06493, 2017

    2017 arXiv

  78. [86]

    Singh, Peter E

    Yedi Zhang, Aaditya K. Singh, Peter E. Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention, 2025. 25

    2025

  79. [87]

    grokking

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks.Advances in Neural Information Processing Systems, 36, 2024. 26 A Extended Related Work EmergenceBenchmark performance of large language...

    2024

  80. [89]

    tiny”, “small

    This scaling was used for the identity models and batch size was not taken into account. It was later brought to our attention that the learning rate should also scale with the batch size. For our later orthogonal runs, we additionally adopted the square-root learning-rate sca...

  81. [2021]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.