Pith. sign in

REVIEW 2 major objections 5 minor 81 references

Imbalanced pretraining separates task circuits so refusal fine-tuning stays selective instead of spilling over.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 12:35 UTC pith:RYJIALGX

load-bearing objection Solid controlled evidence that task-imbalance curricula produce separable circuits and cleaner selective refusal; the only real limit is scale. the 2 major comments →

arxiv 2607.04846 v1 pith:RYJIALGX submitted 2026-07-06 cs.LG cs.AI

Pretraining Curricula Enable Selective Fine-tuning

Sebastian A. Bruijns , Jirko Rubruck , Mia H. Whitefield , Kai J. Sandbrink , Fazl Barez , Christopher Summerfield This is my paper
classification cs.LG cs.AI
keywords pretraining curriculain-context learningrefusal fine-tuningcircuit disentanglementoverrefusalactivation patchingsynthetic language learningAI safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the order in which tasks appear during pretraining shapes how transformers store those tasks inside the network. When two conflicting copy tasks are presented in an imbalanced schedule—one early, one late—models reliably learn generalizing in-context solutions and later accept refusal fine-tuning that suppresses only the refused task. Balanced schedules, by contrast, more often produce memorization and route both tasks through a shared pathway, so refusing one task also refuses the other. The same pattern appears in a synthetic language setting with rule-following and rule-violating sentences: early emphasis on valid data yields more localized rule circuits and more reliable rule adherence after aligned fine-tuning. The practical stake is safety training: selective suppression of unwanted behaviors works more cleanly when pretraining has already disentangled the relevant circuits.

Core claim

Imbalanced pretraining curricula induce task-specialized circuitry that supports selective refusal fine-tuning with low overrefusal, whereas balanced pretraining routes both tasks through a common pathway that is co-opted during fine-tuning and produces substantial spillover. The same curriculum effect appears in a rule-based synthetic language task, where imbalance yields more localized early-layer rule representations and more robust rule-following after fine-tuning on aligned data.

What carries the argument

Curriculum-dependent circuit organization, diagnosed by mean ablations, direct logit attribution, and denoising activation patching: under imbalance, copy-first routes mainly through layer-2 attention (especially keys) and copy-last through layer-2 feed-forward, while balanced training recovers both tasks from the same layer-2 key pathway.

Load-bearing premise

The circuit separation and selective-fine-tuning gains seen in small transformers on two synthetic tasks will still matter for large models trained on many real, entangled behaviors.

What would settle it

Train multi-task models under matched imbalanced versus balanced pretraining schedules, then apply selective refusal fine-tuning; if overrefusal rates and shared-circuit recovery under activation patching are indistinguishable across curricula, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety fine-tuning that refuses one capability will leave related kept capabilities more intact when pretraining was imbalanced rather than interleaved.
  • Pretraining data order can be treated as a design lever for promoting disentangled representations, not only as a speed or difficulty schedule.
  • Surface-aligned free generation after fine-tuning can mask residual misalignment that only appears under targeted elicitation prompts.
  • Tasks learned together are more likely to share circuitry and therefore more likely to interfere under later selective unlearning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Ordering safety-relevant versus ordinary data early in large-scale pretraining corpora may be as consequential as the total fraction of each.
  • Cosine similarity between residual-stream task directions during fine-tuning could serve as a cheap early-warning signal for unwanted capability interference.
  • Curriculum design for alignment may need to prioritize when rule-consistent examples appear, not only how many of them exist.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies how explicit pretraining curricula shape circuit organization and the selectivity of subsequent fine-tuning in small transformers. In a controlled ICL setting with two conflicting copy tasks (copy-first vs copy-last), an imbalanced curriculum (one task early, annealed to 50:50) yields reliable generalization and low overrefusal under selective refusal fine-tuning, whereas a balanced 50:50 mixture often memorizes and, when it generalizes, routes both tasks through a shared L2-attention pathway that is co-opted during fine-tuning. Mean ablations, direct logit attribution, and denoising activation patching (with full recovery tables) support task-specialized circuitry under imbalance (copy-first primarily via L2 K, copy-last primarily via L2 FFN). A second synthetic language-learning task with rule-consistent vs rule-violating profiles shows that imbalanced curricula produce more localized early-layer condition-token attention and more robust rule adherence after aligned fine-tuning. The authors conclude that imbalanced pretraining can promote disentangled representations that improve the precision of safety-style fine-tuning.

Significance. If the reported circuit-level mechanism holds, the work supplies a concrete, testable link between pretraining data order, task disentanglement, and selective fine-tuning—directly relevant to overrefusal and safety alignment. Strengths include multi-seed evidence (up to 100 seeds for ICL), reverse and constant-imbalance controls, residual-stream cosine similarity and linear probes, causal interventions (mean ablation, DLA, denoising patching of L2 Q/K/V/FFN with recovery tables), and a second SLL setting with attention ablations and patching. The hyperparameter sweeps (2- and 4-layer nets; learning rate, batch size, width, weight decay, imbalance fraction) and the explicit external-validity caveat in §4 further strengthen the contribution as a careful mechanistic study rather than an overclaimed scaling result.

major comments (2)
  1. The central causal claim—that curriculum-induced circuit specialization drives selective refusal—is well supported within the 2–4 layer synthetic settings (Figs. 2–4, Table 3, residual cosine similarities in App. C.4, and the specialization–overrefusal correlation in Fig. 33). No load-bearing internal inconsistency was found. The remaining substantive limitation is external validity: §4 correctly notes that transfer to frontier-scale multi-task LLMs is untested. This does not invalidate the reported results, but the abstract and conclusion should keep the claim scoped to the demonstrated regimes and treat LLM safety implications as a hypothesis rather than a demonstrated consequence.
  2. In the SLL experiments (§3, Figs. 6–7), free-generation misalignment collapses similarly under both curricula after fine-tuning; the curriculum advantage appears only under elicitation prompts. That is a legitimate and interesting distinction, but the paper’s framing of “more robust rule-following” and “direct consequences for safety fine-tuning” should more explicitly state that surface behavior is matched and that the advantage is in targeted probing / localization of condition information. Without that qualification, readers may over-read the practical safety claim.
minor comments (5)
  1. Typo in Discussion §4: “verrefusal for tasks that were trained simultaneously” appears to be a missing capital / incomplete sentence.
  2. Fig. 3A is a schematic summary of pathways; ensure the caption and main text consistently distinguish primary vs residual capacity (the App. D.3 note that L2 ATT is sufficient but not necessary for copy-last under imbalance is important and could be cross-referenced in the main figure caption).
  3. Appendix B.4 says “We train 30 seeds per condition” while the main text reports 100 seeds for the primary ICL comparison; reconcile seed counts across text and figures.
  4. Clarify early that “aligned/misaligned” in the SLL task is an operational label for rule satisfaction/violation within the synthetic grammar, not a claim about real-world AI alignment (the parenthetical in §3 helps; a single sentence in the abstract would further reduce ambiguity).
  5. Minor notation: k=5 tokens and n=4 context pairs are introduced clearly in §2, but the total sequence length (38) and vocabulary size (53) appear only in the appendix; a brief main-text mention would aid reproducibility.

Circularity Check

0 steps flagged

No circularity: empirical curriculum comparison with causal circuit interventions; no fitted parameters renamed as predictions and no load-bearing self-citation chain.

full rationale

The paper is an empirical comparative study of balanced vs. imbalanced pretraining curricula on two synthetic multi-task settings (copy-first/copy-last ICL and a rule-based synthetic language). Curriculum schedules (e.g., linear interpolation of task mixture from 90:10 to 50:50, or fixed 50:50), architecture sizes, and optimizers are experimental choices, not parameters fitted to the target outcomes and then re-presented as predictions. Generalization rates, overrefusal rates, and rule-adherence metrics are measured on held-out data. Circuit conclusions rest on causal interventions (mean ablations of L2 ATT/FFN, direct logit attribution, denoising activation patching of L2 Q/K/V/FFN with recovery metric (Acc_patched - Acc_corrupt)/(Acc_clean - Acc_corrupt), residual-stream cosine similarities, and SLL condition-token attention ablations/patching). These interventions do not reduce by construction to the curriculum definitions. Related-work citations (ICL data properties, induction heads, overrefusal, curriculum learning, disentanglement) provide context; none is a uniqueness theorem or ansatz that forces the central claim. The Discussion explicitly scopes external validity to larger models as an open question rather than smuggling it in as a derivation. Therefore the derivation chain is self-contained experimental evidence with score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central comparative claims rest on standard transformer inductive biases, the validity of common mechanistic-interpretability interventions as causal probes, and the experimental design choices that define the two curricula and the two synthetic tasks. No new physical entities or free parameters are fitted to force the main result; the free parameters listed are ordinary experimental knobs whose values are swept or fixed a priori.

free parameters (3)
  • imbalanced curriculum start ratio and anneal schedule = 90:10 → 50:50 (or reverse)
    90:10 (or 10:90) held for 10 epochs then linear fade to 50:50 by epoch 90; chosen by hand and shown to be robust under constant-imbalance and reverse variants, but still a design choice that defines the independent variable.
  • model width / depth / heads = 128 / 2–4 / 4
    d_model=128, 2 or 4 layers, 4 heads; selected for interpretability amenability rather than fitted to maximize the claimed effect, and swept in Appendix I.
  • SLL misaligned weight schedule (w_low, w_high, plateau) = 5 % → 80 % (avg 40 %)
    Imbalanced SLL ramps misaligned fraction from 5 % to 80 % so average exposure matches balanced 40 %; the endpoints are design choices matched for total exposure.
axioms (3)
  • domain assumption Mean ablation, direct logit attribution and denoising activation patching identify necessary/sufficient components of the residual-stream computation for the studied behaviors.
    Standard in the mechanistic-interpretability literature the paper cites (Elhage, Heimersheim & Nanda, Wang et al.); the paper applies them without re-deriving their validity.
  • domain assumption A 2–4 layer causal transformer with learned or rotary positional embeddings is a sufficient model class in which curriculum-induced circuit differences can be observed and causally tested.
    Architecture choice justified by prior small-model interpretability work; results replicated for 4 layers and multiple widths.
  • ad hoc to paper In the synthetic language task, rule satisfaction vs violation is a valid operationalization of 'aligned' vs 'misaligned' behavior for studying selective fine-tuning.
    Authors explicitly scope the terms to the SLL task and do not claim broader AI-alignment equivalence; the mapping is a modeling choice of the paper.

pith-pipeline@v1.1.0-grok45 · 36343 in / 3195 out tokens · 31123 ms · 2026-07-11T12:35:26.242882+00:00 · methodology

0 comments
read the original abstract

Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selectivity of fine-tuning is unclear. This is important for AI safety, where fine-tuning is used to selectively suppress misaligned behaviors. Here, we compare curricula that pretrain tasks in a balanced (sampled uniformly) or an imbalanced (one task early, the other late) fashion. We show that imbalanced learning of two conflicting copy tasks promotes in-context learning and improves the selectivity of refusal fine-tuning. Ablations and activation patching show that this occurs because imbalanced pretraining encourages tasks to be disentangled in separable neural circuits, whereas balanced training routes both tasks through a common pathway. We extend these findings to a synthetic language learning task involving rule-consistent and rule-violating data, where imbalanced curricula similarly lead to more localized, less entangled rule representations, resulting in more robust rule-following behavior. Together, these results suggest that imbalanced pretraining curricula may be an important tool for promoting disentangled representations, with direct consequences for the precision and reliability of safety fine-tuning.

Figures

Figures reproduced from arXiv: 2607.04846 by Christopher Summerfield, Fazl Barez, Jirko Rubruck, Kai J. Sandbrink, Mia H. Whitefield, Sebastian A. Bruijns.

Figure 1
Figure 1. Figure 1: Task and training dynamics under imbalanced versus balanced. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Refusal fine-tuning produces less overrefusal under imbalanced training. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Imbalanced training produces task-specialized sub-circuits, balanced training pro [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Activation patching reveals curriculum-dependent circuitry for pretrained and fine [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Synthetic language learning (SLL) task for studying rule-based alignment under [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Targeted prompts reveal robustness of rule adherence. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: First layer attention is critical for rule adherence, especially under imbalanced [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Difference in learning time in epochs between the two sub-tasks, time of copy-first minus time of copy-last. For each seed, we computed the first epoch at which copy-first and copy-last each exceeded 90% test accuracy, retaining only seeds that reached threshold on both tasks. Bars show the mean signed difference, with standard-error error bars and individual seeds overlaid. C.2 Development of model policy… view at source ↗
Figure 9
Figure 9. Figure 9: Probability distribution of the model over different groups of outputs (copy-first [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Task identity is linearly represented only for imbalanced models. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Residual stream task directions for the two tasks is closer to orthogonal under [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Training dynamics for the imbalanced copy-first-early curriculum. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Refusal training introduces only minimal interference in the inverse imbalanced [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Accuracy on the kept task is maintained in imbalanced models over the course [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Training dynamics for constant imbalanced datasets. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Refusal training introduces only minimal interference in the constant imbalanced [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Training curricula in the infinite data regime. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: DLA analysis on refusal fine-tuned models reveals that refusal is primarily imple [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Additional curricula for SSL task. (A) Top row: To properly control our curriculum manipulation, we train models under additional regimes, beyond the balanced and imbalanced one presented in the paper. Critically, the imbalanced reverse curriculum starts with a large proportion of misaligned data, which decreases over time, flipping the order of the imbalanced curriculum. Bottom: Loss on the full training… view at source ↗
Figure 20
Figure 20. Figure 20: Misalignment and grammaticality of free generations and elicitations. [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Alignment on elicitation prompts is more responsive to fine-tuning after imbalanced [PITH_FULL_IMAGE:figures/full_fig_p033_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Development of condition token attention during pretraining. (A) Condition-token [PITH_FULL_IMAGE:figures/full_fig_p034_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Attention ablation and activation patching on condition-tokens does not affect [PITH_FULL_IMAGE:figures/full_fig_p035_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: First layer QKV attention weights stabilize earlier than later layers [PITH_FULL_IMAGE:figures/full_fig_p036_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Fraction of seeds that reach 90% test accuracy for different hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p037_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Speed of learning for different hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p038_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Specialization of the different components, as revealed by ablations, for different [PITH_FULL_IMAGE:figures/full_fig_p039_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Overrefusal during refusal fine-tuning. (A) Maximum (transient) decrease in accuracy on the task that is not being fine-tuned during refusal fine-tuning, for both copy-last sequences evaluated during refusal fine-tuning of copy-first sequences (blue), and copy-first sequences evaluated during refusal fine-tuning of copy-last sequences (red). Error bars represent the standard error. (B) Same as (A), but fo… view at source ↗
Figure 29
Figure 29. Figure 29: Fraction of seeds that reach 90% test accuracy for different hyperparameters for [PITH_FULL_IMAGE:figures/full_fig_p040_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Speed of learning for different hyperparameters for networks with 4 layers. [PITH_FULL_IMAGE:figures/full_fig_p041_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Specialization of the different components, as revealed by ablations, for different [PITH_FULL_IMAGE:figures/full_fig_p042_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Overrefusal during refusal fine-tuning. (A) Maximum (transient) decrease in accuracy on the task that is not being fine-tuned during refusal fine-tuning, for both copy-last sequences evaluated during refusal fine-tuning of copy-first sequences (blue) , and copy-first sequences evaluated during refusal fine-tuning of copy-last sequences (red). Error bars represent the standard error. (B) Same as (A), but f… view at source ↗
Figure 33
Figure 33. Figure 33: Overrefusal vs. specialization across all seeds and settings. [PITH_FULL_IMAGE:figures/full_fig_p043_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Rule adherence under elicitation across a range of hyperparameter settings. [PITH_FULL_IMAGE:figures/full_fig_p044_34.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 9 canonical work pages

  1. [1]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y . Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction, July 2024. URLhttp://arxiv.org/abs/2309.14316. arXiv:2309.14316 [cs]

    Pith/arXiv arXiv 2024

  2. [2]

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan....

  3. [3]

    arXiv:2204.05862 [cs]

    URLhttp://arxiv.org/abs/2204.05862. arXiv:2204.05862 [cs]

    Pith/arXiv arXiv

  4. [4]

    Belrose, Q

    N. Belrose, Q. Pope, L. Quirke, A. Mallen, and X. Fern. Neural networks learn statistics of increasing complexity. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofICML’24, pages 3382–3409, Vienna, Austria, July 2024. JMLR.org

    2024

  5. [5]

    Bengio, J

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 41–48, New York, NY , USA, June 2009. Association for Computing Machinery. ISBN 978-1-60558-516-1. doi: 10.1145/1553374.1553380. URL https://dl.acm.org/doi/10.1145/1553374.1553380

    doi:10.1145/1553374.1553380 2009

  6. [6]

    Bengio, A

    Y . Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives,

  7. [7]

    URLhttps://arxiv.org/abs/1206.5538

    Pith/arXiv arXiv

  8. [8]

    Betley, D

    J. Betley, D. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.Nature, 649 (8097):584–589, Jan. 2026. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-025-09937-5. URLhttp://arxiv.org/abs/2502.17424. arXiv:2502.17424 [cs]

    doi:10.1038/s41586-025-09937-5 2026

  9. [9]

    S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClel- land, and F. Hill. Data Distributional Properties Drive Emergent In-Context Learning in Transformers.Advances in Neural Information Processing Systems, 35:18878–18891, Dec. 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 77c6ccacfd9962e2307fc64680fc5ace-Abs...

    2022

  10. [10]

    J. Chen, X. Pan, D. Yu, K. Song, X. Wang, D. Yu, and J. Chen. Skills-in-Context: Unlock- ing Compositionality in Large Language Models. In Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13838–13890, Miami, Florida, USA, Nov. 2024. Association for Computational Linguis- tics....

    doi:10.18653/v1/2024.findings-emnlp.812 2024

  11. [11]

    M. F. Chen, N. Roberts, K. Bhatia, J. Wang, C. Zhang, F. Sala, and C. Ré. Skill-it! A Data- Driven Skills Framework for Understanding and Training Language Models, July 2023. URL http://arxiv.org/abs/2307.14430. arXiv:2307.14430 [cs]

    Pith/arXiv arXiv 2023

  12. [12]

    Conmy, A

    A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability.Ad- vances in Neural Information Processing Systems, 36:16318–16352, Dec. 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html

    2023

  13. [13]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, et al. A mathematical framework for transformer circuits.Transformer Circuits Thread, 1(1):12, 2021

    2021

  14. [14]

    Flesch, J

    T. Flesch, J. Balaguer, R. Dekker, H. Nili, and C. Summerfield. Comparing continual task learning in minds and machines.Proceedings of the National Academy of Sciences, 115 (44), Oct. 2018. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1800755115. URL https: //pnas.org/doi/full/10.1073/pnas.1800755115. 11

    work page doi:10.1073/pnas.1800755115 2018

  15. [15]

    Flesch, K

    T. Flesch, K. Juechems, T. Dumbalska, A. Saxe, and C. Summerfield. Orthogonal representations for robust context-dependent task performance in brains and neural networks.Neuron, 110 (7):1258–1270.e11, Apr. 2022. ISSN 08966273. doi: 10.1016/j.neuron.2022.01.005. URL https://linkinghub.elsevier.com/retrieve/pii/S0896627322000058

    work page doi:10.1016/j.neuron.2022.01.005 2022

  16. [16]

    S. Garg, D. Tsipras, P. Liang, and G. Valiant. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, Aug. 2023. URL http://arxiv.org/abs/2208.01066. arXiv:2208.01066 [cs]

    Pith/arXiv arXiv 2023

  17. [17]

    M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer Feed-Forward Layers Are Key-Value Memories, Sept. 2021. URL http://arxiv.org/abs/2012.14913. arXiv:2012.14913 [cs]

    Pith/arXiv arXiv 2021

  18. [18]

    Gibson, W

    C. Gibson, W. Cui, and G. Reddy. Distinct mechanisms underlying in-context learning in transformers, Apr. 2026. URL http://arxiv.org/abs/2604.12151. arXiv:2604.12151 [cs] version: 1

    Pith/arXiv arXiv 2026

  19. [19]

    Hanna, O

    M. Hanna, O. Liu, and A. Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, Nov. 2023. URL http://arxiv.org/ abs/2305.00586. arXiv:2305.00586 [cs]

    Pith/arXiv arXiv 2023

  20. [20]

    Heimersheim and N

    S. Heimersheim and N. Nanda. How to use and interpret activation patching, Apr. 2024. URL http://arxiv.org/abs/2404.15255. arXiv:2404.15255 [cs]

    Pith/arXiv arXiv 2024

  21. [21]

    Higgins, L

    I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-V AE: Learning Basic Visual Concepts with a Constrained Variational Framework. Feb. 2017. URLhttps://openreview.net/forum?id=Sy2fzU9gl

    2017

  22. [22]

    Kalimeris, G

    D. Kalimeris, G. Kaplun, P. Nakkiran, B. Edelman, T. Yang, B. Barak, and H. Zhang. SGD on Neural Networks Learns Functions of Increasing Complexity. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  23. [23]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114. URLhttps://w...

    doi:10.1073/pnas.1611835114 2017

  24. [24]

    K. A. Krueger and P. Dayan. Flexible shaping: How learning in small steps helps.Cognition, 110(3):380–394, Mar. 2009. ISSN 00100277. doi: 10.1016/j.cognition.2008.11.014. URL https://linkinghub.elsevier.com/retrieve/pii/S0010027708002850

    work page doi:10.1016/j.cognition.2008.11.014 2009

  25. [25]

    B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people.Behavioral and Brain Sciences, 40:e253, 2017. doi: 10.1017/ S0140525X16001837

    2017

  26. [26]

    J. H. Lee, S. S. Mannelli, and A. Saxe. Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning, June 2024. URL http://arxiv.org/abs/2402. 18361. arXiv:2402.18361 [q-bio]

    Pith/arXiv arXiv 2024

  27. [27]

    J. H. Lee, A. K. Lampinen, A. K. Singh, and A. M. Saxe. Distinct Computations Emerge From Compositional Curricula in In-Context Learning, June 2025. URL http://arxiv.org/abs/ 2506.13253. arXiv:2506.13253 [cs]

    Pith/arXiv arXiv 2025

  28. [28]

    Lieberum, M

    T. Lieberum, M. Rahtz, J. Kramár, N. Nanda, G. Irving, R. Shah, and V . Mikulik. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla, July

  29. [29]

    URLhttps://arxiv.org/abs/2307.09458v3

    Pith/arXiv arXiv

  30. [30]

    Lin and K

    Z. Lin and K. Lee. Dual Operating Modes of In-Context Learning, Aug. 2024. URL http: //arxiv.org/abs/2402.18819. arXiv:2402.18819 [cs]

    Pith/arXiv arXiv 2024

  31. [31]

    E. Liu, K. Sun, M. Li, I. Lee, L. Tjuatja, J.-t. Huang, and G. Neubig. What do Language Models Learn and When? The Implicit Curriculum Hypothesis, Apr. 2026. URL http: //arxiv.org/abs/2604.08510. arXiv:2604.08510 [cs] version: 1. 12

    Pith/arXiv arXiv 2026

  32. [32]

    Locatello, S

    F. Locatello, S. Bauer, M. Lu ˇci´c, G. Rätsch, S. Gelly, B. Schölkopf, and O. F. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. InInternational Conference on Machine Learning, 2019. URL http://proceedings.mlr. press/v97/locatello19a.html. Best Paper Award

    2019

  33. [33]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization, Jan. 2019. URL http://arxiv.org/abs/1711.05101. arXiv:1711.05101 [cs]

    Pith/arXiv arXiv 2019

  34. [34]

    J. M. Mandler and L. McDonough. Concept Formation in Infaricy.Cognitive Development, (8): 291–318, 1993

    1993

  35. [35]

    S. S. Mannelli, Y . Ivashinka, A. Saxe, and L. Saglietti. Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks, June 2024. URL http://arxiv.org/abs/2406.01589. arXiv:2406.01589 [cond-mat, q-bio, stat]

    Pith/arXiv arXiv 2024

  36. [36]

    Mante, D

    V . Mante, D. Sussillo, K. V . Shenoy, and W. T. Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex.Nature, 503(7474):78–84, Nov. 2013. ISSN 1476-4687. doi: 10.1038/nature12742

    doi:10.1038/nature12742 2013

  37. [37]

    Mathy and J

    F. Mathy and J. Feldman. A rule-based presentation order facilitates category learning. Psychonomic Bulletin & Review, 16(6):1050–1057, Dec. 2009. ISSN 1531-5320. doi: 10.3758/PBR.16.6.1050. URLhttps://doi.org/10.3758/PBR.16.6.1050

    work page doi:10.3758/pbr.16.6.1050 2009

  38. [38]

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov. Locating and Editing Factual Associations in GPT, Jan. 2023. URLhttp://arxiv.org/abs/2202.05262. arXiv:2202.05262 [cs]

    Pith/arXiv arXiv 2023

  39. [39]

    Merullo, C

    J. Merullo, C. Eickhoff, and E. Pavlick. Circuit Component Reuse Across Tasks in Transformer Language Models, May 2024. URL http://arxiv.org/abs/2310.08744. arXiv:2310.08744 [cs] version: 3

    Pith/arXiv arXiv 2024

  40. [40]

    Michaud, Z

    E. Michaud, Z. Liu, U. Girit, and M. Tegmark. The Quantization Model of Neu- ral Scaling.Advances in Neural Information Processing Systems, 36:28699–28722, Dec. 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 5b6346a05a537d4cdb2f50323452a9fe-Abstract-Conference.html

    2023

  41. [41]

    S. M. Noh, V . X. Yan, R. A. Bjork, and W. T. Maddox. Optimal sequencing during category learning: Testing a dual-learning systems perspective.Cognition, 155:23–29, Oct. 2016. ISSN 0010-0277. doi: 10.1016/j.cognition.2016.06.007. URL https://www.sciencedirect. com/science/article/pii/S0010027716301548

    work page doi:10.1016/j.cognition.2016.06.007 2016

  42. [42]

    Olsson, N

    C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. In-context Learning and Induction Heads, Sept. 2022. UR...

    Pith/arXiv arXiv 2022

  43. [43]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, Mar. 2022. URL http://arxiv.org/abs/2203.02155. arXiv:2203....

    Pith/arXiv arXiv 2022

  44. [44]

    J. Pan, T. Gao, H. Chen, and D. Chen. What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning, May 2023. URL http://arxiv.org/ abs/2305.09731. arXiv:2305.09731 [cs]

    Pith/arXiv arXiv 2023

  45. [45]

    L. Pan, Y . Tong, X. Zhang, X. Zhang, J. Zhou, and Z. Chu. Understanding and Mitigat- ing Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21057–21075, Suzhou, ...

    work page doi:10.18653/v1/2025.emnlp-main.1065 2025

  46. [46]

    Pastrana

    R. Pastrana. Disentangling Variational Autoencoders, Nov. 2022. URL http://arxiv.org/ abs/2211.07700. arXiv:2211.07700 [cs]

    Pith/arXiv arXiv 2022

  47. [47]

    Pesnot Lerousseau and C

    J. Pesnot Lerousseau and C. Summerfield. Shared sensitivity to data distribution during learning in humans and transformer networks.Nature Human Behaviour, 10(3):601–614, Mar. 2026. ISSN 2397-3374. doi: 10.1038/s41562-025-02359-3. URL https://www.nature.com/ articles/s41562-025-02359-3

    work page doi:10.1038/s41562-025-02359-3 2026

  48. [48]

    X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson. Safety Alignment Should Be Made More Than Just a Few Tokens Deep, June 2024. URL https: //arxiv.org/abs/2406.05946v1

    Pith/arXiv arXiv 2024

  49. [49]

    Quirke and F

    P. Quirke and F. Barez. Understanding Addition in Transformers, Apr. 2024. URL http: //arxiv.org/abs/2310.13121. arXiv:2310.13121 [cs]

    Pith/arXiv arXiv 2024

  50. [50]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, July 2024. URL http: //arxiv.org/abs/2305.18290. arXiv:2305.18290 [cs]

    Pith/arXiv arXiv 2024

  51. [51]

    Rahaman, A

    N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y . Bengio, and A. Courville. On the Spectral Bias of Neural Networks. InProceedings of the 36th International Conference on Machine Learning, pages 5301–5310. PMLR, May 2019

    2019

  52. [52]

    G. Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. InThe Twelfth International Conference on Learning Representations, Oct

  53. [53]

    URLhttps://openreview.net/forum?id=aN4Jf6Cx69

  54. [54]

    Rubruck, J

    J. Rubruck, J. P. Bauer, A. M. Saxe, and C. Summerfield. Early learning of the optimal constant solution in neural networks and humans. InProceedings of the 8th Annual Conference on Cognitive Computational Neuroscience, July 2025. URL https://openreview.net/forum? id=6Xyu486HRh

    2025

  55. [55]

    Russin, E

    J. Russin, E. Pavlick, and M. J. Frank. Parallel trade-offs in human cognition and neural networks: The dynamic interplay between in-context and in-weight learning.Proceedings of the National Academy of Sciences, 122(35):e2510270122, 2025. doi: 10.1073/pnas.2510270122. URLhttps://www.pnas.org/doi/abs/10.1073/pnas.2510270122

    work page doi:10.1073/pnas.2510270122 2025

  56. [56]

    A. M. Saxe, J. L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks.Proceedings of the National Academy of Sciences, 116(23):11537– 11546, June 2019. doi: 10.1073/pnas.1820226116

    doi:10.1073/pnas.1820226116 2019

  57. [57]

    S. Si, X. Wang, G. Zhai, N. Navab, and B. Plank. Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior, Mar. 2025. URLhttp://arxiv.org/ abs/2503.17882. arXiv:2503.17882 [cs]

    Pith/arXiv arXiv 2025

  58. [58]

    Singh, S

    A. Singh, S. Chan, T. Moskovitz, E. Grant, A. Saxe, and F. Hill. The Transient Nature of Emer- gent In-Context Learning in Transformers.Advances in Neural Information Processing Systems, 36:27801–27819, Dec. 2023. URL https://proceedings.neurips.cc/paper_files/ paper/2023/hash/58692a1701314e09cbd7a5f5f3871cc9-Abstract-Conference. html

    2023

  59. [59]

    A. K. Singh, T. Moskovitz, F. Hill, S. C. Y . Chan, and A. M. Saxe. What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation, Apr

  60. [60]

    arXiv:2404.07129 [cs]

    URLhttp://arxiv.org/abs/2404.07129. arXiv:2404.07129 [cs]

    Pith/arXiv arXiv

  61. [61]

    A. K. Singh, T. Moskovitz, S. Dragutinovic, F. Hill, S. C. Y . Chan, and A. M. Saxe. Strategy Coopetition Explains the Emergence and Transience of In-Context Learning, Mar. 2025. URL http://arxiv.org/abs/2503.05631. arXiv:2503.05631 [cs]

    Pith/arXiv arXiv 2025

  62. [62]

    B. F. Skinner. Reinforcement today.American Psychologist, 13(3):94–99, 1958. doi: 10.1037/ h0049039. URLhttps://doi.org/10.1037/h0049039

    work page doi:10.1037/h0049039 1958

  63. [63]

    Soviany, R

    P. Soviany, R. T. Ionescu, P. Rota, and N. Sebe. Curriculum learning: A survey.CoRR, abs/2101.10382, 2021. URLhttps://arxiv.org/abs/2101.10382. 14

    Pith/arXiv arXiv 2021

  64. [64]

    Srivastava, G

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014. ISSN 1533-7928. URL http://jmlr.org/papers/ v15/srivastava14a.html

    1929

  65. [65]

    Steenbrugge, S

    X. Steenbrugge, S. Leroux, T. Verbelen, and B. Dhoedt. Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations, Nov. 2018. URL http:// arxiv.org/abs/1811.04784. arXiv:1811.04784 [cs]

    Pith/arXiv arXiv 2018

  66. [66]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V . Kerkez, M. Khabsa, I. Kloumann, A. Koren...

    Pith/arXiv arXiv 2023

  67. [67]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. InAdvances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

    2017

  68. [68]

    V olk, C

    C. V olk, C. C. Pack, and S. Bakhtiari. The curriculum effect in visual learning: the role of readout dimensionality, Sept. 2025. URL https://www.biorxiv.org/content/10.1101/ 2025.09.19.677409v1. ISSN: 2692-8205 Pages: 2025.09.19.677409 Section: New Results

    2025

  69. [69]

    K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. Sept. 2022. URL https://openreview.net/forum?id=NpsVSN6o4ul

    2022

  70. [70]

    X. Wang, C. Hu, P. Röttger, and B. Plank. Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation, Oct. 2024. URL https://arxiv. org/abs/2410.03415v2

    Pith/arXiv arXiv 2024

  71. [71]

    Warstadt, A

    A. Warstadt, A. Mueller, L. Choshen, E. Wilcox, C. Zhuang, J. Ciro, R. Mosquera, B. Paranjabe, A. Williams, T. Linzen, et al. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. InProceedings of the babylm challenge at the 27th conference on computational natural language learning, pages 1–34, 2023

    2023

  72. [72]

    X. Wu, E. Dyer, and B. Neyshabur. When Do Curricula Work?, Feb. 2021. URL http: //arxiv.org/abs/2012.03107. arXiv:2012.03107

    Pith/arXiv arXiv 2021

  73. [73]

    Z. Xue, Z. Qi, G. Liu, B. Chen, and R. Pedarsani. Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment, Mar. 2026. URL http://arxiv.org/abs/ 2603.11388. arXiv:2603.11388 [cs]

    arXiv 2026

  74. [74]

    Zhang, A

    Y . Zhang, A. M. Saxe, and P. E. Latham. Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures. Oct. 2025. URL https://openreview.net/ forum?id=Vit5M0G5Gb

    2025

  75. [75]

    Zhang, A

    Y . Zhang, A. Mohamed, H. Abdine, G. Shang, and M. Vazirgiannis. Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning, Jan. 2026. URL http:// arxiv.org/abs/2506.11300. arXiv:2506.11300 [cs]

    arXiv 2026

  76. [76]

    overrefusal directions

    N. Zucchet, J. Bornschein, S. Chan, A. Lampinen, R. Pascanu, and S. De. How do language models learn facts? Dynamics, curricula and hallucinations, July 2025. URL http://arxiv. org/abs/2503.21676. arXiv:2503.21676 [cs] version: 2. 15 Appendix Roadmap The appendix is organized as follows: Section Contents Appendix A Extended related work: data-distribution...

    Pith/arXiv arXiv 2025

  77. [77]

    or through the targeted prompting with specific safety reflections [53]. While previous work has focused on representational factors driving overrefusal, our work seeks to explain how learning dynamics and curricula can give rise to entangled task representations that can give rise to unintended overgeneralization behaviors. Curriculum learningHuman and a...

  78. [78]

    clean" sequence drawn from one task, we construct a

    perhaps the partial acquisition of memorizing, in-weights solutions facilitates in-context learning in a more cooperative manner [56]. 0 20 40 60 80 100 120 0 20 40 60 80 100Fraction (%) A Copy First Copy Last 0 20 40 60 80 100 120 0 20 40 60 80 100T est Acc. (%) 0 20 40 60 80 100 120 0.0 0.5 1.0 1.5 2.0 2.5Train Loss 0 20 40 60 80 100 120 Epoch 0 20 40 6...

  79. [79]

    We increased the number of values (Alice, Bob, ...) for each feature (NAME, JOB, ...) from 4 to 15, to allow for more rule variety

  80. [80]

    We vary the number of rules (from 5 to 40), and randomly sample that number of rules for each run independently

Showing first 80 references.