Pith. sign in

REVIEW 2 major objections 1 minor 60 cited by

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Induction heads implement the core copying algorithm behind in-context learning in transformers.

2026-05-11 03:44 UTC pith:FJNRNXM5

load-bearing objection Induction heads drive in-context learning with solid causal evidence in small attention-only models but only correlational support in larger ones with MLPs. the 2 major comments →

arxiv 2209.11895 v1 pith:FJNRNXM5 submitted 2022-09-24 cs.LG

In-context Learning and Induction Heads

Catherine Olsson , Nelson Elhage , Neel Nanda , Nicholas Joseph , Nova DasSarma , Tom Henighan , Ben Mann , Amanda Askell
show 18 more authors
Yuntao Bai Anna Chen Tom Conerly Dawn Drain Deep Ganguli Zac Hatfield-Dodds Danny Hernandez Scott Johnston Andy Jones Jackson Kernion Liane Lovitt Kamal Ndousse Dario Amodei Tom Brown Jack Clark Jared Kaplan Sam McCandlish Chris Olah
This is my paper
classification cs.LG
keywords induction headsin-context learningtransformer modelsattention mechanismstraining dynamicssequence copying
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that induction heads are the main mechanism driving in-context learning, the steady drop in loss on later tokens within a sequence. These heads detect a repeated token and copy the token that followed it last time, completing patterns like [A][B]...[A] to [B]. The authors show that these heads appear at the exact training step where a sharp bump in in-context performance occurs. They give causal evidence in small attention-only models by editing the heads and correlational evidence in larger models. If the claim holds, it would mean that a simple, local copying rule explains most of the rapid adaptation transformers show during inference.

Core claim

Induction heads are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. The authors present six lines of evidence that these heads constitute the mechanism for the majority of all in-context learning in large transformer models, developing precisely when a sudden sharp increase in in-context learning ability occurs during training.

What carries the argument

Induction heads, attention heads that detect a prior token match and copy the subsequent token from that earlier occurrence.

Load-bearing premise

The formation of induction heads directly causes the observed jump in in-context learning rather than both changes arising together from some other training dynamic.

What would settle it

Train a transformer in which induction heads never form yet a sharp increase in in-context learning still appears at the same training step.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Induction heads emerge at the same moment training loss shows a sharp improvement on later tokens.
  • In small attention-only models, directly ablating induction heads reduces in-context learning performance.
  • The timing correlation between head formation and performance gains holds across model sizes.
  • The mechanism appears general enough to explain in-context learning in transformers of any scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If induction heads are the primary driver, then interventions that speed their formation could shorten the training needed for strong few-shot behavior.
  • The copying rule might also explain why transformers handle many different in-context tasks without task-specific fine-tuning.
  • Checking whether non-attention architectures develop analogous copying circuits would test how specific this mechanism is to transformers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper hypothesizes that 'induction heads' (attention heads implementing a simple [A][B]...[A] -> [B] completion algorithm) are the primary mechanistic source of in-context learning in transformers, defined as the decrease in loss at increasing token indices. It reports that these heads emerge at the same training point as a sharp loss bump signaling increased in-context ability, presenting six lines of evidence: strong causal interventions (ablations/patching) for small attention-only models and correlational/timing-based evidence for larger models containing MLPs.

Significance. If the causal link holds, the work would supply a concrete mechanistic account of in-context learning, a core capability of large language models. The strong, reproducible causal interventions in small attention-only models constitute a clear strength, as do the multiple complementary observational measures (timing correlations, head activation patterns) that could guide future targeted experiments. The paper thereby advances mechanistic interpretability by linking a specific circuit to a broad behavioral phenomenon.

major comments (2)
  1. Abstract: The claim that induction heads 'might constitute the mechanism for the majority of all in-context learning' in large transformer models rests on correlational evidence only; the text states that the six lines of evidence for models with MLPs are 'preliminary and indirect' and 'correlational,' with no ablation, patching, or causal intervention results reported to show that disabling induction heads specifically impairs the observed in-context loss reduction.
  2. Description of the six lines of evidence (larger models): These lines rely on coincidence of induction-head emergence with the training loss bump and on observational metrics such as head activation timing; they do not include controls that would distinguish whether both phenomena are parallel downstream effects of an earlier training dynamic (e.g., a phase transition in optimization or representation geometry), leaving the causal inference untested for models containing MLPs.
minor comments (1)
  1. Abstract: Quantitative details on the magnitude of the loss bump, the fraction of heads identified as induction heads, and any error controls or statistical tests for the six lines of evidence would improve clarity and allow readers to assess the strength of the correlational results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the value of the causal interventions in small models as well as the potential of the observational measures to guide future work. We agree that the distinction between causal and correlational evidence must be drawn more sharply in the abstract and discussion, and we will revise the manuscript to address both major comments.

read point-by-point responses

  1. Referee: Abstract: The claim that induction heads 'might constitute the mechanism for the majority of all in-context learning' in large transformer models rests on correlational evidence only; the text states that the six lines of evidence for models with MLPs are 'preliminary and indirect' and 'correlational,' with no ablation, patching, or causal intervention results reported to show that disabling induction heads specifically impairs the observed in-context loss reduction.

    Authors: We accept the point. While the body of the paper already describes the evidence for models with MLPs as preliminary, indirect, and correlational, the abstract phrasing risks implying stronger support than exists. We will revise the abstract to state explicitly that the hypothesis for large models rests on correlational evidence from the six lines, without causal interventions such as ablation or patching, and to moderate the language concerning induction heads as the mechanism for the majority of in-context learning. revision: yes

  2. Referee: Description of the six lines of evidence (larger models): These lines rely on coincidence of induction-head emergence with the training loss bump and on observational metrics such as head activation timing; they do not include controls that would distinguish whether both phenomena are parallel downstream effects of an earlier training dynamic (e.g., a phase transition in optimization or representation geometry), leaving the causal inference untested for models containing MLPs.

    Authors: The referee correctly notes that the six lines are observational and lack controls that could rule out alternative accounts in which induction-head emergence and the loss bump are both downstream of an earlier training dynamic. We do not claim to have performed such controls. In revision we will add an explicit limitations paragraph in the discussion that acknowledges this gap, lists possible alternative explanations (including phase transitions in optimization or representation geometry), and clarifies that the lines of evidence are intended to be suggestive and to motivate targeted causal experiments rather than to demonstrate causality. revision: yes

Circularity Check

0 steps flagged

No significant circularity; hypothesis rests on timing correlations and interventions rather than definitional reduction

full rationale

The paper defines induction heads via their observable attention pattern on token sequences and presents empirical evidence (simultaneous emergence with loss bump, six lines of correlational evidence for large models, and causal ablations for small attention-only models) that they contribute to in-context learning. No step reduces a claimed prediction or result to a fitted parameter or self-citation by construction; the central hypothesis is explicitly labeled preliminary and indirect, with the link to decreasing loss at later token indices argued via external observations rather than tautological redefinition. The derivation chain is self-contained against the provided benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The claim rests on the operational definition of induction heads as pattern-completion circuits and on the assumption that their sudden appearance during training is the direct cause of improved in-context performance rather than a correlated side effect.

axioms (2)
  • domain assumption Induction heads implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]
    This is the paper's working definition of the heads whose causal role is being tested.
  • domain assumption A sharp increase in in-context learning ability is visible as a bump in the training loss curve
    The timing alignment between this bump and the emergence of induction heads is treated as a key signature.

pith-pipeline@v0.9.0 · 5519 in / 1325 out tokens · 45996 ms · 2026-05-11T03:44:13.815348+00:00 · methodology

0 comments
read the original abstract

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficiently Representing Algorithms With Chain-of-Thought Transformers

    cs.LG 2026-06 conditional novelty 8.0

    CoT transformers simulate any Word RAM algorithm with poly-logarithmic overhead in three architectures, improving on quadratic TM overhead.

  2. Looped Transformers with Layer Normalization Provably Learn the Power Method

    cs.LG 2026-05 unverdicted novelty 8.0

    Looped linear transformers with LN provably converge via GD to implement the power method on principal component prediction.

  3. Towards Verifiable Transformers: Solver-Checkable Circuit Explanations

    cs.LG 2026-05 unverdicted novelty 8.0

    Presents a solver-verifiable framework for Transformer circuits, with exhaustive checks on small symbolic tasks and surrogate methods for larger models.

  4. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE introduces sparse autoencoders with rank-1 matrix atoms for recurrent state updates, allowing replacement tests that outperform deletion on 92.4% of positions and a formula predicting logit changes with R²=0.98.

  5. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE is the first sparse autoencoder that factors decoder atoms into the native d_k x d_v cache write shape of recurrent models and supplies a closed-form per-token logit shift for atom substitution.

  6. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0

    WriteSAE decomposes recurrent model cache writes into substitutable atoms with a closed-form logit shift, achieving high substitution success and targeted behavioral installs on models like Qwen3.5 and Mamba-2.

  7. Slot Machines: How LLMs Keep Track of Multiple Entities

    cs.CL 2026-04 unverdicted novelty 8.0

    LLM activations encode current and prior entities in orthogonal slots, but models only use the current slot for explicit factual retrieval despite prior-slot information being linearly decodable.

  8. Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification

    cs.LG 2026-04 unverdicted novelty 8.0

    Enforcing feature- and label-permutation equivariance in transformers for in-context classification yields an identifiable emergent update rule driven by mixed feature-label Gram matrices that amplifies class separation.

  9. The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

    cs.LG 2026-04 unverdicted novelty 8.0

    Transformer weight spectra exhibit transient compression waves that propagate layer-wise, persistent non-monotonic depth gradients in power-law exponents, and Q/K-V asymmetry, with the spectral exponent alpha predicti...

  10. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    cs.AI 2024-08 unverdicted novelty 8.0

    The AI Scientist framework enables LLMs to independently conduct the full scientific process from idea generation to paper writing and review, demonstrated across three ML subfields with papers costing under $15 each.

  11. KAN: Kolmogorov-Arnold Networks

    cs.LG 2024-04 conditional novelty 8.0

    KANs with learnable univariate spline activations on edges achieve better accuracy than MLPs with fewer parameters, faster scaling, and direct visualization for scientific discovery.

  12. Localizing Model Behavior with Path Patching

    cs.LG 2023-04 unverdicted novelty 8.0

    Path patching provides a method to express and quantitatively test hypotheses that neural network behaviors are localized to sets of paths.

  13. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

    cs.LG 2022-11 conditional novelty 8.0

    GPT-2 small solves indirect object identification via a circuit of 26 attention heads organized into seven functional classes discovered through causal interventions.

  14. How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

    cs.LG 2026-07 conditional novelty 7.0

    RoPE frequency usage is determined by a data-induced dependency width W, with the optimal frequency scaling as π/W, explaining both learned spectra and the success of position interpolation.

  15. ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

    cs.AI 2026-07 unverdicted novelty 7.0

    RECONTEXT is a recursive evidence replay technique that improves long-context reasoning in LLMs by constructing and replaying a query-conditioned evidence pool before final generation.

  16. Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

    cs.CL 2026-07 unverdicted novelty 7.0

    A 0.6B LM with length-aware attention adjustments performs competitive in-context retrieval at million-token scale on MS MARCO, NQ, and LIMIT benchmarks.

  17. Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

    cs.CL 2026-07 unverdicted novelty 7.0

    LOCOS scores attention heads via OV-circuit output projection onto answer-token unembedding directions and identifies non-literal retrieval heads whose ablation collapses performance on non-literal benchmarks more tha...

  18. SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    SemRF supplies fixed semantic anchors and pseudo-inverse tying to produce stable coordinates for residual dynamics, Voronoi traces, and minimum-action canonical paths that link to parameter efficiency under controlled...

  19. ECHO: Learning Epistemically Adaptive Language Agents with Turn-Level Credit

    cs.MA 2026-06 unverdicted novelty 7.0

    ECHO is a clipped policy-gradient method that uses posterior-sensitive rewards to give turn-level epistemic credit in multi-turn information-seeking tasks, outperforming trajectory-level GRPO on a new Clue Selector Ga...

  20. Symbolic Mechanistic Data Attribution: Tracing Training Influence to Learned Behavioral Policies

    cs.LG 2026-06 unverdicted novelty 7.0

    SMDA fits ridge regression on SAE features to distill symbolic policies then decomposes each SFT example's influence via feature-activation and output-probability deltas, demonstrated on refusal behavior in Llama-3.2-...

  21. Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

    cs.LG 2026-06 unverdicted novelty 7.0

    During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.

  22. DREAM: Dense Retrieval Embeddings via Autoregressive Modeling

    cs.CL 2026-06 unverdicted novelty 7.0

    DREAM enables training of dense retrieval embeddings using autoregressive next-token prediction from LLMs by modulating attention with retriever scores.

  23. Mind the Heads: Topological Representation Alignment for Multimodal LLMs

    cs.CV 2026-06 conditional novelty 7.0

    HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 b...

  24. Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agents

    cs.CR 2026-06 unverdicted novelty 7.0

    Relinking is a new compression-boundary attack on LLM agents where summarization of split benign fragments produces malicious instructions, shown via Relink tool at 86.9% success rate and mitigated by KBRA defense to 0%.

  25. Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

    cs.CV 2026-06 conditional novelty 7.0

    Reroute turns irreversible visual-token pruning into recoverable routing that reuses existing attention scores, improving grounding performance under aggressive reduction on LLaVA-1.5 and Qwen while preserving TFLOPs ...

  26. Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

    stat.ML 2026-06 unverdicted novelty 7.0

    Bayesian reduction of attention posterior on copy task predicts first-order phase transition for softmax attention and second-order followed by crossover for linear attention.

  27. STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations

    cs.LG 2026-06 unverdicted novelty 7.0

    STRIDE formulates TDA as sparse recovery using steering operators that mimic subset training effects in activation space, claiming SOTA LLM pre-training attribution at 13x prior speed.

  28. What Makes Chain-of-Thought Work at Probe Time? Local Co-occurrence Rather Than Global Derivation

    cs.AI 2026-05 unverdicted novelty 7.0

    CoT probe-time gains arise primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation.

  29. Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift

    cs.LG 2026-05 unverdicted novelty 7.0

    GAE reduces the faithfulness gap in dictionary-based explainers under distribution shift by geometrically realigning the ID dictionary to the OOD-active subspace, with a quadratic excess-loss bound.

  30. Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space

    cs.LG 2026-05 unverdicted novelty 7.0

    Pretraining and alignment induce asymmetric geometric traces in transformer weights because alignment updates concentrate in read pathways due to activation covariance while write pathways inherit less structure from ...

  31. Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers

    cs.CV 2026-05 unverdicted novelty 7.0

    Text embeddings in MM-DiTs contain a detectable omission signal for missing concepts, and amplifying it via OSI reduces concept omission in generated images on FLUX.1-Dev and SD3.5-Medium.

  32. Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

    cs.CL 2026-05 conditional novelty 7.0

    Many-shot CoT-ICL functions as test-time learning when demonstrations are ordered for smooth conceptual progression rather than similarity, enabling a new selection method that improves reasoning performance.

  33. Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers

    cs.AI 2026-05 conditional novelty 7.0

    The Divergent Remote Association Test (DRAT) is the first creativity test that significantly predicts LLMs' scientific ideation ability, unlike prior tests such as DAT or RAT.

  34. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 7.0

    WriteSAE factors sparse autoencoder decoder atoms to the native d_k x d_v cache write shape in recurrent models, provides a closed-form logit shift, and demonstrates high success in atom substitution and behavioral ed...

  35. Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

    cs.CL 2026-05 unverdicted novelty 7.0

    Temporarily reducing the learning rate on upper-layer query and key projections during early GPT pretraining prevents premature attention specialization and improves model performance.

  36. Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

    cs.LG 2026-05 unverdicted novelty 7.0

    Self-attention acts as a covariance readout that unifies in-context learning via population gradient descent and repetitive generation via asymptotic Markov behavior.

  37. From Mechanistic to Compositional Interpretability

    cs.LG 2026-05 unverdicted novelty 7.0

    The paper introduces compositional interpretability as a category-theoretic framework that casts mechanistic explanations as commuting syntactic-semantic mappings optimized under faithfulness and complexity constraint...

  38. From Mechanistic to Compositional Interpretability

    cs.LG 2026-05 unverdicted novelty 7.0

    Compositional interpretability defines explanations as commuting syntactic-semantic mapping pairs grounded in compositionality and minimum description length, with compressive refinement and a parsimony theorem guaran...

  39. Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions

    cs.CL 2026-05 unverdicted novelty 7.0

    Performance collapse in layer-pruned LLMs stems from disrupting the Silent Phase of decision-making, which blocks the transition to correct predictions, while the later Decisive Phase is robust to pruning.

  40. Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations

    cs.CL 2026-05 unverdicted novelty 7.0

    LLM surrogate beliefs under sparse observations depend on prompts and query protocols, with structural prompts as priors, pointwise vs joint querying producing different beliefs, and sequential evidence causing non-mo...

  41. Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

    cs.LG 2026-05 unverdicted novelty 7.0

    In a controlled synthetic setting, transformers implement in-distribution task inference via convex combinations of task vectors and out-of-distribution inference via nearly orthogonal extrapolative representations.

  42. Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection

    cs.CR 2026-05 unverdicted novelty 7.0

    Causal tracing reveals a persistent Refusal Trajectory in LLM hidden states; SALO detector using sparse activations from a layer window improves jailbreak detection across Qwen, Llama, and Mistral models.

  43. When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    LLM first-answer accuracy on procedural arithmetic drops from 61% on 5-step tasks to 20% on 95-step tasks, with frequent failures including skipped steps, premature answers, and hallucinated operations.

  44. How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

    cs.LG 2026-04 unverdicted novelty 7.0

    A fitted iso-depth scaling law measures that one recurrence in looped transformers is worth r^0.46 unique blocks in validation loss.

  45. Cell-Based Representation of Relational Binding in Language Models

    cs.CL 2026-04 unverdicted novelty 7.0

    Large language models encode relational bindings via a cell-based representation: a low-dimensional linear subspace in which each cell corresponds to an entity-relation index pair and attributes are retrieved from the...

  46. Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

    cs.CR 2026-04 unverdicted novelty 7.0

    A Merkle-committed SAE feature-trace protocol detects model substitutions in hosted LLMs at a stable threshold where parallel-probe baselines fail, including against adaptive LoRA attackers.

  47. HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads

    cs.IR 2026-04 unverdicted novelty 7.0

    HeadRank lifts preference optimization into attention space via entropy-regularized head selection and distribution regularizers to sharpen discriminability for efficient listwise reranking.

  48. HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads

    cs.IR 2026-04 unverdicted novelty 7.0

    HeadRank improves decoding-free passage reranking by preference-aligning attention heads to increase discriminability in middle-context documents, outperforming baselines on 14 benchmarks with only 211 training queries.

  49. Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs

    cs.AI 2026-04 accept novelty 7.0

    The paper delivers the first survey of abductive reasoning in LLMs, a unified two-stage taxonomy, a compact benchmark, and an analysis of gaps relative to deductive and inductive reasoning.

  50. Screening Is Enough

    cs.LG 2026-04 unverdicted novelty 7.0

    Multiscreen replaces softmax attention with screening to provide absolute query-key relevance, resulting in models with 30% fewer parameters that maintain stable performance at long contexts.

  51. Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory

    cs.LG 2026-03 unverdicted novelty 7.0

    Muon achieves higher storage capacity than SGD and matches Newton's method in one-step recovery rates for associative memory under power-law distributions, while saturating at larger critical batch sizes and showing f...

  52. How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability

    cs.CL 2026-01 unverdicted novelty 7.0

    Transformer weights at early training stages are closed-form compositions of bigram, token-interchangeability, and context mappings that directly reflect text-corpus statistics and explain the emergence of semantic as...

  53. The Bayesian Geometry of Transformer Attention

    cs.LG 2025-12 unverdicted novelty 7.0

    Small transformers reproduce known Bayesian posteriors with 10^{-3} to 10^{-4} bit accuracy in verifiable wind-tunnel tasks via residual belief states, FFN updates, and attention routing, while MLPs do not.

  54. DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning

    cs.AI 2025-11 unverdicted novelty 7.0

    DecompSR is a large, symbolically verified benchmark dataset and generation framework that independently varies productivity, substitutivity, overgeneralisation, and systematicity to probe compositional multihop spati...

  55. Soft Head Selection for Injecting ICL-Derived Task Embeddings

    cs.CL 2025-07 conditional novelty 7.0

    SITE applies soft gradient-based head selection to inject ICL-derived task embeddings, outperforming prior embedding adaptation and few-shot ICL across generation, reasoning, and NLU tasks on 12 LLMs from 4B to 70B pa...

  56. A Markov Categorical Framework for Language Modeling

    cs.LG 2025-07 unverdicted novelty 7.0

    A Markov category framework for language models provides an information-theoretic rationale for speculative decoding and shows that a quadratic surrogate to negative log-likelihood induces generalized CCA alignment in...

  57. A ghost mechanism: An analytical model of abrupt learning in recurrent networks

    cs.LG 2025-01 unverdicted novelty 7.0

    The ghost mechanism derives a 1D canonical model of abrupt learning in RNNs from ghost points of saddle-node bifurcations, predicting an inverse-power-law critical learning rate and gradient-based failure modes.

  58. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  59. Jamba: A Hybrid Transformer-Mamba Language Model

    cs.CL 2024-03 conditional novelty 7.0

    Jamba presents a hybrid Transformer-Mamba MoE architecture for LLMs that delivers state-of-the-art benchmark performance and strong results up to 256K token contexts while fitting in one 80GB GPU with high throughput.

  60. Steering Language Models With Activation Engineering

    cs.CL 2023-08 unverdicted novelty 7.0

    Activation Addition steers language models by adding contrastive activation vectors from prompt pairs to control high-level properties like sentiment and toxicity at inference time without training.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 147 Pith papers · 8 internal anchors

  1. [1]

    Language Models are Few-Shot Learners

    arXiv preprint arXiv:2005.14165. . LaMDA: our breakthrough conversation technology[link] Collins, E. and Ghahramani, Z.,

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  2. [2]

    Evaluating Large Language Models Trained on Code

    arXiv preprint arXiv:2107.03374. . Towards a human-like open-domain chatbot Adiwardana, D., Luong, M., So, D.R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y. and others,,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2001.09977. . Scaling Language Models: Methods, Analysis & Insights from Training Gopher[PDF] Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G.v.d., Hendricks, L.A., Rauh, M., Huang...

    work page arXiv 2001

  4. [4]

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

    arXiv preprint arXiv:2201.02177. . The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models[link] Pan, A., Bhatia, K. and Steinhardt, J.,

    work page internal anchor Pith review arXiv

  5. [5]

    Scaling Laws for Neural Language Models

    arXiv preprint arXiv:2001.08361. . Risks from Learned Optimization in Advanced Machine Learning Systems Hubinger, E., Merwijk, C.v., Mikulik, V., Skalse, J. and Garrabrant, S.,

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  6. [6]

    Shortformer: Better language modeling using shorter inputs

    arXiv preprint arXiv:2012.15832. . It wouldʼve been better if we had used a start of sequence token for the copying head evaluator as well, but we omitted it by mistake. Without the “start of sequence” token, some heads that were doing prefix matching on real data would get anomalously low scores on our test sequences[ ↩ ] ,,, ,, pp . A General Language A...

    work page arXiv 2012

  7. [7]

    A General Language Assistant as a Laboratory for Alignment

    arXiv preprint arXiv:2112.00861. . Common Crawl[link] Foundation, T.C.C.. . The Pile: An 800GB Dataset of Diverse Text for Language Modeling Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S. and Leahy, C.,

    work page internal anchor Pith review arXiv

  8. [8]

    A Multiscale Visualization of Attention in the Transformer Model

    arXiv preprint arXiv:1906.05714. . Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned[PDF] Voita, E., Talbot, D., Moiseev, F., Sennrich, R. and Titov, I.,

    work page Pith review arXiv 1906

  9. [9]

    Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

    arXiv preprint arXiv:1905.09418. . What does bert look at? an analysis of bert's attention[PDF] Clark, K., Khandelwal, U., Levy, O. and Manning, C.D.,

    work page Pith review arXiv 1905

  10. [10]

    What Does BERT Look At? An Analysis of BERT's Attention

    arXiv preprint arXiv:1906.04341. . Do attention heads in bert track syntactic dependencies?[PDF] Htut, P.M., Phang, J., Bordia, S. and Bowman, S.R.,

    work page Pith review arXiv 1906

  11. [11]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others

    arXiv preprint arXiv:1911.12246. . Attention is not all you need: Pure attention loses rank doubly exponentially with depth[PDF] Dong, Y., Cordonnier, J. and Loukas, A.,

    work page arXiv 1911

  12. [12]

    Dong, J.-B

    arXiv preprint arXiv:2103.03404. . What Context Features Can Transformer Language Models Use? O'Connor, J. and Andreas, J.,

    work page arXiv

  13. [13]

    An Explanation of In-context Learning as Implicit Bayesian Inference Xie, S.M., Raghunathan, A., Liang, P

    arXiv preprint arXiv:2106.08367. . An Explanation of In-context Learning as Implicit Bayesian Inference Xie, S.M., Raghunathan, A., Liang, P. and Ma, T.,

    work page arXiv

  14. [14]

    An Explanation of In-context Learning as Implicit Bayesian Inference

    arXiv preprint arXiv:2111.02080. . Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H. and Zettlemoyer, L.,

    work page internal anchor Pith review arXiv

  15. [15]

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    arXiv preprint arXiv:2202.12837. . Reconciling modern machine-learning practice and the classical bias--variance trade-off Belkin, M., Hsu, D., Ma, S. and Mandal, S.,

    work page internal anchor Pith review arXiv

  16. [16]

    Journal of Statistical Mechanics: Theory and Experiment, Vol 2021(12), pp. 124003. IOP Publishing. . Future ML Systems Will Be Qualitatively Different[link] Steinhardt, J.,

    work page 2021

  17. [17]

    Qualitatively characterizing neural network optimization problems

    arXiv preprint arXiv:1412.6544. . Analyzing monotonic linear interpolation in neural network loss landscapes Lucas, J., Bae, J., Zhang, M.R., Fort, S., Zemel, R. and Grosse, R.,

    work page Pith review arXiv

  18. [18]

    Analyzing monotonic linear interpolation in neural network loss landscapes, 2021

    arXiv preprint arXiv:2104.11044. . Geometry of neural network loss surfaces via random matrix theory Pennington, J. and Bahri, Y.,

    work page arXiv

  19. [19]

    emnlp-main.466/

    Distill. DOI: 10.23915/distill.00024.001 . Convergent learning: Do different neural networks learn the same representations? Li, Y., Yosinski, J., Clune, J., Lipson, H., Hopcroft, J.E. and others,,

    work page doi:10.23915/distill.00024.001

  20. [20]

    Similarity of Neural Network Representations Revisited

    arXiv preprint arXiv:1905.00414. . High-Low Frequency Detectors Schubert, L., Voss, C., Cammarata, N., Goh, G. and Olah, C.,

    work page Pith review arXiv 1905

  21. [21]

    Distill , author =

    Distill. DOI: 10.23915/distill.00024.005 . Performance-optimized hierarchical models predict neural responses in higher visual cortex Yamins, D.L., Hong, H., Cadieu, C.F., Solomon, E.A., Seibert, D. and DiCarlo, J.J.,

    work page doi:10.23915/distill.00024.005

  22. [22]

    Distill , author =

    Distill. DOI: 10.23915/distill.00030 . Neural machine translation by jointly learning to align and translate Bahdanau, D., Cho, K. and Bengio, Y.,

    work page doi:10.23915/distill.00030

  23. [23]

    Neural Machine Translation by Jointly Learning to Align and Translate

    arXiv preprint arXiv:1409.0473. . Listen, attend and spell Chan, W., Jaitly, N., Le, Q.V. and Vinyals, O.,

    work page internal anchor Pith review arXiv

  24. [24]

    Listen, Attend and Spell

    arXiv preprint arXiv:1508.01211. Acknowledgments In writing this paper, our thinking and exposition was greatly clarified by detailed correspondence with Sam Bowman, Paul Christiano, Aidan Gomez, Dan Hendrycks, Jacob Hilton, Evan Hubinger, Andrew Ilyas, Percy Liang, Tom Lieberum, Chris Maddison, Aleksander Madry, Ethan Perez, Jacob Steinhardt, and Martin ...

    work page Pith review arXiv

  25. [25]

    copying”, “prefix matching

    BibTeX Citation: @article{olsson2022context, title={In-context Learning and Induction Heads}, author={Olsson, Catherine and Elhage, Nelson and Nanda, Neel and Joseph, Nicholas and DasSarma, Nova and Henighan, Tom and Mann, Ben and Askell, Amanda and Bai, Yuntao and Chen, Anna and Conerly, Tom and Drain, Dawn and Ganguli, Deep and Hatfield-Dodds, Zac and H...

    work page 2022