{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:FJNRNXM56FQ5232WSXQWFHHIY4","short_pith_number":"pith:FJNRNXM5","schema_version":"1.0","canonical_sha256":"2a5b16dd9df161dd6f5695e1629ce8c733369779ac09705c36369356641a2981","source":{"kind":"arxiv","id":"2209.11895","version":1},"attestation_state":"computed","paper":{"title":"In-context Learning and Induction Heads","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Induction heads implement the core copying algorithm behind in-context learning in transformers.","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Amanda Askell, Andy Jones, Anna Chen, Ben Mann, Catherine Olsson, Chris Olah, Danny Hernandez, Dario Amodei, Dawn Drain, Deep Ganguli, Jack Clark, Jackson Kernion, Jared Kaplan, Kamal Ndousse, Liane Lovitt, Neel Nanda, Nelson Elhage, Nicholas Joseph, Nova DasSarma, Sam McCandlish, Scott Johnston, Tom Brown, Tom Conerly, Tom Henighan, Yuntao Bai, Zac Hatfield-Dodds","submitted_at":"2022-09-24T00:43:19Z","abstract_excerpt":"\"Induction heads\" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all \"in-context learning\" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguin"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2209.11895","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.LG","submitted_at":"2022-09-24T00:43:19Z","cross_cats_sorted":[],"title_canon_sha256":"f8b2a0a341303093e25b2e1d2834fdfca6fa832a89c63e46757e7ceabc1cc78c","abstract_canon_sha256":"a9d9d8560c4381d747a14e1cdf6e26154df31a0faafb25eba1a2760b94c23879"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T05:00:29.390548Z","signature_b64":"1hFl83cyeG2vist99J0EW+BDpghn+io9G2gxSWma+XIIldykzkoCaGdjw50cxuZHFYDoUyV0l+7hl8GReASMCw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"2a5b16dd9df161dd6f5695e1629ce8c733369779ac09705c36369356641a2981","last_reissued_at":"2026-07-05T05:00:29.389977Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T05:00:29.389977Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"In-context Learning and Induction Heads","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Induction heads implement the core copying algorithm behind in-context learning in transformers.","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Amanda Askell, Andy Jones, Anna Chen, Ben Mann, Catherine Olsson, Chris Olah, Danny Hernandez, Dario Amodei, Dawn Drain, Deep Ganguli, Jack Clark, Jackson Kernion, Jared Kaplan, Kamal Ndousse, Liane Lovitt, Neel Nanda, Nelson Elhage, Nicholas Joseph, Nova DasSarma, Sam McCandlish, Scott Johnston, Tom Brown, Tom Conerly, Tom Henighan, Yuntao Bai, Zac Hatfield-Dodds","submitted_at":"2022-09-24T00:43:19Z","abstract_excerpt":"\"Induction heads\" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all \"in-context learning\" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguin"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"induction heads might constitute the mechanism for the majority of all 'in-context learning' in large transformer models (i.e. decreasing loss at increasing token indices)","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the emergence of induction heads is causally responsible for the observed increase in in-context learning ability rather than both phenomena being downstream effects of some other training dynamic.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Induction heads, which implement pattern completion in attention, develop at the same training stage as a sudden rise in in-context learning, providing evidence they are the primary mechanism for in-context learning in transformers.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Induction heads implement the core copying algorithm behind in-context learning in transformers.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"82e3dd5268fe1c82b5db72e78fda60ad9d8712d0d81a9f7134c3f9c55571f66e"},"source":{"id":"2209.11895","kind":"arxiv","version":1},"verdict":{"id":"b3aad1ed-d955-4cb3-a3c7-d89ce6c0dd91","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-11T03:44:13.815348Z","strongest_claim":"induction heads might constitute the mechanism for the majority of all 'in-context learning' in large transformer models (i.e. decreasing loss at increasing token indices)","one_line_summary":"Induction heads, which implement pattern completion in attention, develop at the same training stage as a sudden rise in in-context learning, providing evidence they are the primary mechanism for in-context learning in transformers.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the emergence of induction heads is causally responsible for the observed increase in in-context learning ability rather than both phenomena being downstream effects of some other training dynamic.","pith_extraction_headline":"Induction heads implement the core copying algorithm behind in-context learning in transformers."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2209.11895/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":25,"sample":[{"doi":"","year":2005,"title":"Language Models are Few-Shot Learners","work_id":"214732c0-2edd-44a0-af9e-28184a2b8279","ref_index":1,"cited_arxiv_id":"2005.14165","is_internal_anchor":true},{"doi":"","year":null,"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","ref_index":2,"cited_arxiv_id":"2107.03374","is_internal_anchor":true},{"doi":"","year":2001,"title":"arXiv preprint arXiv:2001.09977 , year=","work_id":"ba5130eb-69cb-4786-944d-f724b5f7876c","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets","work_id":"a3c30ead-1625-4c18-a9c1-e4928dcd0da6","ref_index":4,"cited_arxiv_id":"2201.02177","is_internal_anchor":true},{"doi":"","year":2001,"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","ref_index":5,"cited_arxiv_id":"2001.08361","is_internal_anchor":true}],"resolved_work":25,"snapshot_sha256":"0a95b95e3169f058af2f8400eb5b907b26253ed25e786e056a4b4e1e051862db","internal_anchors":8},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2209.11895","created_at":"2026-07-05T05:00:29.390058+00:00"},{"alias_kind":"arxiv_version","alias_value":"2209.11895v1","created_at":"2026-07-05T05:00:29.390058+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2209.11895","created_at":"2026-07-05T05:00:29.390058+00:00"},{"alias_kind":"pith_short_12","alias_value":"FJNRNXM56FQ5","created_at":"2026-07-05T05:00:29.390058+00:00"},{"alias_kind":"pith_short_16","alias_value":"FJNRNXM56FQ5232W","created_at":"2026-07-05T05:00:29.390058+00:00"},{"alias_kind":"pith_short_8","alias_value":"FJNRNXM5","created_at":"2026-07-05T05:00:29.390058+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":160,"internal_anchor_count":160,"sample":[{"citing_arxiv_id":"2607.06621","citing_title":"Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2607.08009","citing_title":"From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs","ref_index":125,"is_internal_anchor":true},{"citing_arxiv_id":"2607.08393","citing_title":"Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2607.07316","citing_title":"Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2607.07678","citing_title":"How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2606.26050","citing_title":"Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2604.22027","citing_title":"Shared Lexical Task Representations Explain Behavioral Variability In LLMs","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2606.25010","citing_title":"Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24667","citing_title":"DREAM: Dense Retrieval Embeddings via Autoregressive Modeling","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24467","citing_title":"CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2606.26660","citing_title":"Generating Special Triangulations with Transformers","ref_index":47,"is_internal_anchor":true},{"citing_arxiv_id":"2606.23885","citing_title":"Mind the Heads: Topological Representation Alignment for Multimodal LLMs","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2606.21732","citing_title":"Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agents","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2606.21228","citing_title":"Sakana Fugu Technical Report","ref_index":113,"is_internal_anchor":true},{"citing_arxiv_id":"2606.19697","citing_title":"Efficiently Representing Algorithms With Chain-of-Thought Transformers","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2606.18164","citing_title":"Learning Dynamics of Chain-of-Thought State Tracking in a Solvable Transformer Model","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01538","citing_title":"Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale","ref_index":39,"is_internal_anchor":true},{"citing_arxiv_id":"2606.20678","citing_title":"Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2606.13607","citing_title":"Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2606.12898","citing_title":"Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2606.12058","citing_title":"Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2607.02509","citing_title":"ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2606.12412","citing_title":"Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2606.10304","citing_title":"MIRAGE: A Polarity-Flipping Encoding Subspace in LLM Agents","ref_index":60,"is_internal_anchor":true},{"citing_arxiv_id":"2606.08804","citing_title":"Q-Delta: Beyond Key-Value Associative State Evolution","ref_index":74,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4","json":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4.json","graph_json":"https://pith.science/api/pith-number/FJNRNXM56FQ5232WSXQWFHHIY4/graph.json","events_json":"https://pith.science/api/pith-number/FJNRNXM56FQ5232WSXQWFHHIY4/events.json","paper":"https://pith.science/paper/FJNRNXM5"},"agent_actions":{"view_html":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4","download_json":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4.json","view_paper":"https://pith.science/paper/FJNRNXM5","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2209.11895&json=true","fetch_graph":"https://pith.science/api/pith-number/FJNRNXM56FQ5232WSXQWFHHIY4/graph.json","fetch_events":"https://pith.science/api/pith-number/FJNRNXM56FQ5232WSXQWFHHIY4/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4/action/timestamp_anchor","attest_storage":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4/action/storage_attestation","attest_author":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4/action/author_attestation","sign_citation":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4/action/citation_signature","submit_replication":"https://pith.science/pith/FJNRNXM56FQ5232WSXQWFHHIY4/action/replication_record"}},"created_at":"2026-07-05T05:00:29.390058+00:00","updated_at":"2026-07-05T05:00:29.390058+00:00"}