Pith. sign in

REVIEW 3 cited by

Copy Suppression: Comprehensively Understanding an Attention Head

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.04625 v1 pith:6CJWDV56 submitted 2023-10-06 cs.LG cs.AIcs.CL

Callum McDougall , Arthur Conmy , Cody Rushing , Thomas McGrath , Neel Nanda This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords copysuppressionheadself-repairattentionmodelrolebehavior
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a single attention head in GPT-2 Small that has one main role across the entire training distribution. If components in earlier layers predict a certain token, and this token appears earlier in the context, the head suppresses it: we call this copy suppression. Attention Head 10.7 (L10H7) suppresses naive copying behavior which improves overall model calibration. This explains why multiple prior works studying certain narrow tasks found negative heads that systematically favored the wrong answer. We uncover the mechanism that the Negative Heads use for copy suppression with weights-based evidence and are able to explain 76.9% of the impact of L10H7 in GPT-2 Small. To the best of our knowledge, this is the most comprehensive description of the complete role of a component in a language model to date. One major effect of copy suppression is its role in self-repair. Self-repair refers to how ablating crucial model components results in downstream neural network parts compensating for this ablation. Copy suppression leads to self-repair: if an initial overconfident copier is ablated, then there is nothing to suppress. We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task. Interactive visualisations of the copy suppression phenomena may be seen at our web app https://copy-suppression.streamlit.app/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.

  2. From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.

  3. Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability

    cs.LG 2025-07 conditional novelty 5.0 of 10

    LayerNorm can be removed from all GPT-2 models by fine-tuning with a linear replacement, losing only a small amount of validation accuracy on filtered data.

Pith tools