Pith. sign in

REVIEW 11 cited by

Copy Suppression: Comprehensively Understanding an Attention Head

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.04625 v1 pith:6CJWDV56 submitted 2023-10-06 cs.LG cs.AIcs.CL

Callum McDougall , Arthur Conmy , Cody Rushing , Thomas McGrath , Neel Nanda This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords copysuppressionheadself-repairattentionmodelrolebehavior
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present a single attention head in GPT-2 Small that has one main role across the entire training distribution. If components in earlier layers predict a certain token, and this token appears earlier in the context, the head suppresses it: we call this copy suppression. Attention Head 10.7 (L10H7) suppresses naive copying behavior which improves overall model calibration. This explains why multiple prior works studying certain narrow tasks found negative heads that systematically favored the wrong answer. We uncover the mechanism that the Negative Heads use for copy suppression with weights-based evidence and are able to explain 76.9% of the impact of L10H7 in GPT-2 Small. To the best of our knowledge, this is the most comprehensive description of the complete role of a component in a language model to date. One major effect of copy suppression is its role in self-repair. Self-repair refers to how ablating crucial model components results in downstream neural network parts compensating for this ablation. Copy suppression leads to self-repair: if an initial overconfident copier is ablated, then there is nothing to suppress. We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task. Interactive visualisations of the copy suppression phenomena may be seen at our web app https://copy-suppression.streamlit.app/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Graph Learning Learn Circuits?

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A GNN trained across synthetic transformer-task pairs predicts held-out circuit edges with median AUROC 0.902, close to but below dedicated per-case methods ACDC and EAP-IG.

  2. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.

  3. RCStat: A Statistical Framework for using Relative Contextualization in Transformers

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RCStat uses pre-softmax attention logits to define a Relative Contextualization score that improves adaptive KV-cache eviction and attention-head selection for attribution on LLaMA models.

  4. Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Lorsa replaces an MHSA layer with thousands of sparsely activated rank-1 attention heads and shows these heads recover known behaviors like induction heads plus new arithmetic and thematic units.

  5. On Mechanistic Circuits for Extractive Question-Answering

    cs.CL 2025-02 conditional novelty 6.0 of 10

    One attention head from the extracted context-faithfulness circuit provides reliable extractive QA attribution and improves context faithfulness when its attributions are added to the prompt.

  6. Structure Development in List-Sorting Transformers

    cs.LG 2025-01 conditional novelty 6.0 of 10

    In a list-sorting transformer, the mean and variance of gaps between adjacent sorted numbers determine whether attention heads split the vocabulary, suppress copying to calibrate, or switch off.

  7. From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.

  8. Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability

    cs.LG 2025-07 conditional novelty 5.0 of 10

    LayerNorm can be removed from all GPT-2 models by fine-tuning with a linear replacement, losing only a small amount of validation accuracy on filtered data.

  9. ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A post-training routing method that uses self-distillation to let frozen pretrained Transformers process only a subset of parameters and tokens, cutting active compute by 20 to 50 percent.

  10. New Faithfulness-Centric Interpretability Paradigms for Natural Language Processing

    cs.CL 2024-11 conditional novelty 4.0 of 10

    The thesis shows that randomly masking input tokens during fine-tuning makes post-hoc explanations of NLP models consistently faithful under an erasure-based faithfulness metric.

  11. Causality for Natural Language Processing

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A dissertation assembling the author's prior publications on causality for NLP, centered on two LLM causal reasoning benchmarks and their implications.

Pith tools