Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attention heads that promote factual output do so by general copy suppression, not by selectively suppressing counterfactual tokens.

desk verdict A genuinely useful reproducibility study whose central claim—that 'factual recall' heads do general copy suppression—is plausible but lacks the one control that would nail it. read the letter →

arxiv 2507.11809 v1 pith:76JIXJR2 submitted 2025-07-16 cs.CL cs.AIcs.LG

Dante Campregher , Yanxu Chen , Sander Hoffman , Maria Heuss This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords MechanisticinterpretabilityAttentionheadsCounterfactualreasoningCopysuppressionFactualrecallDomainspecializationReproducibilitystudyLargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to settle a dispute about why certain attention heads in generative language models shift the output away from a counterfactual statement in the prompt and toward a fact stored in the model's parameters. Prior studies called these 'factual recall heads' and proposed that they selectively suppress the copied counterfactual token. Working with GPT-2 and Pythia-6.9B on CounterFact-derived prompts, this study reproduces the factual-versus-counterfactual competition and then tests the two candidate mechanisms directly. The decisive result is that strengthening the heads also suppresses correct facts when the prompt itself states the fact, indicating general copy suppression rather than targeted counterfactual vetoing. The paper also shows that the heads' contributions are domain-dependent, with larger models displaying more specialized and category-sensitive patterns.

What carries the argument

The central object is the transformer attention head, specifically the heads L10H7 and L11H10 in GPT-2 and L17H28, L20H18, L21H8 in Pythia-6.9B, which earlier work flagged as 'factual recall heads.' The paper carries the argument with two tools: attention modification, which scales (by a factor $\alpha$) the attention paid by the final-token position to an earlier position, and the logit-difference measure $\Delta_{\mathrm{cofa}} = \mathrm{logit}(t_{\mathrm{cofa}}) - \mathrm{logit}(t_{\mathrm{fact}})$, which says which side a head supports. The discriminating experiment is the counterfactual-to-factual replacement in the prompt, which cleanly separates selective counterfactual suppression (hypothesis 2.2.1) from general induction/copy suppression (hypothesis 2.2.2). Domain specialization is probed by splitting prompts by answer category and by decomposing each head's OV matrix with SVD to inspect which tokens a head encodes.

What would settle it

A direct falsifier would be a matched-pair experiment in which the context states either a falsehood or the corresponding fact with the same surface structure, and some head raises factual output only in the falsehood case; if such a head exists, the general copy-suppression account fails. Simpler still, on the fact-replaced prompts one could measure whether the suppression is specific to tokens that match an earlier context position or instead hits all high-probability tokens, since copy suppression predicts the former.

Watch

Extended reading notes

Core claim

The paper's central claim is that the attention heads previously labeled 'factual recall heads' operate as general copy-suppression (anti-induction) heads: they inhibit the copying of tokens from earlier context positions, regardless of whether the token is a counterfactual or the true fact. The decisive experiment (Section 4.3.1) replaces each counterfactual token in the prompt with the factual token; if the heads were selective counterfactual suppressors, boosting them should leave factual output largely intact, but instead the factual prediction rate falls steadily as the boosting factor $\alpha$ increases, falling below 70% at $\alpha=10$ in Pythia-6.9B and below 90%–70% in GPT-2. A secondary claim is that these heads are domain-specific: their effect on the counterfactual-factual logit difference varies strongly with the answer category, and in the larger Pythia model some heads even flip which side they support across categories, while heads near zero mean influence can still be highly influential on individual samples.

Load-bearing premise

The central conclusion hinges on the assumption that replacing a counterfactual token with a factual token in the prompt isolates the heads' copy-suppression function, rather than changing the attention pattern in some other way or destabilizing the model's behavior.

Editorial extensions

If this is right

  • Despite the earlier framing, strengthening these heads is not a reliable lever for increasing factual output: when the correct answer is already present in the context, boosting them suppresses it too.
  • The mechanism of these heads aligns with the anti-induction/copy-suppression motif previously described for L10H7 in GPT-2, so the three prior studies can be reconciled under one account.
  • The competition between factual and counterfactual tokens is itself reproducible across different premises and sentence structures, though the optimal boosting strength shifts.
  • Any intervention on these heads must be applied with knowledge of the answer domain, because a head that supports the factual side in one category can support the counterfactual side in another, especially in larger models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • For retrieval-augmented generation, this suggests that interventions tuned to suppress hallucinated or counterfactual context will also suppress genuinely retrieved facts when the same copying pathway is engaged, so faithfulness scores should be monitored when these heads are modified.
  • The near-zero-mean, high-variance heads found in Pythia-6.9B imply that full-dataset summary statistics can hide strong per-category behavior; future mechanistic studies should report per-category effects before assigning a head a global role.
  • A testable extension would check whether the general copy-suppression account transfers to instruction-tuned or encoder-decoder models, where the copied span and the final-token position are separated differently in the attention pattern.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper is a reproducibility study of prior mechanistic-interpretability work on factual and counterfactual competition in GPT-2 and Pythia-6.9B. The authors replicate the attention-head modification experiments of Ortu et al., probe the effect of prompt wording and sentence structure, and test two competing mechanisms: selective suppression of counterfactual tokens versus general copy suppression (anti-induction). Using prompts in which the counterfactual token is replaced by the factual token, they find that strengthening tfact-supporting heads lowers the proportion of factual predictions, which they interpret as evidence for general copy suppression. They also split the data by subject and answer categories and report that head contributions are category-dependent, with stronger category selectivity in Pythia-6.9B than in GPT-2. The paper makes its code available and largely reproduces the prior results it builds on.

Significance. If the copy-suppression conclusion holds, it changes the interpretation of heads labeled 'factual recall heads': rather than selectively suppressing counterfactual continuations, these heads suppress copying from context indiscriminately, which would explain why strengthening them can hurt factual outputs when the fact is present in the prompt. This is a valuable, falsifiable result with direct consequences for attention-editing interventions and for the robustness of mechanistic explanations. The study is strengthened by its use of held-out manipulations (fact-replaced prompts, category splits) rather than parameter fitting, by its reliance on the original code and datasets, and by its open-source release. The main interpretive risk is that the key intervention lacks a same-dataset random-head control, so the conclusion is conditional on that control.

major comments (4)
  1. [§4.3.1–4.3.2, Figure 5] The central conclusion that tfact-supporting heads perform general copy suppression rather than counterfactual-specific suppression rests on the fact-replaced-prompt intervention in Figure 5, but that experiment has no random-head control on the fact-replaced dataset. The random-head baseline in Appendix B.1 (Figure 9) is run only on the original counterfactual prompts, and the paper itself reports that Pythia-6.9B at alpha=10 'mostly return[s] "the" or "also"' (Section 4.3.2), which is exactly the kind of non-specific degradation that a control is meant to detect. Without a random-head control on the same fact-replaced prompts, the observed drop in factual predictions could be a generic perturbation artifact rather than copy suppression, so Hypothesis 2.2.2 is not yet decisively preferred over Hypothesis 2.2.1.
  2. [§4.4.2, §5 (Hypothesis 2.3.2)] The claim that larger models exhibit more specialized and category-sensitive attention heads is based on comparing GPT-2 (117M parameters, 12 layers, 12 heads) with Pythia-6.9B (32 layers, 32 heads). This comparison confounds model size with model family, vocabulary, training data, and architecture, so Figures 7, 10, and 11 cannot by themselves support Hypothesis 2.3.2. Section 5 acknowledges that experiments across multiple sizes within one model family are needed; this should be treated as a required experiment, not a future-work remark.
  3. [§4.1–4.4 (Tables 1, Figures 2–5)] Most quantitative results are reported as single-run counts with no error bars or seed variation (for example, Table 1 and Figures 2 and 5), even though the random-head baseline in Appendix B.1 uses four seeds. Without variance estimates, statements such as 'α = 10 yields better results than α = 5' for same-structure sentences (Section 4.2.2) and the numerical size of the drop in Figure 5 cannot be distinguished from run-to-run noise. The authors should report repeated runs or bootstrap intervals for at least the main intervention experiments.
  4. [§4.4.1] The domain-specialization analysis depends entirely on GPT-4o-generated category labels, but the paper only says the classification results were 'manually checked' without specifying the prompting scheme, the number or definition of categories, or the checking and agreement procedure. Since Figures 6–8 and the head-selection analysis in Appendix B.2 are defined by these labels, this omission is a reproducibility gap for the domain claims.
minor comments (4)
  1. [Introduction] The typo 'answe,r' in the first section should be corrected.
  2. [Table 1 caption] The caption lists values as counts but does not state the model or dataset; the text implies GPT-2 and the 10,000-entry CounterFact-derived set, but the caption should say so explicitly.
  3. [§4.2.2] Phrases such as 'This may be due to that' and 'the results become less pronounced' should be copy-edited for clarity.
  4. [Appendix B.1, Figure 9] Figure 9 would be easier to compare with Figure 5 if both plots used the same axes and response categories.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are tested on held-out prompt manipulations rather than on the data used to identify the heads.

full rationale

The paper's main mechanistic claim is that attention heads supporting the factual token act via general copy suppression rather than selective counterfactual suppression. This is tested in Section 4.3.1 by replacing the counterfactual token in each prompt with the factual token and then measuring how strengthening the previously identified heads affects factual prediction rates. The heads of interest come from prior work (Ortu et al.) and from logit attribution on the original CounterFact prompts; the fact-replaced prompts are a distinct, modified dataset that is not used to fit any parameter and is not equivalent by construction to the original head-identification input. The observed drop in factual predictions when alpha increases is an empirical outcome, not an algebraic consequence of the experimental setup. Similarly, the domain-specialization experiments split the data by category and compare head contributions across categories; no parameter is fitted to the specific result being claimed. The paper does cite prior work, including Ortu et al. and McDougall et al., but no load-bearing claim is justified solely by a self-citation whose content is unverified, and the authors' own experiments provide independent evidence for the hypotheses. The absence of a random-head control on the fact-replaced prompts is a real methodological limitation and a threat to the internal validity of the copy-suppression conclusion, but it is not a circularity: a missing control is not the same as a definitional reduction or a fitted-input prediction. No equation in the paper reduces a predicted quantity to an input by construction, and no result is renamed as a discovery when it is actually the fit itself. Therefore, under the stated criteria, the paper does not exhibit significant circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The analysis relies on standard mechanistic interpretability techniques (logit attribution, attention modification, SVD of OV matrices) whose faithfulness is assumed, and on the CounterFact-derived datasets and GPT-4o category labels, which are taken as given. No free parameters are fitted; alpha values are experimental interventions.

assumptions (6)
  • domain assumption Logit attribution using transformer_lens accurately attributes the causal contribution of attention heads to output logits.
    Section 3.1.1; the paper relies on this standard technique from the reproduced papers without independent verification of its faithfulness.
  • domain assumption Attention modification A'_{hl}^{ij} = alpha * A_{hl}^{ij} is a causally interpretable intervention on the model.
    Section 3.1.2; the paper assumes scaling attention scores changes model behavior in a way that reveals the head's role.
  • domain assumption SVD of OV matrices yields interpretable token encodings.
    Section 3.1.3, following Millidge and Black; interpretability of singular vectors is asserted, not proven.
  • domain assumption The CounterFact-derived datasets, filtered to prompts the model answers correctly, are representative of factual/counterfactual competition.
    Section 3.3; the dataset construction from Ortu et al. is taken as given, including the 10,000-prompt sample.
  • domain assumption GPT-4o category labels (Location, Language, Organization) are accurate after manual checking.
    Section 4.4.1; classification produced by GPT-4o and manually checked, but the manual check criteria are not specified.
  • ad hoc to paper Heads identified by Ortu et al. as tfact-supporting are the relevant heads for testing the mechanism hypotheses.
    Sections 2.2 and 4.3; the study restricts analysis to L10H7/L11H10 for GPT-2 and L17H28/L20H18/L21H8 for Pythia-6.9B based on prior work, so conclusions about 'these heads' do not cover all heads.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models." pith.science (2026). https://pith.science/paper/76JIXJR2

@misc{pith2026250711809,
  author       = {Pith},
  title        = {Pith review of: Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76JIXJR2}},
  note         = {Machine review of arXiv:2507.11809}
}
read the original abstract

This paper presents a reproducibility study examining how Large Language Models (LLMs) manage competing factual and counterfactual information, focusing on the role of attention heads in this process. We attempt to reproduce and reconcile findings from three recent studies by Ortu et al., Yu, Merullo, and Pavlick and McDougall et al. that investigate the competition between model-learned facts and contradictory context information through Mechanistic Interpretability tools. Our study specifically examines the relationship between attention head strength and factual output ratios, evaluates competing hypotheses about attention heads' suppression mechanisms, and investigates the domain specificity of these attention patterns. Our findings suggest that attention heads promoting factual output do so via general copy suppression rather than selective counterfactual suppression, as strengthening them can also inhibit correct facts. Additionally, we show that attention head behavior is domain-dependent, with larger models exhibiting more specialized and category-sensitive patterns.

Figures

Figures reproduced from arXiv: 2507.11809 by the authors.

Figure 1
Figure 1. Visualisation of the choice between counterfactual and factual token. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Results of attention modification on GPT-2 and Pythia-6.9B. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Results of GPT-2 attention modification when splitting the dataset based on whether or not [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Results of Pythia-6.9B attention modification when splitting the dataset based on whether or not [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Results of attention modification after changing the counterfactual targets to factual targets in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Attention scores of relevant heads for the most common categories that GPT-2 can predict. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of the strength of the logit difference for different heads and categories. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Heatmap of the top-k tokens of the singular vectors sorted by their singular values. 5 Discussion Generally, our experiment results support the hypothesis 2.1 that when the prompt contains a counterfactual token tcofa, there are attention heads that contribute signific…
Figure 9
Figure 9. Figure 9: Attention modification on random heads in the same layer as heads of interest. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: This highlights which heads differ the most across categories. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 10
Figure 10. Figure 10: Mean ∆cofa values of each head in Pythia-6.9B for each of the analysed categories 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Standard deviation between the mean ∆cofa value of the three analyzed categories of the dataset 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Standard deviation of ∆cofa values of each head in Pythia-6.9B for each the analysed categories 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Heatmap of the top-k tokens of the singular vectors sorted by their singular values. B.3 Further SVD analysis of relevant heads Starting with L15H17 (fig 13b), a head that globally promotes tcofa. We do not see many tokens associated with meaning beyond the low-level …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

  1. [1]

    Mechanistic Interpretability for AI Safety–A Review

    Leonard Bereska and Efstratios Gavves. “Mechanistic Interpretability for AI Safety–A Review”. In: arXiv preprint arXiv:2404.14082 (2024)

    arXiv 2024

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman et al. “Pythia: A suite for analyzing large language models across training and scaling”. In: International Conference on Machine Learning . PMLR. 2023, pp. 2397–2430

    work page 2023

  3. [3]

    Measuring and Improving Consistency in Pretrained Language Models

    Yanai Elazar et al. “Measuring and Improving Consistency in Pretrained Language Models”. In: Transactions of the Association for Computational Linguistics 9 (2021). Ed. by Brian Roark and Ani Nenkova, pp. 1012–1031.doi: 10.1162/tacl_a_00410. url: https://aclanthology.org/2021. tacl-1.60/

    doi:10.1162/tacl_a_00410 2021

  4. [4]

    A mathematical framework for transformer circuits

    Nelson Elhage et al. “A mathematical framework for transformer circuits”. In:Transformer Circuits Thread 1.1 (2021), p. 12

    work page 2021

  5. [5]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis et al. “Retrieval-augmented generation for knowledge-intensive nlp tasks”. In:Advances in neural information processing systems 33 (2020), pp. 9459–9474

    work page 2020

  6. [6]

    Understanding neural networks through representation erasure

    Jiwei Li, Will Monroe, and Dan Jurafsky. “Understanding neural networks through representation erasure”. In: arXiv preprint arXiv:1612.08220 (2016)

    arXiv 2016

  7. [7]

    Copy Suppression: Comprehensively Understanding a Motif in Lan- guage Model Attention Heads

    Callum Stuart McDougall et al. “Copy Suppression: Comprehensively Understanding a Motif in Lan- guage Model Attention Heads”. In:Proceedings of the 7th BlackboxNLP Workshop: Analyzing and In- terpreting Neural Networks for NLP . Ed. by Yonatan Belinkov et al. Miami, Florida, US: Association for Computational Linguistics, Nov. 2024, pp. 337–363.doi: 10.186...

    doi:10.18653/v1/2024.blackboxnlp-1.22 2024

  8. [8]

    Locating and Editing Factual Associations in GPT

    Kevin Meng et al. “Locating and Editing Factual Associations in GPT”. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 17359–17372. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2022 / file / 6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf

    work page 2022

Show all 17 references
  1. [9]

    The singular value decompositions of transformer weight matrices are highly interpretable

    Beren Millidge and Sid Black. “The singular value decompositions of transformer weight matrices are highly interpretable”. In: LessWrong (2022). url: https : / / www . lesswrong . com / posts / mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight

    2022

  2. [10]

    TransformerLens

    Neel Nanda and Joseph Bloom. TransformerLens. https : / / github . com / TransformerLensOrg / TransformerLens. 2022

    2022

  3. [11]

    interpreting GPT: the logit lens

    nostalgebraist. “interpreting GPT: the logit lens”. In: LessWrong (2020). url: https : / / www . lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  4. [12]

    In-context Learning and Induction Heads

    Catherine Olsson et al. “In-context Learning and Induction Heads”. In:Transformer Circuits Thread (2022). https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html

    2022

  5. [13]

    Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals

    Francesco Ortu et al. “Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals”. In: Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers) . Ed. by Lun-Wei Ku, Andre Martins, and Vivek ...

    2024 doi

  6. [14]

    Language Models are Unsupervised Multitask Learners

    Alec Radford et al. “Language Models are Unsupervised Multitask Learners”. In: (2019)

    2019

  7. [15]

    How Much Knowledge Can You Pack Into the Parameters of a Language Model?

    Adam Roberts, Colin Raffel, and Noam Shazeer. “How Much Knowledge Can You Pack Into the Parameters of a Language Model?” In:Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, pp. 5418–5426

    2020

  8. [16]

    Correctness is not Faithfulness in RAG Attributions

    Jonas Wallat et al. “Correctness is not Faithfulness in RAG Attributions”. In: arXiv preprint arXiv:2412.18004 (Dec. 2024)

    2024 arXiv

  9. [17]

    Characterizing Mechanisms for Factual Recall in Language Models

    Qinan Yu, Jack Merullo, and Ellie Pavlick. “Characterizing Mechanisms for Factual Recall in Language Models”. In: The 2023 Conference on Empirical Methods in Natural Language Processing . 2023. 13 Published in Transactions on Machine Learning Research (07/2025) Figure 9: Atten...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.