Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Backdoor triggers in LLMs appear to hijack the same attention heads the model already uses to decide its output language, rather than creating new circuits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:08 UTC pith:BLBMN3M3

load-bearing objection Plausible and novel, but the fake-trigger control confounds 'backdoor' with 'meaningful phrase'; needs a matched non-trigger before the headline Jaccards are trustworthy. the 3 major comments →

arxiv 2602.10382 v3 pith:BLBMN3M3 submitted 2026-02-11 cs.CL

Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Models

Th\'eo Lasnier , Wissam Antoun , Francis Kulumba , Beno\^it Sagot , Djam\'e Seddah This is my paper
classification cs.CL
keywords backdoor attacksmechanistic interpretabilityactivation patchinglanguage-switching triggersattention headsGaperon modelsmultilingual representationcircuit co-option
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that when a backdoor trigger is injected during pre-training to make a language model switch its output language, the trigger does not create a new dedicated circuit. Instead, the trigger appears to activate the same attention heads the model already uses to decide what language it is writing in. The evidence comes from activation patching on a family of three model scales (1B, 8B, 24B parameters): the set of heads a trigger engages overlaps with the set of heads that encode output language, with Jaccard indices between 0.18 and 0.66 over the top-10 heads, far above shuffled baselines. Trigger information also consolidates early — at roughly 7.5% to 25% of model depth. If correct, this means backdoor detection and defense could target the model's known language components rather than hunting for hidden trigger circuits.

Core claim

On the paper's own terms, the central discovery is that language-switching backdoor triggers co-opt the model's existing language circuitry rather than forming isolated mechanisms. By comparing activation-patching results over top-10 attention head sets, the authors show that heads activated by a genuine trigger overlap substantially with heads that naturally encode output language identity, with Jaccard indices of 0.18 to 0.66 across model sizes and languages — against near-zero shuffled baselines. The overlap holds for both French and German triggers, and the very same 'language heads' are shared across four tested target languages, including two that have no trigger at all. The authors al

What carries the argument

Activation patching is the central method: the authors run the model on a clean input (with a genuine trigger or a non-English context) and on a corrupted input (a matched fake trigger or English context), then replace a head's mean activation from the corrupted run with the clean one and measure the change in output-token log probability. Heads are ranked by this patching effect, and the top-10 heads for trigger processing and for natural language encoding form the two sets whose overlap is measured with the Jaccard index. Layer-wise patching across trigger token positions then locates where trigger information consolidates. The load-bearing comparison is between these two head sets, and th

Load-bearing premise

The counterfactual 'fake triggers' used as corrupted inputs are random token sequences matched only for length, while the real trigger is a meaningful Latin phrase, so the patching contrast may isolate 'meaningful vs meaningless' rather than 'trigger vs non-trigger'.

What would settle it

A concrete test: patch the identified trigger heads using, as the corrupted input, a meaningful Latin phrase that is not a backdoor trigger (matched in length and per-word token count). If the Jaccard overlap with natural language heads drops to near baseline, the overlap is driven by semantic content, not by trigger-specific processing. Alternatively, ablate the overlapping language/trigger heads and check whether the trigger still switches the output language — the paper itself lists causal verification of head necessity as future work; if the switch survives ablation, the overlap is not loa

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Backdoor detection could shift from hunting for anomalous hidden circuits to monitoring the activation patterns of known functional components, such as language heads.
  • If triggers co-opt existing circuitry, mitigation may be possible by intervening on those shared heads, for example by steering or ablating the language-identity direction.
  • Because the same language heads serve multiple target languages, a single detection or defense mechanism may generalize across trigger languages without retraining.
  • Early trigger formation (7.5%–25% of depth) suggests that shallow-layer interventions could catch backdoor behavior before it propagates to the output.
  • The shared-heads finding across scales indicates the co-option mechanism is a general property of these models, not a quirk of one size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If backdoors must route through pre-existing functional components, then analogous backdoors (e.g., sentiment or topic shifts) might similarly co-opt the model's natural sentiment or topic circuits — a prediction that could be tested on other backdoor types.
  • The 1B German trigger's two-stage formation pattern, which the paper notes is consistent with an induction head, suggests a concrete copying mechanism; probing whether that head copies the language representation from earlier positions would give a causal test of the co-option story.
  • The counterfactual control's asymmetry (meaningful Latin phrase vs random fake tokens) means the overlap may partly reflect content-bearingness rather than backdoor-specific processing; using a meaningful non-trigger Latin phrase as corrupted input would discriminate between these interpretations.
  • If the entanglement is real, backdoor 'stealth' may be limited by the model's own functional architecture — an attacker cannot fully hide a trigger if it must excite the same heads as normal language behavior, which would also explain why the injected triggers in these models are discoverable by these patching methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a mechanistic interpretability analysis of language-switching backdoors in the GAPperon model family (1B, 8B, 24B). Using activation patching, the authors localize where trigger information forms and identify attention heads involved in trigger processing and in natural language identity. Their central claim is that trigger-activated heads substantially overlap with heads that naturally encode output language, with Jaccard indices above shuffled baselines across model scales, suggesting that backdoor triggers co-opt existing language circuitry rather than forming new circuits. The paper includes three experiments: trigger-head identification (fake-trigger counterfactuals), natural-language-head identification, and layer-wise localization of trigger formation.

Significance. If the central claim holds, this is a valuable contribution: it would be the first mechanistic account of pretraining-injected language-switching backdoors across multiple scales, with direct implications for backdoor detection and mitigation. The paper benefits from a publicly available testbed, a clearly specified activation-patching protocol, shuffled baselines, and the inclusion of Italian/Spanish as trigger-free language controls. The main weaknesses are a confounded counterfactual control and the absence of uncertainty/sensitivity analysis for the headline Jaccard indices. Both are fixable, but they currently leave the co-option claim under-supported.

major comments (3)
  1. [§2.1, Eq. (2) and Exp. 1] The fake-trigger control is confounded. Fake triggers are matched only on total token length and tokens per word, while the genuine trigger is a meaningful three-word Latin phrase. Thus the patching metric Δ_l in Eq. (1) isolates the contrast between a meaningful, content-bearing sequence and random tokens, not specifically between a trained backdoor trigger and a non-trigger. Heads sensitive to lexical plausibility or 'natural input' would appear in H_trigger. Since Exp. 2 (Eq. 3) also contrasts natural-language inputs against English, shared 'naturalness' heads could inflate the Jaccard indices without any backdoor-specific co-option. A matched control using a non-trigger meaningful Latin phrase (or a set of such phrases) is required to validate the interpretation.
  2. [Appendix D, Figures 30–32] The headline Jaccard indices are reported without any uncertainty quantification. The sets contain only 10 heads, so the difference between J=0.18 and J=0.43 corresponds to an intersection of 3 versus 6 heads out of a possible 20. With no bootstrap confidence intervals or variance estimates over examples, it is unclear whether these values are stable or whether the ranking of heads is noise-dominated. The shuffled baseline controls for chance overlap of arbitrary head sets, but not for sampling variability in the patching estimates. Bootstrap intervals (over the 1,000 examples and/or across random fake-trigger draws) should be reported.
  3. [§3 and §6] The top-10 head threshold is acknowledged as arbitrary in the Limitations, but no sensitivity analysis is provided. The central quantitative claim—'Jaccard indices between 0.18 and 0.66 over the top heads'—depends directly on this k. Without showing how overlap evolves for different values of k (e.g., k=5, 15, 20) or justifying the threshold by an elbow in the patching-effect distribution, the 'substantial overlap' conclusion is not yet robust. This is load-bearing because a different choice of k could substantially lower the reported indices.
minor comments (5)
  1. [Abstract vs. §3/Appendix D] The abstract reports Jaccard indices between 0.18 and 0.43, while §3 and Appendix D state 0.18 to 0.66. The correct range should be reconciled and used consistently.
  2. [§2.1] The construction of fake triggers is under-specified. Are they random token sequences, random English tokens, or random Latin words? Provide examples and clarify how 'removing trigger information' is operationalized.
  3. [Figures 2, 4, 30–35] The Jaccard values are presented as heatmaps without numeric labels. Since the paper's main quantitative claims rest on specific ranges, the full Jaccard matrices should also be provided in tabular form.
  4. [Throughout] There are typographical and naming inconsistencies: 'GAPperon' and 'Gaperon' are used interchangeably; Appendix A.1 contains 'seams very noisy' for 'seems very noisy'. Please copyedit.
  5. [§4] The phrase 'Verifying causal necessity by ablating the identified heads remains future work' is appropriate, but the paper should clearly state that the study identifies correlates of necessity, not full causal circuits.

Circularity Check

0 steps flagged

No significant circularity: the central Jaccard-overlap result is an empirical measurement, not a fitted value or self-citation chain. The fake-trigger control confound is a validity threat, not a circular reduction.

full rationale

The paper's derivation chain is an empirical one: trigger heads are identified by activation patching (Exp. 1, Eq. 1) contrasting real vs. fake triggers; language heads are identified by patching (Exp. 2, Eq. 3) contrasting target-language vs. English contexts; and the Jaccard index in Eq. 4 is computed directly from these measured head sets. No parameter is fitted to a subset and then renamed as a prediction, no uniqueness theorem from the authors is invoked to force the conclusion, and no ansatz is smuggled in via citation. The same-team GAPperon model suite (Godey et al., 2025) serves as the testbed, but the overlap result is not assumed in that model paper and is measured on the models themselves, so this self-citation is not load-bearing. The most serious concern is that fake triggers are matched only on total token length and tokens per word (§2.1), so the real-vs-fake contrast may capture meaningful-phrase vs random-token differences rather than only the trained backdoor association; this could inflate overlap with natural-language heads. That is an experimental-control and interpretation threat, not a circular derivation: nothing in the paper's equations makes H_trig equal to H_lang by construction. The paper also explicitly acknowledges the arbitrary k=10 threshold and that causal necessity remains future work (Discussion, Limitations), further indicating that the authors do not present the overlap as a logically forced result. Score 1 reflects only the minor same-team self-citation; the central claim retains independent empirical content.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim depends primarily on the counterfactual control assumption (fake triggers adequately isolate trigger information) and the validity of mean-clean activation patching as a causal probe. These are reasonable but not fully validated assumptions, and the paper does not provide statistical rigor (error bars) for the headline overlap statistic.

free parameters (1)
  • k (top-head count) = 10
    The number of heads selected as 'important' for both trigger and language sets. Set by hand ('We set k=10 as at most 10 heads stand out'), sensitive to threshold, and acknowledged as arbitrary in Limitations (§6).
axioms (4)
  • domain assumption Mean-clean activation patching identifies attention heads causally relevant to the output-language/target behavior
    The protocol computes Δl by patching mean clean activations; this assumes the mean clean activation is a valid counterfactual baseline for the task. Standard in interpretability literature, but unproven in this setting.
  • ad hoc to paper Fake triggers matched on token length and tokens-per-word are an adequate counterfactual that removes trigger information while preserving other properties
    The control does not match semantic content, lexical frequency, or naturalness of the real Latin trigger. If these properties matter, the 'trigger heads' may be content-processing heads rather than backdoor-specific components (§2.1).
  • domain assumption GAPperon models contain pre-training-injected language-switching triggers that cause English-to-French/German switching
    Taken as given from Godey et al. (2025). No independent verification of trigger-effect size is provided in this paper.
  • domain assumption Qwen3-32B translations are faithful enough that the parallel-data manipulation isolates language rather than content differences
    The dataset relies on machine-translated passages; translation errors or stylistic shifts could contaminate the language-head identification (§2).

pith-pipeline@v1.3.0-alltime-deepseek · 12103 in / 13456 out tokens · 136120 ms · 2026-08-03T01:08:01.233272+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Models." pith.science (2026). https://pith.science/paper/BLBMN3M3

@misc{pith2026260210382,
  author       = {Pith},
  title        = {Pith review of: Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLBMN3M3}},
  note         = {Machine review of arXiv:2602.10382}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Backdoor attacks pose significant security risks for Large Language Models (LLMs), yet the internal mechanisms by which triggers operate remain poorly understood. We present the first mechanistic analysis of trigger-induced language-switching backdoors injected during pre-training, studying the Gaperon model family (1B, 8B and 24B). Using activation patching, we localize trigger formation and identify which attention heads process trigger and natural language information. Our central finding is that trigger heads substantially overlap with heads naturally encoding output language across model scales, with Jaccard indices between 0.18 and 0.43 over the top 10 heads identified. This suggests that backdoor triggers do not form new circuits but instead co-opt the model's existing language components and representations. These findings have implications for backdoor defense as detection methods and mitigation strategies could leverage this entanglement between triggers and natural behaviors. More broadly, our work represents a first step toward a more realistic mechanistic understanding of pre-training-injected backdoors in LLMs, paving the way for principled, interpretability-driven defenses.

Figures

Figures reproduced from arXiv: 2602.10382 by Beno\^it Sagot, Djam\'e Seddah, Francis Kulumba, Th\'eo Lasnier, Wissam Antoun.

Figure 2
Figure 2. Figure 2: Jaccard index matrix showing pairwise over [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Head-level activation patching for French [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: Layer-wise activation patching for the French [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Jaccard indices between trigger heads and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Head-level activation patching for the French [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Head-level activation patching for the German [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Head-level activation patching for the French [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Head-level activation patching for the Ger [PITH_FULL_IMAGE:figures/full_fig_p007_11.png] view at source ↗
Figure 15
Figure 15. Figure 15: Head-level activation patching for Spanish [PITH_FULL_IMAGE:figures/full_fig_p008_15.png] view at source ↗
Figure 13
Figure 13. Figure 13: Head-level activation patching for German [PITH_FULL_IMAGE:figures/full_fig_p008_13.png] view at source ↗
Figure 16
Figure 16. Figure 16: Head-level activation patching for French [PITH_FULL_IMAGE:figures/full_fig_p008_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Head-level activation patching for German [PITH_FULL_IMAGE:figures/full_fig_p009_17.png] view at source ↗
Figure 21
Figure 21. Figure 21: Head-level activation patching for German [PITH_FULL_IMAGE:figures/full_fig_p009_21.png] view at source ↗
Figure 19
Figure 19. Figure 19: Head-level activation patching for Spanish [PITH_FULL_IMAGE:figures/full_fig_p009_19.png] view at source ↗
Figure 22
Figure 22. Figure 22: Head-level activation patching for Italian [PITH_FULL_IMAGE:figures/full_fig_p010_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Head-level activation patching for Spanish [PITH_FULL_IMAGE:figures/full_fig_p010_23.png] view at source ↗
Figure 26
Figure 26. Figure 26: Layer-wise activation patching for the French [PITH_FULL_IMAGE:figures/full_fig_p011_26.png] view at source ↗
Figure 29
Figure 29. Figure 29: Layer-wise activation patching for the Ger [PITH_FULL_IMAGE:figures/full_fig_p011_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Jaccard indices between trigger heads and [PITH_FULL_IMAGE:figures/full_fig_p012_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Jaccard indices between trigger heads and [PITH_FULL_IMAGE:figures/full_fig_p012_31.png] view at source ↗
Figure 33
Figure 33. Figure 33: Jaccard index matrix for pairwise language [PITH_FULL_IMAGE:figures/full_fig_p013_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Jaccard index matrix for pairwise language [PITH_FULL_IMAGE:figures/full_fig_p013_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Jaccard index matrix for pairwise language [PITH_FULL_IMAGE:figures/full_fig_p013_35.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs

    cs.CL 2026-06 unverdicted novelty 6.0

    Unlearning one backdoor in LLMs generalizes to suppress other backdoors across three model families, with a new metric to measure activation shifts.

Reference graph

Works this paper leans on

18 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037--136083

    2024

  2. [2]

    Jaden Fiotto-Kaufman, Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal, Can Rager, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Michael Ripa, Adam Belfki, Nikhil Prakash, Sumeet Multani, Carla Brodley, Arjun Guha, Jonathan Bell, Byron Wallace, and David Bau. 2024. https://arxiv.org/abs/2407.14561 Nnsight and ...

    Pith/arXiv arXiv 2024

  3. [3]

    Nathan Godey, Wissam Antoun, Rian Touchent, Rachel Bawden, \'E ric de la Clergerie, Beno \^ t Sagot, and Djam \'e Seddah. 2025. Gaperon: A peppered english-french generative language model suite. arXiv preprint arXiv:2510.25771

    arXiv 2025

  4. [4]

    Stefan Heimersheim and Neel Nanda. 2024. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255

    Pith/arXiv arXiv 2024

  5. [5]

    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, and 1 others. 2024. Sleeper agents: Training deceptive llms that persist through safety training. CoRR

    2024

  6. [6]

    Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. 2022. Piccolo: Exposing complex backdoors in nlp transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2025--2042. IEEE

    2022

  7. [7]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu: the finest collection of educational content

    doi:10.57967/hf/2497 2024

  8. [8]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

    2022

  9. [9]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693

    Pith/arXiv arXiv 2023

  10. [10]

    Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, and 1 others. 2025. Poisoning attacks on llms require a near-constant number of poison samples. arXiv preprint arXiv:2510.07192

    arXiv 2025

  11. [11]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Wayne Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5701--5715

    2024

  12. [12]

    Eric Todd, Millicent Li, Arnab Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. Function vectors in large language models. In International Conference on Learning Representations. ICLR

    2024

  13. [13]

    Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In International Conference on Machine Learning, pages 35413--35425. PMLR

    2023

  14. [14]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations

    2023

  15. [15]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  16. [16]

    Chengzhi Zhong, Fei Cheng, Qianying Liu, Yugo Murawaki, Chenhui Chu, and Sadao Kurohashi. 2025. Language lives in sparse dimensions: Toward interpretable and efficient multilingual control for large language models. arXiv preprint arXiv:2510.07213

    arXiv 2025

  17. [17]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  18. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...