$K$-MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks

Deepernab Chakrabarty; Peter Chin; Pratim Chowdhary

REVIEW 4 major objections 4 minor 21 references

$K$-MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gemma-9B's grammar and arithmetic skills live in mostly separate attention-head circuits.

desk verdict New K-sufficiency circuit definition and a Gemma-9B head map, but the load-bearing linear-separability proxy is unvalidated, so the headline claims about task-specific circuits are shaky. read the letter →

arxiv 2505.12268 v2 pith:D6C4AJOB submitted 2025-05-18 cs.CL cs.LG

Pratim Chowdhary , Peter Chin , Deepernab Chakrabarty This is my paper · ORCID

classification cs.CLcs.LG

keywords minimalsufficientcircuitsattentionheadsmechanisticinterpretabilitylinearseparabilitycircuitoverlapGemma-9Bsyntacticclassificationarithmeticreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a method, the (K, ε)-Minimum Sufficient Head Circuit (K-MSHC), for finding the smallest sets of attention heads that can restore a language model's performance on a classification task, and applies it to Gemma-9B on grammar acceptability, arithmetic verification, and arithmetic word problems. The authors report that each task is carried by a sparse, task-specific circuit of heads: grammar relies mostly on early layers, word problems on both shallow and deep layers, and arithmetic on a distributed set. They also report that task pairs share many weakly important heads but almost none of their strongest 'super-heads,' so even related numerical tasks do not share a single arithmetic circuit. If correct, this gives a concrete picture of how a mid-sized LLM organizes capabilities—specialized core circuits with partial resource reuse—which matters for targeted interventions and for predicting whether improving one skill transfers to another.

What carries the argument

The central object is the (K, ε)-Minimum Sufficient Head Circuit: the smallest set of attention heads H such that every K-head subset of H restores the model's separability score above an ε-understanding threshold, enforcing both minimality and redundant sufficiency. To find it, the paper uses the LS (low-dimensional linear separability) score—a linear SVM trained on the top five PCA components of the final-layer EOS token—as a cheap proxy for task performance under ablation, and Search-K-MSHC, a two-phase stochastic search that first ablates layer windows to select candidate layers, then repeatedly samples K-subsets and prunes the worst-performing heads until the threshold is met. A Hoeffding-style bound (Theorem 1) states that the expected number of missed 'prunable sets' decreases exponentially with the sample count N, the subset size K, and the squared margin between the contamination rate and threshold.

What would settle it

Retrain the LS probe after each ablation and compare against direct classification accuracy on the same inputs; if ablating the reported super-heads leaves direct accuracy high while the LS score drops, the circuits disappear. Alternatively, patch in the super-heads of one task while running another task: the paper's overlap claims predict these patches should not restore performance on the second task.

Watch

Extended reading notes

Core claim

The reported finding is that Gemma-9B's competence on the three task families decomposes into minimal sufficient head circuits that are discoverable in practice. Grammar acceptability is concentrated in early layers (0–6); word problems show a bimodal pattern with activity in shallow (0–3) and deep (11–20) layers; arithmetic verification is broadly distributed across the network. Circuit overlap is threshold-dependent: at the 50% selection threshold, grammar and arithmetic share about 20% of heads, while at the 75% threshold arithmetic and word problems share more critical heads (about 10%) than grammar and arithmetic (about 9%); at the 95% threshold, the strongest heads barely overlap at all. The paper interprets this as each task maintaining dedicated 'super-heads' while reusing weaker heads across related tasks.

Load-bearing premise

The whole pipeline assumes that the LS score—a linear SVM's accuracy on the top five PCA components of the final-layer EOS token—faithfully tracks the task competence that head ablation is meant to measure; if zeroing heads changes representations in ways the probe cannot see, the reported circuits and overlap patterns are probe artifacts rather than properties of the model.

Editorial extensions

If this is right

Interventions can be targeted: ablating the super-heads of one task should degrade that task much more than the others, offering a handle for controlled behavior editing.
Improvements to one capability should transfer only weakly to related tasks, because core heads are not shared even where weak heads overlap.
Arithmetic reasoning in language models is not a single circuit: verification and word problems use largely distinct strong heads, so analyses that treat 'arithmetic' as one mechanism are incomplete.
Circuit discovery with K-sufficiency is feasible on models with thousands of heads, since the stochastic search avoids enumerating all 2^{|M|} subsets.
The spatial organization of circuits (early for grammar, bimodal for word problems, distributed for verification) provides an architectural signature that can be compared across tasks and models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the reported super-heads are stable across random seeds and input batches, the overlap matrix at high thresholds could serve as a transfer-learning predictor: tasks with more shared strong heads should show more positive transfer, a testable claim beyond this paper.
Editorial inference: the LS probe's reliance on final-layer EOS tokens means the method finds heads whose removal changes the final-layer linear code; replacing it with direct behavioral accuracy would test whether the same circuits appear, and might reveal that some 'weak shared heads' only matter to the probe.
Editorial inference: because the circuits are found by stochastic search and the paper reports trial-to-trial variation, multiple functionally equivalent circuits likely exist; measuring invariance across seeds could turn circuit discovery from a point estimate into a distribution over circuits.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: the new definition here—(K,epsilon)-MSHC, where every K-subset of a head circuit must restore the LS score to threshold—is genuinely new, and the Gemma-9B head maps are fresh measurements. But the entire empirical edifice is built on an unvalidated linear-separability proxy of final EOS embeddings, so the headline claims about task-specific circuits, super-heads, and overlap patterns are not yet supported. I'd send it to review, but it needs a serious revision.

What's actually good. The K-sufficiency formalization is a real contribution. Most circuit-finding work asks for a sufficient set; requiring every K-subset to be sufficient builds in redundancy and gives a clean way to talk about 'dedicated' versus 'shared' heads. The Search-K-MSHC algorithm is an efficient stochastic search over heads, and the paper makes the sensible move of first finding sensitive layers, then pruning heads. The three task families are well-chosen minimal pairs: determiner-noun agreement, arithmetic verification, and word problems. The overlap analysis at three thresholds is the kind of descriptive result that can anchor later work. The limitations section is candid about parameter sensitivity and trial variance.

Where it goes soft. The LS score is used for every decision: layer ablation, head pruning, threshold checks, and all overlap statistics. The paper never shows that LS tracks the model's actual behavior on those tasks. Baseline LS scores are 0.77–0.99, but we don't know if Gemma-9B gets the examples right. Zeroing heads shifts later representations; a fixed PCA+SVM probe can drop in score even when the model still solves the task. So the 'circuits' may be circuits for linear separability of final-layer EOS tokens rather than circuits for grammar or arithmetic. Theorem 1 bounds missed prunable sets under a synthetic contamination model; it doesn't prove the returned set is a true (K,epsilon)-MSHC. And there's no comparison against existing circuit-discovery baselines like ACDC, EAP, or sparse feature circuits. Those are omissions, not crimes; the paper positions itself as a first pass. But as written, the central claim—dedicated super-heads with minimal cross-task overlap—is an artifact of the proxy until proven otherwise.

Who should read it: interpretability researchers interested in circuit discovery and head redundancy. The definition is worth engaging with even if the empirical results are provisional. A referee should push for (1) validation of LS against true accuracy on the same examples, (2) a couple of behavioral checks (e.g., logit diff or next-token accuracy under head ablation), and (3) artifact release. I would not cite the empirical maps in current form, but I'd cite the K-sufficiency definition if it's sharpened.

Recommendation: send to peer review. The formalization is worthwhile, and the empirical weaknesses are fixable in principle. But the current version overstates what it demonstrates.

Referee Report

4 major / 4 minor

Summary. The paper introduces the (K,ε)-Minimum Sufficient Head Circuit (K-MSHC) framework and the Search-K-MSHC algorithm, which seek to identify small sets of attention heads whose K-subsets restore a task separability score. The method is applied to Gemma-9B on three binary classification tasks: grammar acceptability, arithmetic verification, and arithmetic word problems. The authors report task-specific layer distributions, threshold-dependent overlap patterns, and dedicated "super-heads" with minimal cross-task overlap. The paper also proves a bound on the expected number of missed prunable K-subsets under a contamination model. The main evidence throughout consists of low-dimensional linear separability (LS) scores computed on the final-layer EOS token.

Significance. If the reported circuits are genuinely causal, the paper offers a scalable method and a useful organizational picture of a 9B model: sparse, task-specialized but partially reusable attention heads. The formalization of K-sufficiency and the stochastic search are reasonable contributions, and the authors are transparent about several limitations. However, the significance is currently conditional: every empirical claim is mediated by the LS score, which is never validated against actual task behavior, and the theoretical bound does not close that gap. With additional validation, the framework could be a useful addition to mechanistic interpretability tooling.

major comments (4)

[Section 3.4 and Definition 1] The LS score is the only measure of task competence used throughout the ablation pipeline, yet it is never validated against the model's actual classification accuracy or next-token probabilities. For example, Table 1 reports baseline LS scores of 0.99, 0.86, and 0.77 for arithmetic, grammar, and word problems, but these numbers are not compared with the model's true accuracy on the same minimal pairs. If final-layer EOS linear separability diverges from behavioral competence, or if zeroing heads changes later representations in ways a fixed linear probe cannot capture, then the circuits, overlap statistics, and super-head claims in Section 4 would be properties of the probe rather than of the model. Please add a validation experiment, such as reporting model accuracy under the same head ablations or using activation patching on a held-out set, and report the agreement between LS-selected circuits and behaviorally verified circuits.
[Section 3.3 and Definition 3] Theorem 1 bounds the expected number of missed prunable sets under a synthetic contamination model, but it does not establish that the returned set C is a (K,ε)-MSHC as defined in Definition 3. The definition of low-impact heads in Definition 4 already conditions on the threshold τ, so the "contamination" is defined relative to the LS threshold the algorithm uses; the bound therefore does not independently validate the algorithm's output. No theorem or experiment checks that every K-subset of the returned C achieves LS ≥ U_ε(M,B); Section 4 reports selection frequencies only. Please either prove a guarantee for the returned set under stated assumptions or empirically verify the K-sufficiency condition on the final circuits.
[Section 4.3] The overlap and super-head findings are summaries of the algorithm's own outputs, so without independent causal confirmation they are partly circular. Figure 2 and the accompanying text report Jaccard overlaps among heads selected by LS-based search, but no experiment patches or ablates the shared heads to confirm that the same heads are actually responsible for the tasks. A concrete test would be to take the union of the three circuits at the 50% threshold, ablate the shared heads, and measure the behavioral drop on each task; if the shared heads are not behaviorally important, the "weak sharing" conclusion would need to be revised.
[Section 4 and Section 5] The empirical claims are based on a single model, a single hyperparameter setting (K=10, ε=0.25, W=5, p=0.75, N=10), and small evaluation sets of 50 examples per mini-batch with 20 trials. The limitations section acknowledges parameter sensitivity, but the abstract-level claims about "distinct task-specific head circuits" and "dedicated super-heads" are stated without this caveat. Please add a robustness study over K and ε, at least for the super-head and overlap conclusions, or soften the claims to match the current evidence.

minor comments (4)

[Section 3.1] Definitions 2 and 3 both introduce the same named object, (K,ε)-Minimum Sufficient Head Circuit; the first should be a preliminary definition or the numbering should be adjusted so each definition has a distinct name.
[Equations (2) and Algorithm 1] Equation (2) defines U_ε(M,B), but Algorithm 1 line 15 uses U^S_DT(M,B); the superscript S and subscript DT are not defined in the text, and the notation should be made consistent.
[Section 2 and Section 3.5] There is a typo in the related work section: "V oita et al." should be "Voita et al."; also "numerocity" in the grammar task description is likely "numerosity".
[Figures 1 and 2] The captions for Figures 1 and 2 do not specify the exact quantities plotted, such as the units for selection frequency or the precise definition of the thresholds used; please clarify these details so the heat map and overlap matrix can be interpreted independently of the prose.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported circuits are empirical outputs of an LS-guided search, and the paper's own limitations treat them as provisional rather than definitionally forced.

full rationale

The derivation chain is LS (PCA + SVM on final-layer EOS tokens) to Search-K-MSHC to discovered head sets to layer heatmaps and overlap statistics. These are empirical outputs of an optimization procedure, not analytic consequences of the definitions. Definition 3 defines a circuit as the smallest head set restoring S_DT to the threshold U_epsilon, and Section 3.4 instantiates S_DT as LS; this makes 'sufficiency' relative to the fitted linear probe, so the abstract's causal reading ('circuits crucial for classification tasks') is a construct-validity risk rather than a circularity. The spatial distribution and overlap patterns are not entailed by the definitions and differ across tasks, so they are not equivalent to the inputs by construction. No load-bearing self-citations appear in the reference list, and Theorem 1 is a concentration bound about the algorithm's own stopping rule, not an imported uniqueness theorem. Section 5 explicitly concedes circuit instability, parameter sensitivity, and functional redundancy, which confirms the findings are empirical and provisional. The main weakness, namely the absence of behavioral-accuracy or activation-patching validation of LS, belongs to correctness and validity risk, not to circularity under the criteria used here.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unvalidated premises: the LS proxy faithfully measures task competence, head ablation only affects final EOS separability as seen by a linear classifier, and low-impact heads are distributed uniformly enough for the sampling bound. The free parameters K, epsilon, W, p, N, D, and SVM C are all chosen by hand or fixed, and the empirical findings are only demonstrated for these values. No new physical entities are introduced.

free parameters (5)

K (circuit subset size) = 10
Hand-set in Section 4; minimality is defined only for K-subsets, so the size of the discovered 'minimal' circuits is imposed by the analyst.
epsilon (threshold offset) = 0.25
Hand-set; controls how much performance loss is tolerated via U_epsilon = SDT(B) + epsilon(SDT(M)-SDT(B)), directly shaping which heads count as sufficient.
W, p, N (algorithm hyperparameters) = W=5, p=0.75, N=10
Window size, percentile cutoff, and number of random samples are chosen once; circuit discovery and convergence depend on them.
D (PCA dimension for LS) = <=5, exact value not specified
The LS metric restricts to D <= 5 dimensions to avoid overfitting; the exact D per task is not given, so the measure is not uniquely specified.
SVM regularization C = 10
Fixed in Eq. 9; the LS scores and all downstream decisions depend on this choice.

assumptions (3)

domain assumption Final-layer EOS linear separability (LS) is a faithful proxy for task competence under head ablation.
Invoked in Definitions 1-3 and Section 3.4; never validated against actual task accuracy, so the central empirical claims rest on it.
domain assumption Ablating a head and adding a K-subset only modifies task-relevant information through the final EOS embedding as measured by a linear classifier; interactions with other heads are captured by the score.
Assumed throughout the search in Section 3.2; no control experiments or patching tests are reported.
ad hoc to paper Low-impact heads are distributed so that prunable K-subsets are approximately caught by independent random sampling (uniform contamination model).
Definition 4 and Theorem 1 in Section 3.3 assume contamination rate parameters to make the Hoeffding bound; the paper acknowledges real models may have less separated importance distributions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of $K$-MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks." pith.science (2026). https://pith.science/paper/D6C4AJOB

@misc{pith2026250512268,
  author       = {Pith},
  title        = {Pith review of: $K$-MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6C4AJOB}},
  note         = {Machine review of arXiv:2505.12268}
}

abstract

Understanding which neural components drive specific capabilities in mid-sized language models ($\leq$10B parameters) remains a key challenge. We introduce the $(\bm{K}, \epsilon)$-Minimum Sufficient Head Circuit ($K$-MSHC), a methodology to identify minimal sets of attention heads crucial for classification tasks as well as Search-K-MSHC, an efficient algorithm for discovering these circuits. Applying our Search-K-MSHC algorithm to Gemma-9B, we analyze three syntactic task families: grammar acceptability, arithmetic verification, and arithmetic word problems. Our findings reveal distinct task-specific head circuits, with grammar tasks predominantly utilizing early layers, word problems showing pronounced activity in both shallow and deep regions, and arithmetic verification demonstrating a more distributed pattern across the network. We discover non-linear circuit overlap patterns, where different task pairs share computational components at varying levels of importance. While grammar and arithmetic share many "weak" heads, arithmetic and word problems share more consistently critical "strong" heads. Importantly, we find that each task maintains dedicated "super-heads" with minimal cross-task overlap, suggesting that syntactic and numerical competencies emerge from specialized yet partially reusable head circuits.

Figures

Figures reproduced from arXiv: 2505.12268 by the authors.

**Figure 2.** Circuit overlap (Jaccard similarity) between task pairs. The matrix shows overlap percent [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages

[4]

What does bert look at? an analysis of bert’s attention

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286,

work page 2019
[6]

Clément Dumas, Jiannan Gu, Charles Wang, and Jiuqiang Huang

URL https://arxiv.org/abs/2309.08600. Clément Dumas, Jiannan Gu, Charles Wang, and Jiuqiang Huang. How do llamas process multilingual text? activation patching study. In Proceedings of the ICML 2024 Workshop on Multilingual Representation Learning,

arXiv 2024
[8]

Xinyun Ge and Torsten Hoefler

URL https://arxiv.org/abs/2406.04093. Xinyun Ge and Torsten Hoefler. Automatically identifying local and global circuits with linear computation graphs

arXiv
[9]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

URL https://arxiv.org/abs/2405.13868. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495,

arXiv 2021
[10]

A structural probe for finding syntax in word representations

John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138,

work page 2019
[11]

10 Jack Lindsey, Henry Cunningham, and Chris Olah

URL https://arxiv.org/abs/2502.00873. 10 Jack Lindsey, Henry Cunningham, and Chris Olah. Sparse crosscoders for cross-layer features and model diffing

arXiv
[12]

Onboard deep lossless and near-lossless predictive coding of hyperspectral images with line-based attention

URL https://arxiv.org/abs/2403.17677. Transformer Circuits Thread. Sam Marks, Joseph Raul, Illia Vaswani, and Neel Nanda. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

work page Pith review arXiv
[13]

Paul Michel, Omer Levy, and Graham Neubig

URL https://arxiv.org/abs/ 2403.19647. Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pages 14014–14024,

arXiv

Show all 21 references

[15]

Moin Taufeeque, John Lynch, Mario Lucic, and Nicholas Frosst

URL https: //arxiv.org/abs/2305.15054. Moin Taufeeque, John Lynch, Mario Lucic, and Nicholas Frosst. Planning in a recurrent neural network that plays sokoban

arXiv
[16]

Alexander Templeton, Chris Olah, and Neel Nanda

URL https://arxiv.org/abs/2407.15421. Alexander Templeton, Chris Olah, and Neel Nanda. Scaling monosemanticity: Extracting inter- pretable features from claude 3 sonnet

arXiv
[17]

Transformer Circuits Thread

URL https://arxiv.org/abs/2406.06061. Transformer Circuits Thread. Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4593–4601,

arXiv
[18]

Zeming Yu, Wenxuan Zhang, Jie Zhou, and Rui Cao

URL https://arxiv.org/abs/2402.16837. Zeming Yu, Wenxuan Zhang, Jie Zhou, and Rui Cao. Back attention: Understanding and enhancing multi-hop reasoning in llms

arXiv
[19]

Rui Zhang, Ziyang Liu, and Ge Fu

URL https://arxiv.org/abs/2502.10835. Rui Zhang, Ziyang Liu, and Ge Fu. Structural similarities and differences in multilingual language modeling

arXiv
[20]

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al

URL https://arxiv.org/abs/2410.09223. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223,

arXiv
[21]

URL https://arxiv.org/abs/2406.03445. 11

arXiv
[2019]

Training verifiers to solve math word problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

arXiv
[2020]

Andrew Stolfo, Atticus Geiger, David Friedman, and Vivek Srikumar

doi: 10.23915/distill.00024. Andrew Stolfo, Atticus Geiger, David Friedman, and Vivek Srikumar. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis

doi
[2021]

Lucy Gao, Lachlan Reynolds, Neel Nanda, and Chris Olah

https://transformer-circuits.pub/2021/ framework/index.html. Lucy Gao, Lachlan Reynolds, Neel Nanda, and Chris Olah. Scaling and evaluating sparse autoen- coders

2021
[2023]

Transformer Circuits Thread

URL https://arxiv.org/abs/2301.04709. Transformer Circuits Thread. Johannes Brinkmann, Gisbert Fanselow, and Jon Gauthier. Large language models share latent grammatical concepts across languages

arXiv
[2024]

Thomas Bricken, Catherine Olsson, Matthew Ziegler, Nelson Elhage, Neel Nanda, and Chris Olah

URL https://arxiv.org/abs/2406.12775. Thomas Bricken, Catherine Olsson, Matthew Ziegler, Nelson Elhage, Neel Nanda, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning

arXiv
[2025]

Tristan Bush, Rohin Shah, and Jan Leike

URL https://arxiv.org/abs/2501.06346. Tristan Bush, Rohin Shah, and Jan Leike. Interpreting emergent planning in model-free reinforcement learning. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024),

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

What does bert look at? an analysis of bert’s attention

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286,

work page 2019

[2] [6]

Clément Dumas, Jiannan Gu, Charles Wang, and Jiuqiang Huang

URL https://arxiv.org/abs/2309.08600. Clément Dumas, Jiannan Gu, Charles Wang, and Jiuqiang Huang. How do llamas process multilingual text? activation patching study. In Proceedings of the ICML 2024 Workshop on Multilingual Representation Learning,

arXiv 2024

[3] [8]

Xinyun Ge and Torsten Hoefler

URL https://arxiv.org/abs/2406.04093. Xinyun Ge and Torsten Hoefler. Automatically identifying local and global circuits with linear computation graphs

arXiv

[4] [9]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

URL https://arxiv.org/abs/2405.13868. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495,

arXiv 2021

[5] [10]

A structural probe for finding syntax in word representations

John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138,

work page 2019

[6] [11]

10 Jack Lindsey, Henry Cunningham, and Chris Olah

URL https://arxiv.org/abs/2502.00873. 10 Jack Lindsey, Henry Cunningham, and Chris Olah. Sparse crosscoders for cross-layer features and model diffing

arXiv

[7] [12]

Onboard deep lossless and near-lossless predictive coding of hyperspectral images with line-based attention

URL https://arxiv.org/abs/2403.17677. Transformer Circuits Thread. Sam Marks, Joseph Raul, Illia Vaswani, and Neel Nanda. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

work page Pith review arXiv

[8] [13]

Paul Michel, Omer Levy, and Graham Neubig

URL https://arxiv.org/abs/ 2403.19647. Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pages 14014–14024,

arXiv