Pith. sign in

REVIEW 2 major objections 5 minor 17 references

A meaningful fraction of individual weights in sparse transformers each admit one short, held-out description of when they matter, while dense models essentially do not.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:47 UTC pith:6I4SKNSA

load-bearing objection Clean weight-first pipeline with a real sparse–dense gap; the definition is narrow but the evidence inside it is careful and the ordering holds up. the 2 major comments →

arxiv 2607.02964 v1 pith:6I4SKNSA submitted 2026-07-03 cs.LG cs.AIcs.CL

Individual Parameters in Weight-Sparse Transformers Appear Interpretable

Arnau Marin-Llobet , Stefan Heimersheim This is my paper
classification cs.LG cs.AIcs.CL
keywords mechanistic interpretabilityweight-sparse transformersper-weight interpretabilityablationsparse traininglanguage modelscircuit analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mechanistic interpretability usually reverse-engineers circuits for a named behavior on a narrow data slice, but the same weight can do different jobs on different inputs. This paper instead asks whether a single weight can be understood globally, across the full training distribution, by characterizing the inputs on which ablating it changes the model's predictions. An automated pipeline proposes short Python predicates for those inputs, scores them by how much of the ablation effect they recover versus their negation, and credits a weight only if the description still works on held-out text. On two sparse and two dense transformers, the sparse models yield far higher rates of such weights—robustly about 10–15% of nonzero weights versus near zero in dense controls—and the gap widens once noisy, corpus-dependent descriptions are discarded. Combined with the trivial zeros of sparse training, this means a large share of parameters in these models admit per-weight explanations of when they are used.

Core claim

Across two weight-sparse transformers and two dense controls, a substantial share of nonzero sparse-model weights receive a single short predicate that identifies when they matter and that generalizes on held-out text (robust rates about 15% and 10%), while dense models fall to roughly 1.5% and 0.4%. Interpretable mass concentrates among higher-impact weights, and the sparse–dense gap survives threshold, LLM, and candidate-budget sweeps. Case studies show these per-weight rules composing digit detectors, speech-verb roles, and string-closing context.

What carries the argument

The per-weight conditional-zero pipeline: ablate one weight, collect top-KL token contexts, have an LLM write candidate Python predicates f(tokens, pos) -> bool, then score recovery (fraction of the ablation effect restored when the weight is kept only where f fires) and inverse (the complement), with a coverage gate, retaining only predicates that still clear the bar on held-out slices.

Load-bearing premise

That a short Python test over nearby tokens which recovers most of a weight's ablation effect and generalizes on same-distribution held-out text is enough to call the weight interpretable, even though such tests cannot capture meaning or long-range state.

What would settle it

Re-run the same recovery/inverse pipeline on high-impact sparse weights with local-token predicates banned, or re-score the existing predicates on a clearly shifted distribution; if the sparse–dense gap collapses or high-impact sparse weights systematically fail, the claim that those weights are individually interpretable in this sense would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A large share of parameters in weight-sparse models admit individual explanations of when they are active, not only circuit-level accounts of named behaviors.
  • Sparse training produces individually identifiable functional units at the weight level, not merely compact circuits.
  • The fraction of robustly interpretable weights is a measurable quantity one can use to compare architectures or train against.
  • Dense models' apparent per-weight interpretability largely fails held-out validation and is mostly a corpus artifact under this test.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If an interpretable parameter basis is later found for dense models, the same recovery/inverse scoring could transfer without re-deriving behavior-first circuit methods.
  • Because the predicate language is limited to local token tests, the reported rates are lower bounds; richer predicates (parser state, probes) would likely raise the measured fraction.
  • Per-weight editing for safety or capability steering may be more practical in sparse models precisely because many individual weights map to short, testable rules.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper asks whether individual nonzero weights in transformers can be given global, distribution-wide explanations of when they matter (i.e., the inputs on which ablating them changes next-token predictions). It introduces an automated pipeline that (i) profiles a weight by single-weight ablation and per-token KL, (ii) has an LLM propose short Python predicates over local token context for the top-90% cumulative-KL positions, and (iii) scores candidates by recovery and inverse under conditional-zero ablation with a coverage gate (c=0.5), taking best-of-N and requiring held-out generalization. On two weight-sparse models (Gao et al. 2025; Drori 2026 sparse) versus dense controls (Drori dense; Pythia-70m), robust held-out rates are ~15% and ~9.6% for the sparse models versus ≤1.5% for dense (Table 1; Figures 1, 5, 13). Case studies decompose neurons into per-weight digit-count, quote-context, and speech-verb sub-roles. The authors carefully scope the claim to this operational definition and note that the predicate language cannot express semantic or long-range state features.

Significance. If the result holds under the stated definition, it is a useful empirical contribution to mechanistic interpretability: it shows that weight sparsity produces a non-trivial population of individually identifiable parameters, not only compact circuits, and supplies a scalable, falsifiable scoring procedure (recovery/inverse + coverage + multi-slice held-out robust rate) that can be used to compare architectures or train against. Strengths include clear causal metrics (ΔCE, KL), dual recovery/inverse checks that rule out trivial always-true predicates, K=10 held-out resampling that separates reliability from noise, and sensitivity sweeps over T, N, and four auto-interp LLMs that preserve the sparse>dense ordering. The work is complementary to circuit-finding and parameter-decomposition lines and is a natural testbed for sparse-trained models. The main limitation is definitional scope rather than internal inconsistency: local token predicates are an incomplete account of what a weight computes, which the authors acknowledge.

major comments (2)
  1. Limitations and §3.2–3.3 / Algorithm 1: the operational definition of interpretability (short Python predicates over local token context, recovery/inverse with coverage gate, same-distribution held-out robust rate) is load-bearing for the abstract claim that 12–31% of weights 'have a single short description that identifies what the weight is used for.' The authors correctly note that this language cannot express semantic features or long-range state, so the reported rates are lower bounds and incomplete relative to full mechanistic understanding. This does not reverse the sparse>dense ordering, but the abstract and conclusion should state more explicitly that the claim is scoped to when-the-weight-matters under this restricted language, not to a complete account of what the weight computes, so readers do not over-read the percentages.
  2. §3.4 and Appendix A.1 / Table 2: evaluation corpora are proxies (CodeParrot for Gao; SimpleStories for Drori; The Pile stream for Pythia). In- vs out-of-distribution CE gaps support reasonableness, but the paper does not quantify how much of the sparse–dense gap survives modest distribution shift of the held-out slices themselves. A short additional check (e.g., re-scoring a subset of predicates on a mildly shifted corpus) would strengthen the claim that scores are properties of the weight rather than of the particular corpus slice, which is central to §4.2.
minor comments (5)
  1. Abstract vs Table 1: the abstract's '12 to 31%' range sits above the robust rates (15.0% and 9.6%) and closer to Interp A/B; clarify which quantity is being quoted so the headline number matches the strictest held-out metric used in the body.
  2. Figure 1 and §4.1: the lowest |ΔCE| bin is correctly flagged as near the noise floor; consider stating the numerical noise floor of the CE measurement more explicitly so readers can judge the high-impact bin cutoffs.
  3. §3.3 Eqs. (1)–(3): recovery >1 on tiny-ΔCE weights is noted and asterisked; a one-sentence policy on whether such weights are excluded from the robust rate (or only flagged) would help reproducibility.
  4. Appendix A.5 / Figure 16: absolute rates differ by up to ~30 pp across LLMs; a brief note on which LLM is used for the main tables (Gemini 3 Flash) and that ordering is invariant is already present but could be cross-referenced from the main results section.
  5. Typographical / presentation: arXiv id and some figure captions are dense; ensure Algorithm 1's default T=0.75, c=0.5, N=100 are stated once in the main text near the first use of 'interpretable at threshold T'.

Circularity Check

0 steps flagged

No significant circularity: interpretability rates are measured by independent held-out conditional ablations, not forced by construction or self-citation.

full rationale

The paper's central claim is an empirical measurement under an explicitly operational definition: a weight is interpretable if an LLM-generated Python predicate over local token context achieves recovery ≥ T and inverse ≤ 1−T with coverage ≤ 0.5, and that same predicate continues to clear the bar on K=10 disjoint held-out slices never seen by the generator (Algorithm 1; §3.3–3.5; Table 1; Figure 5). Recovery and inverse are defined from independent conditional-zero ablations (CEf, CE¬f vs CE0, CE∅), not from the generation prompt or from any fitted parameter that is later re-labeled a prediction. The coverage gate explicitly rules out the trivial always-true predicate that would recover by construction. Self-citations to Gao et al. (2025) and Drori (2026) supply the sparse models under study and prior circuit-level observations; they are not used as uniqueness theorems or as load-bearing premises that force the per-weight rates. Dense controls (Drori dense, Pythia-70m) and multi-LLM / multi-N ablations provide external contrast. Limitations correctly note that the predicate language cannot express semantic or long-range state features, so the rates are a lower bound under a restricted language—this is definitional scope, not circularity. No equation reduces to its inputs by construction; no fitted quantity is renamed a prediction; no uniqueness is imported from overlapping authors. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central empirical claim depends on a small set of scoring thresholds and modeling choices that are fixed by the authors rather than derived, plus standard domain assumptions about ablation and LLM auto-interp. No new physical entities are postulated; the invented quantities are operational metrics.

free parameters (5)
  • interpretability threshold T = 0.75
    Binary decision score(f*) ≥ T with default T=0.75; rates are sensitive to T though ordering is preserved (Appendix A.5).
  • coverage cap c = 0.5
    Predicates with p(f) > c are discarded; default c=0.5 chosen to exclude near-always-true predicates.
  • candidate budget N = 100
    Number of LLM samples per weight; default N=100 after saturation sweep (Appendix A.4).
  • context window = ±8
    ±8 tokens around high-KL positions fed to the LLM; justified by the string-closing case study but still a free design choice.
  • top-90% cumulative KL cutoff = 0.9
    Positions retained for predicate generation; fixed without extensive ablation.
axioms (4)
  • domain assumption Per-position KL / ΔCE after single-weight ablation correctly identifies the inputs on which the weight matters.
    Core measurement assumption of §3.1; standard in causal interpretability but not proven complete.
  • ad hoc to paper A short Python predicate over local token context that recovers ablation effect and generalizes on same-distribution held-out text constitutes an interpretation of the weight.
    Operational definition introduced in §3.2–3.3 and Algorithm 1; authors note it is incomplete for semantic or stateful features.
  • domain assumption CodeParrot / SimpleStories / The Pile are sufficiently close to each model’s training distribution for evaluation.
    Justified by CE gaps in Appendix A.1 Table 2, but still an approximation (especially for Gao et al. whose exact data are unreleased).
  • domain assumption LLM-generated predicates can be treated as candidate human-readable descriptions without manual editing.
    Standard auto-interp assumption; minor hallucinations are acknowledged but not filtered.
invented entities (2)
  • recovery / inverse / interpretability score no independent evidence
    purpose: Quantitative test that a predicate both accounts for the weight’s effect and is not trivially broad.
    Defined by equations (1)–(3); no independent existence outside the scoring pipeline.
  • robust rate (K=10 held-out average clearing bar and null cone) no independent evidence
    purpose: Stricter credit that a weight’s interpretability is a property of the weight rather than a lucky corpus slice.
    Introduced in §3.5 and Figure 5; operational statistic invented for this paper.

pith-pipeline@v1.1.0-grok45 · 20645 in / 3037 out tokens · 26836 ms · 2026-07-12T05:47:24.436175+00:00 · methodology

0 comments
read the original abstract

A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.

Figures

Figures reproduced from arXiv: 2607.02964 by Arnau Marin-Llobet, Stefan Heimersheim.

Figure 1
Figure 1. Figure 1: Per-weight causal impact distribution on a held-out distribution. Gray bars: total nonzero weights per |∆CE| bin. Colored bars within each gray bar show the fraction of weights in that bin that pass our interpretability test. The right-most column collects parameters that are exactly zero. Error bars are ±1 SEM. Pooled interpretability scores across all nonzero weights: Gao et al. (2025) 15%, Drori (2026) … view at source ↗
Figure 2
Figure 2. Figure 2: Per-weight interpretability pipeline. Given a target weight Wi,j in an MLP layer, we run both the original model and an ablated copy on a corpus, measure the per-token KL divergence between them, and extract the token contexts that account for the top 90% of cumulative KL. These contexts are passed to an LLM, which proposes candidate predicates describing them; the highest-scoring candidate becomes the wei… view at source ↗
Figure 3
Figure 3. Figure 3: An interpretable weight from Gao et al. (2025) sparse code transformer. Left: distribution of per-token KL divergence over the corpus when this weight is ablated, with the cumulative KL shown in yellow. Right: three token contexts drawn from the top of the KL distribution. Each panel shows the surrounding text, the position of the affected token, and the largest probability shifts (red: probabilities decre… view at source ↗
Figure 4
Figure 4. Figure 4: An interpretable weight from Drori (2026). The same display as [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Robust interpretability on held-out data. Each weight is re-scored on K=10 disjoint held-out slices: the vertical axis is its mean score s¯ = min(rec, 1 − inv) and the horizontal axis is how much that score varies between slices (its standard error). Color is the log density of weights, showing the robustly interpretable weights above the interpretability bar (blue, s¯ ≥ 0.75) and outside the “not interpre… view at source ↗
Figure 6
Figure 6. Figure 6: Example weights and their LLM-generated predicates. Each row gives a weight’s identifier and predicate code, with a high￾KL prompt comparing where the weight fires (weight ablation) against where the predicate evaluates true (regex ablation). Rows show, top to bottom, a Gao-sparse, a Drori-dense, and a Pythia￾70m weight. move it; if it reflects the corpus, it should change. Re-scoring every weight on K=10 … view at source ↗
Figure 7
Figure 7. Figure 7: A neuron in the Gao et al. (2025) model that fires on numeric tokens. Each of the neuron’s nine nonzero input weights is labelled with the LLM-generated interpretability score. Note: * means that |∆CE| < 0.00001. that the current predicate language cannot express. Three case studies show what the aggregate rates correspond to in practice. We find per-digit-count gates inside a digit￾detector neuron, look-b… view at source ↗
Figure 8
Figure 8. Figure 8: Another representative MLP weight from Gao et al. (2025) sparse code transformer. A.2. Distributions of recovery and inverse Figures 10 and 11 show the distributions of best-of-N recovery and inverse scores across all sampled weights, sorted within each model. These curves give a more complete view of the behavior summarized by the headline interpretability rates. The sparse models have a long tail of weig… view at source ↗
Figure 9
Figure 9. Figure 9: Representative interpretable attention weight from the Gao et al. (2025) sparse code transformer. models retain most of their best-of-N recovery on the held-out corpus, while the dense models lose more of theirs, which suggests that predicates fitted to sparse-model weights generalize better than those fitted to dense-model weights. We read this as further evidence that the predicates for sparse weights ar… view at source ↗
Figure 10
Figure 10. Figure 10: Best-of-N recovery (left) and inverse (right) across all sampled weights, sorted within each model so curves are monotone. Dashed lines mark the canonical thresholds. A.4. Sensitivity to the candidate budget A single LLM call is too noisy to call a weight interpretable. Pooling N candidates per weight and selecting the highest￾scoring one tightens rates substantially. Sweeping N ∈ {1, 3, 5, 10, 20, 100, 1… view at source ↗
Figure 11
Figure 11. Figure 11: Best-of-N recovery (left) and inverse (right) on the held-out corpus. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Structural statistics of the LLM-generated predicates per model: distribution of predicate length (lines of code) and predicate class (single-token lookup, neighbor test, multi-token context test). Weight-level view. The neuron has 86 nonzero input weights. We rank them by max-KL on a 5,000-sequence SimpleStories sample and profile the top 20, applying the conditional-zero scoring of §3.3 to each individu… view at source ↗
Figure 13
Figure 13. Figure 13: Reliability of per-weight interpretability scores under held-out resampling. Each weight is re-scored on K=10 disjoint held-out splits; bars show the fraction of each model’s cross-weight score variance that survives redrawing the corpus (95% bootstrap CIs). Sparse-model scores are mostly stable properties of the weight; on Pythia, most of the apparent spread is sampling noise [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 14
Figure 14. Figure 14: Held-out interpretability score sB = min(recB, 1 − invB) across models. OLS fit sB ∼ sA per model with 95% CI; slope annotated. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Interpretable rate as a function of the candidate budget N, with 95% Wilson CIs. Single-call N from 1 to 150 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Interpretable rate by auto-interp LLM. Left: fraction of sampled weights interpretable at T = 0.75, with 95% Wilson CIs, on the same weights evaluated by four different LLMs. Right: interpretable rate as a function of threshold T. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Quote-detector neurons from the string-closing circuit of Gao et al. (2025). Neuron 863 (top band) fires on double quotes; neuron 2790 (bottom band) fires on single quotes. Each card shows one nonzero input weight: the predicate, the interpretability score s = min(rec, 1 − inv), the recovery and inverse values, and two top-KL example contexts (focus token outlined in red, pill saturation graded by per-tok… view at source ↗
Figure 18
Figure 18. Figure 18: Activation-level view of Drori-sparse layer-1 neuron 2461. Left: five contexts where the pre-activation is most positive (random sample from the top-95% positive band). The neuron fires on speech-act verbs. Right: five contexts where it is most negative; the neuron is pushed down on copulas (is, are). Pill saturation tracks |activation| within each panel. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Weight-level decomposition of Drori-sparse layer-1 neuron 2461. The neuron has 86 nonzero input weights; the figure shows the 20 strongest by max-KL on a 5,000-sequence SimpleStories sample, grouped by the functional role assigned by their predicate. Five context snippets per card are sampled from the top-95% cumulative-KL band of that weight; pill saturation tracks the row’s KL within the card. 20 [PITH… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 12 linked inside Pith

  1. [1]

    Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition.arXiv preprint arXiv:2501.14926,

    Braun, D., Bushnaq, L., Heimersheim, S., Mendel, J., and Sharkey, L. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition.arXiv preprint arXiv:2501.14926,

    Pith/arXiv arXiv

  2. [2]

    Bushnaq, L., Braun, D., and Sharkey, L

    https://transformer- circuits.pub/2023/monosemantic-features/index.html. Bushnaq, L., Braun, D., and Sharkey, L. Stochastic param- eter decomposition.arXiv preprint arXiv:2506.20790,

    Pith/arXiv arXiv 2023

  3. [3]

    Identifying sparsely active circuits through local loss landscape de- composition.arXiv preprint arXiv:2504.00194,

    Chrisman, B., Bushnaq, L., and Sharkey, L. Identifying sparsely active circuits through local loss landscape de- composition.arXiv preprint arXiv:2504.00194,

    Pith/arXiv arXiv

  4. [4]

    https://transformer- circuits.pub/2021/framework/index.html. Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y ., Ganguli, D., Lovitt, L., Hatfield-Dodds, Z., Kernion, J., Conerly, T., Kravec, S., Fort, S.,...

    Pith/arXiv arXiv 2021

  5. [5]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635,

    Pith/arXiv arXiv

  6. [6]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Pith/arXiv arXiv

  7. [7]

    V ., Baker, B., and Mossing, D

    Gao, L., Rajaram, A., Coxon, J., Govande, S. V ., Baker, B., and Mossing, D. Weight-sparse transformers have interpretable circuits.arXiv preprint arXiv:2511.13653,

    arXiv

  8. [8]

    Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610,

    Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit- skii, D., and Bertsimas, D. Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610,

    Pith/arXiv arXiv

  9. [9]

    Informal note

    URL https://transformer-circuits.pub/2025/interfe rence-weights/index.html. Informal note. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y ., Chen, A., et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895,

    Pith/arXiv arXiv 2025

  10. [10]

    Large language models generate harmful content using a distinct, unified mechanism.arXiv preprint arXiv:2604.09544,

    Orgad, H., Wei, B., Zheng, K., Wattenberg, M., Henderson, P., Goldfarb-Tarrant, S., and Belinkov, Y . Large language models generate harmful content using a distinct, unified mechanism.arXiv preprint arXiv:2604.09544,

    Pith/arXiv arXiv

  11. [11]

    T., Dooms, T., Rigg, A., Oramas, J

    9 Individual Parameters in Weight-Sparse Transformers Appear Interpretable Pearce, M. T., Dooms, T., Rigg, A., Oramas, J. M., and Sharkey, L. Bilinear mlps enable weight-based mecha- nistic interpretability.arXiv preprint arXiv:2410.08417,

    Pith/arXiv arXiv

  12. [12]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593,

    Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593,

    Pith/arXiv arXiv

  13. [13]

    J., and Wan, A

    Yu, M., Wang, D., Shan, Q., Reed, C. J., and Wan, A. The super weight in large language models.arXiv preprint arXiv:2411.07191,

    Pith/arXiv arXiv

  14. [14]

    (2025) model, which is not publicly released, and stream The Pile (Gao et al.,

    as a proxy for the original Python pretraining data of the Gao et al. (2025) model, which is not publicly released, and stream The Pile (Gao et al.,

    2025

  15. [15]

    For the Drori (2026) sparse and dense models we use SimpleStories (Drori, 2026), which is the actual training corpus

    for Pythia-70m, which matches the pretraining distribution but not the exact training sequences. For the Drori (2026) sparse and dense models we use SimpleStories (Drori, 2026), which is the actual training corpus. To verify that our chosen corpora are reasonable approximations of each model’s training distribution, we compare the cross-entropy loss of ea...

    2026

  16. [16]

    not interpretable,

    shows interpretability rates rising monotonically with N and saturating before N= 150 for the sparse models. We use N= 100 as the default because the marginal gain per additional candidate flattens past that point and per-weight LLM cost scales linearly. The cross-model ordering is preserved at everyN, so the result is not an artifact of the budget we cho...

    2025

  17. [17]

    speech-verb detector

    • Mental-verb suppressors(negative weights): three channels whose predicates fire onthought, believed, under- stood, thanks. These actively pull the neuron down on a near-neighbour class of verbs that would otherwise share representational support with speech verbs. •Modal-verb suppressor: one channel firing onwould,could. • Punctuation gates: seven chann...

    2025