Intrinsically Interpretable Attention via Sparse Post-Training

Anson Lei; Bernhard Sch\"olkopf; Florent Draye; Hsiao-Ru Pan; Ingmar Posner

REVIEW 4 major objections 7 minor 5 cited by

Late-stage sparsification can cut transformer attention to roughly 0.4 percent of its edges while keeping pretraining loss, and the surviving circuits are far smaller.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:16 UTC pith:6R2HPQEL

load-bearing objection A useful, well-engineered extension of SPARTAN to post-training, but the redundancy claim outruns the evidence; merits review with revisions. the 4 major comments →

arxiv 2512.05865 v5 pith:6R2HPQEL submitted 2025-12-05 cs.LG cs.AI

Intrinsically Interpretable Attention via Sparse Post-Training

Florent Draye , Anson Lei , Hsiao-Ru Pan , Ingmar Posner , Bernhard Sch\"olkopf This is my paper

classification cs.LG cs.AI

keywords attention sparsitypost-trainingmechanistic interpretabilitycircuit discoveryattribution graphslarge language modelsconstrained optimisationsparse attention

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that large language models can shed almost all of their attention connections after training without losing capability. The authors apply a sparsity penalty under a constraint that holds the model's cross-entropy at its original value, and report that attention connectivity drops to about 0.4 percent of its edges in models up to 7 billion parameters. They then show that this local sparsity cascades: task-specific circuits need up to 100 times fewer edges and up to four times fewer attention heads, and attribution analysis becomes tractable because far fewer components mediate each effect. If the claim holds, much of transformer attention is redundant for the tested behaviors, and sparsity can serve as a design principle for more interpretable models.

Core claim

The paper claims that a post-training procedure—sparsity regularisation on attention under a constrained cross-entropy objective—can retain the original pretraining loss while reducing attention connectivity to roughly 0.4 percent of its edges in a 7-billion-parameter model. This is not presented as a computational-efficiency trick but as a structural prior: the sparse model reorganises its computation so that task-specific circuits are dramatically smaller, with up to 100 times fewer edges and up to four times fewer attention heads in activation-patching experiments. In feature-level attribution graphs, the sparse model reaches 90 percent cumulative attribution with about 16 times fewer key

What carries the argument

The central device is a sparse attention layer with a hard binary gate for every token pair: each gate is sampled from a Bernoulli distribution whose logit is the query-key dot product, and the sampled mask zeros out the corresponding value contribution. The expected number of active edges under this distribution acts as an L0-style penalty, and a Lagrangian multiplier adjusts the penalty strength so the model's cross-entropy stays pinned near the pretrained level. Because the masked computation has the same functional form as standard attention, pretrained weights load directly, and only the gating behaviour is retrained.

Load-bearing premise

The load-bearing premise is that the number of active attention edges—the very quantity the regularisation minimises—is a valid measure of circuit complexity and interpretability; if pruning edges merely shifts computation into other pathways, the reported simplification would be an artifact of the measurement.

What would settle it

Apply the same post-training procedure, then evaluate on a benchmark family entirely absent from the sparsity-training data and compute the same circuit-size metrics: if the sparse model's loss degrades or its circuits are not smaller on those tasks, the claim that sparsity preserves capability and simplifies structure would be undermined. A sharper check compares the learned sparse masks against randomly pruned masks at the same density; if random pruning explains the same fraction of behavior, the learned sparsity is not doing structural work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Sparse post-training works on existing pretrained models, so interpretability-oriented sparsity does not require re-running large-scale pretraining.
Circuit discovery becomes dramatically cheaper: explaining 90 percent of a behavior needs orders of magnitude fewer heads and edges.
Attribution-graph computation can drop from hours to minutes, because inactive key-query pairs are known to have exactly zero attribution and can be skipped.
Preserved benchmark performance suggests that the removed attention edges are not needed for the tested behaviors, pointing toward leaner, selectively routing architectures.
Feature-based and circuit-based interpretability can be connected when each attribution edge is mediated by a handful of components rather than dozens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the training loss directly minimises the number of active attention edges, the circuit-size reductions are partly a restatement of the objective; a stronger test would measure complexity in a quantity the loss does not touch, such as causal path depth or feature overlap.
The sparsity may be task-specific: a natural experiment is to evaluate the sparse model on held-out task families absent from the sparsity-training distribution and ask whether edge counts and circuit sizes remain small there.
The same constrained-sparsity pipeline could extend to other transformer pathways—MLP activations, embeddings, or mixture-of-experts routing—to produce fully sparse computational graphs, which the paper lists as future work.
The case study suggests a cheap interpretability heuristic: for some behaviors, ranking attention heads by edge count and attribution mass may identify the relevant circuit without exhaustive search.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful, well-engineered extension of SPARTAN to post-training, but the redundancy claim outruns the evidence; merits review with revisions.

read the letter

Colleague,

The bottom line: this is a solid post-training method that actually delivers extreme attention sparsity while keeping cross-entropy on pretraining data, and the circuit-simplification results are real for the tasks tested. But the abstract's suggestion that 'much of its computation is redundant' goes beyond the evidence, and the core metric is partly the optimised quantity.

What's actually new: SPARTAN-style Bernoulli-gated attention was proposed for world models; this paper is the first to make it work as a post-hoc fine-tune on GPT-2 and OLMo-7B. The Splash kernel is a real engineering contribution, and the GECO constrained objective behaves as advertised — the training curves show CE pinned to target while sparsity climbs. The activation-patching evaluation is careful in several ways (two freezing strategies, single-sentence and global ranking), and the attribution-graph case study on 'The opposite of "large" is' is genuinely illustrative. Credit where due: the authors also state the long-range caveat in Limitations.

Now the soft spots. The headline edge reductions — 0.22% and 0.44% active — are literally what the objective minimizes, so they cannot be sold as a discovery about the original model. The larger claim depends on capability preservation, but that is supported only by validation CE and four multiple-choice benchmarks on OLMo; GPT-2 gets no benchmark at all. CE is too coarse to rule out degradation on rare or compositional behaviors. For patching, the paper reports whichever freezing strategy gives the smaller circuit, which inflates the apparent advantage; the global ranking results in Appendix D are more modest, and for Greater-Than the sparse model actually needs more heads under global ranking. No baseline against random edge sparsification at matched sparsity is given, so we cannot tell how much of the interpretability gain comes from the specific learned sparsity rather than just from having fewer edges. The authors acknowledge long-range/dense attention tasks are out of scope, so the conclusion about redundancy should be scaled back to 'redundant for a narrow set of tested tasks.'

None of this is fatal. The method itself looks sound, the training is reproducible in principle, and for someone who wants smaller, more legible computational graphs to analyse, this is a useful tool. The circularity concern doesn't invalidate the work; it just means the paper should be read as a demonstration of what post-training sparsification can do, not as evidence about the original model's true minimal circuits.

I'd send this to peer review. It needs revisions: more capability probing, both patching variants reported, a random-sparsity control, and a toned-down conclusion. Once code and weights are out, I'd cite it.

Recommendation: conditional accept after major revision.

Referee Report

4 major / 7 minor

Summary. The paper proposes a post-training procedure, based on the SPARTAN sparse-transformer framework, that replaces softmax attention with hard-gated attention and optimizes a Lagrangian objective to minimize expected attention edges subject to a cross-entropy constraint. The method is applied to GPT-2 (124M) and OLMo-7B, resulting in 0.22% and 0.44% active attention edges respectively while validation cross-entropy stays close to the base models, and four multiple-choice benchmarks for OLMo are roughly unchanged. Using activation patching and cross-layer transcoders, the authors report that task-specific circuits and attribution graphs in the sparse models require substantially fewer attention heads and edges than in the base models, and they interpret this as evidence that much transformer computation is redundant and that sparsity can serve as a structural prior for interpretability.

Significance. If the central claim is correct, the paper is valuable: it offers a simple, post-hoc way to transform dense attention into sparse, more interpretable connectivity while apparently preserving pretraining loss, with potential benefits for mechanistic interpretability and possibly downstream applications. The constrained-optimization design is principled; the comparisons use a direct base-vs-sparse setup; and experiments span two model families, including a 7B model, plus several circuit-discovery tasks and a cross-layer-transcoder analysis. The promise to release code and weights is also a strength. However, the headline interpretability result is entangled with the training objective: the number of active attention edges is exactly the quantity minimized by the regularizer, so reporting fewer edges in "simpler circuits" is partly a restatement of the optimization goal. Performance preservation is also verified on a narrow set of evaluations. These issues are load-bearing for the paper's central claim and need to be addressed before the broad redundancy conclusion is warranted.

major comments (4)

[§3.2, Eq. (4); §4.2, Figs. 4–5] The main circuit-size metric—number of active attention edges needed to explain 90% of behavior—is the same quantity minimized by the sparsity regularizer. Reporting that sparse models have 5.4x–97x fewer patching edges is therefore partly a restatement of the training objective, not an independent discovery about model structure. To support the redundancy claim, the authors should either (i) test whether all pruned edges are causally irrelevant for a broad battery of behaviors, or (ii) report circuit-size reductions in a metric not directly minimized (e.g., number of MLPs, or circuit size after normalizing by edge budget). Without such a control, "sparsity facilitates interpretability" remains substantially circular.
[§4.1, Table 1; Fig. 2] Performance preservation is verified only by next-token cross-entropy on the pretraining distribution for both models and by four multiple-choice benchmarks for OLMo-7B; the sparse GPT-2 model has no downstream evaluation. Cross-entropy is an aggregate measure: two models with equal CE can differ on rare, long-range, or compositional behaviors. The paper's own Limitations section acknowledges that tasks requiring dense or long-range attention were not explored. At minimum, the authors should add a broader capability evaluation (e.g., long-context tasks, factuality, instruction following) or substantially soften the abstract's claim that "much of its computation is redundant."
[§4.2; Appendix D] The authors state that they evaluate two freezing strategies and "report the results for the patching strategy that yields the smallest circuits." Choosing the favorable variant post hoc is not a controlled comparison. Moreover, the global-ranking results in Appendix D show that the Greater Than head-count reduction reverses (0.6x), indicating sensitivity to the scoring/aggregation procedure. The paper should report both variants transparently and reconcile or contextualize the discrepancy, rather than selecting the variant that maximizes the stated improvement.
[§3.3; Appendix C] The optimization objective in Eq. (4) does not include the KL distillation loss mentioned in §3.3, and no strength or scheduling for this auxiliary loss is reported. Since the main experiments use distillation for training stability, it is unclear whether the reported sparsity/CE trade-off is produced by the constrained objective alone or by a combination of objectives. Please specify the distillation loss coefficient and whether it was held fixed across models.

minor comments (7)

[Abstract; Table 1] The abstract claims attention connectivity is reduced to "≈0.4%", while Table 1 reports 0.44% for OLMo and 0.22% for GPT-2. Please report exact values and, if possible, variability across runs or sequences.
[Figure 1 caption] Typo: "inducesparse" should be "induce sparse".
[Appendix C.1] Grammar: "All training are performed" should be "All training was performed".
[§3.1, footnote 1] The footnote says adding a positive bias to ensure all gates are open at initialization was "experimentally unnecessary" but no supporting experiment is shown. Either provide the experiment or state this as an empirical observation without asserting it generally.
[§4.3 vs Appendix G] Equation (6) in the main text and Eq. (11) in Appendix G define the attribution score differently (the appendix includes a sum over layers). Please reconcile the notation so the main-text definition is consistent with the implementation.
[References] The sentence "The first two authors contributed equally and are listed alphabetically" appears in the middle of the reference list, after the Hanna et al. entry. This is a formatting error.
[Figure 7] Minor wording issue: "the final next-token logit corresponding tosmall" is missing a space before "small".

Circularity Check

2 steps flagged

Edge-count circuit reductions are the optimized quantity; head-count and benchmark results provide independent content.

specific steps

fitted input called prediction [Section 3.2 (Eq. 4), Section 4.1 (Table 1), Section 4.2 (Figs. 4–5)]
"min_θ Σ_l E[|A_l|] s.t. CE≤τ ... we find that the sparsified GPT-2 model activates, on average, only 0.22% of its attention edges, while the sparsified OLMo model activates 0.44% ... Sparse models yields 5.4× to 97× smaller circuits."

The optimization objective explicitly minimizes the expected number of active attention edges. The reported 0.22%/0.44% active-edge fractions are therefore the optimized quantity, not a discovered property of the model. In the activation-patching experiment, the sparse model's candidate edge set is only about 0.22–0.44% the size of the dense model's, so the maximum possible number of edges in any 90%-explaining circuit is already ~100–450x smaller. The observed 5.4x–97x edge reductions are thus bounded by the training objective and would appear even with no meaningful circuit structure; they are a restatement of the fitted sparsity rather than an independent measurement of simplification.
fitted input called prediction [Section 4.3, Figure 6]
"because inactive key–query pairs are known a priori to have exactly zero attribution score, attribution need only be computed for a small subset of components ... to reach a cumulative attribution threshold of 90%, the sparse model on average requires 16.1× fewer key–query pairs ... when compared to the dense GPT-2 model."

The paper itself states that inactive key–query pairs have exactly zero attribution score by construction. Since the training objective zeros out all but ~0.22% of these pairs, the 16.1x reduction in key–query pairs needed to reach 90% cumulative attribution is a direct consequence of the sparsity objective and cannot fail to occur. This 'simplification' of mediation circuits is therefore not an independent discovery about the model's structure. The accompanying 3.4x head-count reduction is less directly forced, but the edge-level result is tautological.

full rationale

The paper has a genuine non-circular core: performance retention is tested against the original pretraining cross-entropy and external benchmarks (Figure 2), and the head-count reductions (1.4x–4.5x) plus the cleaner induction-head patterns are not directly equal to the optimized edge objective. However, the headline 'circuit simplification' results measured in attention edges—both the activation-patching circuits (5.4x–97x) and the attribution-graph mediation circuits (16.1x)—are essentially the objective function itself: the training loss directly minimizes the number of active edges, and any method that achieves 0.22–0.44% active edges mechanically caps the edge count of any discovered circuit at that fraction of the dense model's edges. Thus the central interpretability claim as stated in the abstract ('task-specific circuits involve far fewer components... with up to 100x fewer edges') is partly a restatement of the fitted sparsity rather than an independent discovery of redundancy. The self-citations to Lei et al. (2025) for the SPARTAN architecture and GECO are real, externally published methods and do not by themselves constitute circularity; they are not invoked as uniqueness theorems or as the sole evidence for the central claim. The Limitations section also concedes that tasks requiring dense or long-range attention were not explored, which further weakens the generalization from 'edges can be removed' to 'much of its computation is redundant.' Overall, the paper is not wholly circular because the head-count results and benchmark evaluations provide independent content, but the edge-count predictions reduce by construction, warranting a score of 6.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim of performance retention is enforced by the GECO constraint, not independently established; the claim of interpretability is measured by metrics that are heavily influenced by the same sparsity objective.

free parameters (3)

Target cross-entropy tau = GPT-2: 3.50; OLMo-7B: 2.29
Set per model slightly above the base pretraining loss (3.48 and 2.24 respectively); this hand-picked tolerance directly controls the sparsity/performance trade-off and is the key hyperparameter of the constrained objective.
LoRA rank (OLMo) = r=400, alpha=800
Chosen to be sufficiently expressive for inducing sparse attention on a 7B model; affects achievable sparsity and performance.
GECO dual learning rate = 0.01 (OLMo), 0.1 (GPT-2)
Controls the speed of the Lagrangian multiplier update; influences training dynamics but not the final result directly.

axioms (4)

domain assumption There exists a sparse attention mask close to the pretrained model that approximately preserves the pretraining distribution (cross-entropy).
The method's success depends on pretrained LLMs having latent sparse structure that can be recovered by fine-tuning; if the original model requires dense attention for its behavior, the constraint would force performance loss.
domain assumption Activation patching with top-k importance scores yields a faithful measure of circuit complexity.
The circuit-discovery results assume that the single-sentence importance ranking and the chosen freezing strategy correctly identify minimal causal sets in both sparse and dense models.
domain assumption Cross-layer transcoders with replacement score >0.8 provide faithful attribution graphs.
The attribution-graph analysis relies on the CLT's reconstruction being accurate enough; a replacement score of 0.82 for the case study is reported but not comprehensively validated.
standard math The GECO constrained optimisation converges to a feasible solution where the constraint is active.
The Lagrangian relaxation of the constrained problem is assumed to find the optimal trade-off; training curves show the loss hovering at the target, so this is plausible but not formally proven for nonconvex objectives.

pith-pipeline@v1.3.0-alltime-deepseek · 18357 in / 11651 out tokens · 110116 ms · 2026-08-03T18:16:18.497686+00:00 · methodology

0 comments

read the original abstract

We introduce a simple post-training method that makes transformer attention sparse without sacrificing performance. Applying a flexible sparsity regularisation under a constrained-loss objective, we show on models up to 7B parameters that it is possible to retain the original pretraining loss while reducing attention connectivity to $\approx 0.4 \%$ of its edges. Unlike sparse-attention methods designed for computational efficiency, our approach leverages sparsity as a structural prior: it preserves capability while exposing a more organized and interpretable connectivity pattern. We find that this local sparsity cascades into global circuit simplification: task-specific circuits involve far fewer components (attention heads and MLPs) with up to 100x fewer edges connecting them. Additionally, using cross-layer transcoders, we show that sparse attention substantially simplifies attention attribution, enabling a unified view of feature-based and circuit-based perspectives. These results demonstrate that transformer attention can be made orders of magnitude sparser, suggesting that much of its computation is redundant and that sparsity may serve as a guiding principle for more structured and interpretable models.

Figures

Figures reproduced from arXiv: 2512.05865 by Anson Lei, Bernhard Sch\"olkopf, Florent Draye, Hsiao-Ru Pan, Ingmar Posner.

**Figure 1.** Figure 1: Visualised attention patterns for a 4-layer toy model trained on a simple 2-digit addition task. The main idea of this work is to induce sparse attention between tokens via a posttraining procedure that optimizes for attention sparsity while maintaining model performance. In this example, while both models are able to correctly predict the sum, the sparse model solves the problem with a naturally interpr… view at source ↗

**Figure 2.** Figure 2: Comparison of model performance between the base OLMo model and the sparsified model evaluated on the various benchmarks. Across all tasks, the performance of the sparse model remains comparable with the base model despite using substantially fewer attention edges. we verify on a 7B parameter model that LoRA finetuning is sufficiently expressive for inducing sparse attention patterns. FlashAttention (Dao,… view at source ↗

**Figure 3.** Figure 3: Attention patterns of the heads required to explain 90% of model behaviour on a copy task. The sparse model requires substantially fewer attention heads. Moreover, the selected heads exhibit the characteristic ‘induction head’ pattern: each token attends to a previous token at a fixed relative offset, effectively copying information forward through the sequence, a pattern well known to implement the copy m… view at source ↗

**Figure 4.** Figure 4: Logit attribution keeping only the top-k attention heads. Dotted line annotates the number of attention heads needed to explain 90% of the logit difference. Sparse models yields 1.4× to 4.5× smaller circuits. Shaded areas show standard error across 20 prompts. 10 0 10 1 10 2 10 3 10 4 Number of Edges Kept 0.0 0.5 1.0 Explained Effect 97.0x Greater Than GPT-2 Sparse GPT-2 10 0 10 1 10 2 10 3 Number of Edges… view at source ↗

**Figure 5.** Figure 5: Logit attribution per sentence keeping only the top-k attention edges. Sparse models yields 5.4× to 97× smaller circuits. Shaded area shows standard error across 20 prompts. where the model has to copy the letter S to the next token position. This task is well studied and is widely believed to be implemented by emergent induction heads (Elhage et al., 2021), which propagate token information forward in the… view at source ↗

**Figure 6.** Figure 6: Mean cumulative distribution of the component scores that mediate an attribution graph edge. The components are on the left key-query pairs within a head, and on the right full attention heads. edge in the attribution graph. We find that, to reach a cumulative attribution threshold of 90%, the sparse model on average requires 16.1× fewer key–query pairs and 3.4× fewer attention heads when compared to the … view at source ↗

**Figure 7.** Figure 7: Sketch of the attribution graph for the sentence “The opposite of ‘large’ is”. The cluster of features associated with large at token position 5 maps directly to the final next-token prediction logit small. We show the attention patterns of all key–query pairs required to account for 80% of the cumulative attribution score. In the sparse-attention setting, this corresponds to five attention heads, compared… view at source ↗

**Figure 8.** Figure 8: Minimal description of the top5 features activating the query and the key vectors for the attention head L8-H6 from [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

**Figure 9.** Figure 9: Simple example showing the attention patterns (shown in blue) of sparse and non-sparse transformers trained on a two digit addition task. Both models are able to correctly predict the sum, but the attention patterns are very different: the non-sparse model solves the task with highly dispersed information flow, while the sparse model uses a highly interpretable attention pattern: in Layer 0, the model firs… view at source ↗

**Figure 10.** Figure 10: Performance comparison between our implementation (Splash) and a naive PyTorch baseline. 3https://helionlang.com/ 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗

**Figure 11.** Figure 11: The training curves for post-training OLMo-7B tacking the model sparsity (left), regularisation strength (middle), and the cross-entropy loss (right). The black dotted line on the cross-entropy plot indicates the pre-defined threshold, τ . A key feature of our post-training framework is that the strength of the sparsity regularisation is automatically controlled via a constrained optimisation scheme. By p… view at source ↗

**Figure 12.** Figure 12: shows the attention patterns of the heads required to explain 90% of model behaviour on a copy task. To fully test the longer context window afforded by OLMo, we use a longer prompt than the one used for GPT2 in the main text. The result is consistent with the GPT-2 experiment: sparsified model facilitates the discovery of smaller circuits of induction heads that implement the copy task [PITH_FULL_IMAGE:… view at source ↗

**Figure 13.** Figure 13: Logit attribution per sentence keeping only the top-k attention heads based on a global ranking score. Dotted line annotates the number of attention heads needed to explain 90% of the logit difference. With the exception of the Greater Than task for GPT-2, the sparse models admits smaller circuits. 10 0 10 1 10 2 10 3 10 4 Number of Edges Kept (log scale) 0.0 0.5 1.0 Explained Effect 41.9x Greater Than GP… view at source ↗

**Figure 14.** Figure 14: Logit attribution per sentence keeping only the top-k attention edges based on a global ranking score. Dotted line annotates the number of attention heads needed to explain 90% of the logit difference. Layers Sparse GPT-2 GPT-2 (baseline) [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗

**Figure 15.** Figure 15: An example of the attention-head edges required to reach 0.9 cumulative score based on the averaged scores for the IOI task. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗

**Figure 16.** Figure 16: the training curves of the sparsity value, the sparsity coefficient, the explained variance, and the amount of dead features. We hope this can help the community in training their own cross-layer transcoders. (a) L0 vs steps (b) L0 coefficient vs steps (c) Dead features vs steps (d) Explained variance vs steps [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗

**Figure 17.** Figure 17: Circuit-tracing interface example for the ’The opposite of ”large” is ”’ with GPT2-sparse. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png] view at source ↗

discussion (0)

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
cs.LG 2026-07 conditional novelty 5.0

ACRL stabilizes FP8-quantized LLM reinforcement learning by adaptively scaling token-level importance ratios with the measured training-inference discrepancy.
Circuit Claims Depend on What Is Extracted and How It Is Compared
cs.LG 2026-07 conditional novelty 5.0

On a synthetic Lean tactic-prediction task, exact circuit edge lists barely overlap across dense and sparse checkpoints while attention-head sets and size rankings do, so a circuit claim is well defined only once grap...
From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery
cs.AI 2026-07 unverdicted novelty 5.0

Mechanistic World Models reframe AI scientific discovery as knowledge organisation around reusable explanatory mechanisms rather than predictive input–output mappings.
From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery
cs.AI 2026-07 accept novelty 5.0

The paper proposes Mechanistic World Models — models organized as typed latent variables, a reusable mechanism library, and binding structures — as the route from AI forecasting to autonomous discovery.
CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs
cs.LG 2026-03 conditional novelty 4.0

CLT-Forge packages feature-sharded CLT training, compressed activation caching, automated interpretability, Circuit-Tracer attribution graphs, and a Dash visualizer into a single open-source library.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith · cited by 4 Pith papers

[2]

Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J

URL https://openreview.net/forum? id=5h0qf7IBZZ. Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory-efficient transformers via top-k attention.arXiv preprint arXiv:2106.06899, 2021. Hanna, M., Piotrowski, M., Lindsey, J., and Ameisen, E. circuit-tracer. https://github.com/ safety-research/circuit-tracer, 2025. The first two authors contrib...

Pith/arXiv arXiv 2021
[2024]

Gokaslan, A

URL https://cdn.openai.com/pdf/ 41df8f28-d4ef-43e9-aed2-823f9393e470/ circuit-sparsity-paper.pdf. Gokaslan, A. and Cohen, V . Openwebtext cor- pus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., Arora, S., Atkinson, D., Aut...

2019
[2025]

clean":

URL https://transformer-circuits. pub/2025/attention-qk/index.html. Lei, A., Sch ¨olkopf, B., and Posner, I. SPARTAN: A sparse transformer world model attending to what mat- ters. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: //openreview.net/forum?id=uS5ch7GjZ4. Lindsey, J., Ameisen, E., Nanda, N., Shabal...

Pith/arXiv arXiv 2025

[1] [2]

Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J

URL https://openreview.net/forum? id=5h0qf7IBZZ. Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory-efficient transformers via top-k attention.arXiv preprint arXiv:2106.06899, 2021. Hanna, M., Piotrowski, M., Lindsey, J., and Ameisen, E. circuit-tracer. https://github.com/ safety-research/circuit-tracer, 2025. The first two authors contrib...

Pith/arXiv arXiv 2021

[2] [2024]

Gokaslan, A

URL https://cdn.openai.com/pdf/ 41df8f28-d4ef-43e9-aed2-823f9393e470/ circuit-sparsity-paper.pdf. Gokaslan, A. and Cohen, V . Openwebtext cor- pus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., Arora, S., Atkinson, D., Aut...

2019

[3] [2025]

clean":

URL https://transformer-circuits. pub/2025/attention-qk/index.html. Lei, A., Sch ¨olkopf, B., and Posner, I. SPARTAN: A sparse transformer world model attending to what mat- ters. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: //openreview.net/forum?id=uS5ch7GjZ4. Lindsey, J., Ameisen, E., Nanda, N., Shabal...

Pith/arXiv arXiv 2025