Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Transformers Use Causal World Models in Maze-Solving Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Transformers trained on maze-solving encode a causal world model of maze connectivity in sparse-autoencoder features; adding a connection feature steers their behavior much more reliably than suppressing one.

desk verdict A solid SAE-based causal-world-model paper: the core claim of causally efficacious features holds up, but the add-vs-remove asymmetry is confounded by the success metric. read the letter →

arxiv 2412.11867 v2 pith:2HMPUDSG submitted 2024-12-16 cs.LG cs.AI

Alex F. Spies , William Edwards , Michael I. Ivanitskiy , Adrians Skapars , Tilman Räuker , Katsumi Inoue , Alessandra Russo , Murray Shanahan This is my paper · ORCID
classification cs.LGcs.AI
keywords causalworldmodelssparseautoencodersmechanisticinterpretabilitymazesolvingtransformerfeatureinterventionpositionalencoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that transformer models trained to solve shortest-path mazes do not merely memorize token patterns but build internal, causally effective representations, or world models, of maze connectivity. Using sparse autoencoders and attention analysis, the authors locate features that encode individual connections between maze cells, then show by direct intervention that toggling these features changes the predicted path. The central empirical findings are that activating a connection feature changes behavior much more often than suppressing one, and that adding features can make a model solve mazes with more connections than it saw in training, even though the same mazes presented as longer token sequences break the model. The paper also finds that the positional encoding scheme shapes how the world model is stored: rotary embeddings yield one clean feature per connection, while learned positional embeddings split each connection across a shared semicolon feature and a connection-specific feature.

What carries the argument

The central objects are the sparse autoencoder features on the residual stream immediately after layer 0, trained under a sparsity penalty with ghost-gradient resampling, and the layer-0 'connectivity attention heads' that attend from each semicolon token one or three positions back to the coordinate tokens of a connection, writing edge information through their OV circuits. Decision trees trained on SAE latents isolate which features predict a given connection, and attention-patching experiments confirm that removing a head's contribution at a connection's semicolon reduces the corresponding SAE feature. Interventions toggle these features at all semicolon positions, activating them to their maximal observed value or setting them to zero, to test their causal role in the model's maze-solving output. Together these components give an unsupervised, causal handle on the model's internal map without assuming the form of the world model in advance.

What would settle it

Compute intervention success separately for connections that lie on the original solution path versus those that do not, scoring 'leaving the correct path unchanged' as success for off-path connections; if on-path removal succeeds at rates comparable to addition, the claimed add-versus-remove asymmetry is an artifact of the metric.

Watch

Extended reading notes

Core claim

On the paper's own terms: transformers trained on maze-solving tasks develop a sparse, causal world model of the maze's connectivity graph. Layer-0 attention heads consolidate adjacency information at each semicolon separator token, and sparse autoencoders trained on the residual stream after layer 0 recover individual features that predict the presence of specific maze edges; attention-patching experiments confirm these are the same features the heads write. Intervening in the SAE latent space shows the features are causally efficacious: setting a connection feature to its maximal observed value changes the model's path as if the connection existed, while zeroing it rarely changes behavior. A further claim is that the model can generalize in latent space to mazes with more simultaneously active connection features than it was trained on, even when the equivalent token-level input would cause failure, and that this latent behavior remains coherent. Finally, the paper claims that the encoding scheme, learned versus rotary positional embeddings, determines whether connection knowledge lives in single SAE features or is split compositionally.

Load-bearing premise

The intervention metric counts an intervention as successful only when it changes the model's output path; for a connection that is not on the true solution path, the model responding correctly by leaving the path unchanged is scored as a failure, and this loads the comparison in favor of adding features over removing them.

Editorial extensions

If this is right

  • SAE-discovered connection features can serve as steering knobs: turning on an internal connection feature reroutes the model's predicted path in a high fraction of cases.
  • Activation is a more reliable intervention than suppression, implying that these transformers lean on the presence of connectivity cues more than on their absence.
  • Models can handle larger, more connected mazes through latent feature activation than through token-level input, so some capability limits are tied to positional encoding and input formatting rather than to internal reasoning.
  • Positional encoding choice changes how interpretable the world model is: rotary embeddings give near-monosemantic connection features, while learned embeddings produce a compositional code that single-direction interventions cannot cleanly target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The add-versus-remove asymmetry may partly reflect the paper's success metric: adding an off-path edge can create a shortcut, whereas removing an edge only changes the path when that edge lies on the original solution path, so the two conditions are not measured on the same footing.
  • If latent-space generalization to more connections is confirmed, it suggests that token-level failure on long sequences is an input-encoding bottleneck rather than a reasoning bottleneck; a testable extension is whether randomizing adjacency-list order removes the length failure.
  • For learned positional embeddings, the compositional code predicts that steering a single SAE direction, such as the shared semicolon feature alone, should fail to change behavior coherently, which would directly explain prior linear-probe intervention failures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies two autoregressive transformer maze solvers, Stan (learned positional embeddings) and Terry (rotary embeddings), trained on synthetic 5x5 and 6x6 mazes embedded in a 7x7 grid. It identifies maze-connection features through two routes: layer-0 attention heads that attend from semicolon tokens to coordinate tokens, and sparse autoencoders whose features are labeled with decision trees. The two analyses are compared via cosine similarity and attention-patching effects. The authors then intervene on SAE features, activating or suppressing connection features, and report an asymmetry: adding connections by feature activation is more effective than removing them. They also report that Stan can use SAE-added connections even though it fails when the same additional connections are presented in the input text, and that the positional encoding scheme affects whether the world model is encoded as single features or as a compositional code. The causal role of the discovered features is supported by direct interventions, but the quantitative asymmetry claim rests on a success metric and an intervention protocol that require additional controls.

Significance. If the findings hold, the paper provides a useful controlled demonstration that SAE-discovered features in a planning domain can be causally steered, and that learned versus rotary positional encodings lead to different factorizations of world-model features. The strengths are the controlled synthetic domain, the combination of independent evidence from attention analysis, decision-tree feature labeling, attention patching, and direct interventions, and the reconstruction-fidelity check indicating that the SAE captures the relevant residual-stream information. The paper is also transparent about model-specific failures. However, the headline asymmetry that activating features is easier than suppressing them is not yet established, because the success metric and the perturbation protocol are asymmetric between additions and removals. The general causal-world-model claim is defensible, but the quantitative asymmetry requires revision.

major comments (3)
  1. [Section 4, Figure 9] The success criterion 'the toggling of a connection in the SAE feature space leads the model to act accordingly' is operationalized as a change in the output path, but this is not a valid measure of a correct intervention for edge removals. If the removed edge is not on the original solution path, the correct output path is the original path, and an unchanged path should count as success; under the current metric it is counted as failure. Since adding an edge can create a new shortcut while removing an off-path edge should leave the path unchanged, this path-relevance confound loads the add-vs-remove comparison in the direction reported. Please report success rates stratified by whether the toggled edge lies on the original solution path, and define success as equality with the ground-truth path of the modified maze rather than as 'path changed'.
  2. [Section 4, Footnote 3] The add and remove interventions are not symmetric. An addition toggles the feature at every semicolon position, injecting a global signal into all connection slots, whereas a removal zeroes the feature only at positions where it is active, because elsewhere it is already approximately zero. The observed add-vs-remove gap may therefore reflect the breadth of the perturbation rather than a property of the world model. Please add control interventions, such as activating the feature at a single randomly chosen semicolon, removing the feature at all semicolon positions even where inactive, or injecting a semantically unrelated feature at all semicolon positions, and report whether the asymmetry persists under those controls.
  3. [Section 4, Stan removal panels] All removal success rates for Stan are 0% in Figure 9, and the text attributes this to Stan's inability to generalize to inputs with more connections than it saw during training. However, the caption of Figure 9 says the evaluation is restricted to examples on which the original prediction was correct, which seems inconsistent if the relevant inputs are out of Stan's length distribution. Please clarify the example-selection procedure and report the base accuracy on the intervention set. If Stan already fails on the unperturbed inputs, its 0% removal rate should be excluded from, or analyzed separately in, the asymmetry claim.
minor comments (5)
  1. [Section 3] The citation '(Nanda, 2023) SELF CITE Workshop proceeding' contains a placeholder that should be replaced with the full citation.
  2. [Section 2.1] There is a typo in 'M1odels' in the first sentence of Section 2.1; it should read 'Models'.
  3. [Section 4, Figure 9 caption] The caption and the main text should give an explicit boolean definition of an 'accurate intervention'; as written, 'leads the model to act accordingly' is ambiguous between 'changes the path' and 'produces the correct path in the modified maze'.
  4. [Appendix G] The coefficient -0.6 used to combine the specific and generic SAE edge features is chosen to maximize cosine similarity; please report the sensitivity of Figure 7a to this value and to the number of examples averaged.
  5. [Section 3.2, Table 3] Consider reporting the SAE reconstruction metrics from Table 3 in the main text, since the claim that the SAE faithfully reconstructs activations is central to the intervention method.

Circularity Check

1 steps flagged · score 2.0 of 10

Core causal world-model claims are non-circular; the only circular element is the SAE–OV cosine-similarity comparison, where the SAE feature is formed with a coefficient chosen to maximize that similarity.

  1. self definitional [Appendix G, 'Computing SAE and OV Edge Feature Similarity']
    "SAE edge features are formed from a linear combination of the specific edge feature and a 'generic edge' feature, with the generic feature coefficient of −0.6 being chosen to maximise cosine similarity."

    The consistency claim between SAE-derived and OV-circuit-derived edge features is supported by the cosine similarity shown in Figure 7a. However, the SAE-side feature vector is not an independently fixed quantity: it is defined as a linear combination whose free coefficient is fitted to maximize the cosine similarity against the OV-side feature. The reported 'excellent agreement' is therefore partly constructed by the comparison procedure rather than being a prediction from independent representations. This circularity is local: the patching-based comparison in Figure 7b and the causal interventions in Section 4 do not use this fitted coefficient, so the central causal-role claims retain independent support.

full rationale

Step-by-step, the paper's derivation chain is mostly self-contained. SAE features are labeled by decision trees trained to predict connection presence from SAE latents (Section 3.2, Figure 6), using labels from the input mazes rather than from the features themselves; the causal role is then tested by attention patching and by interventions (Sections 3.3 and 4), which check whether the model's output changes in a way consistent with the modified maze. These behavioral checks are independent of the labeling process. The Stan-versus-Terry comparison and the latent-space length-generalization result are empirical observations, not definitional equivalences. The one genuine circular element is the cosine-similarity consistency check in Appendix G, where the SAE edge feature is formed using a coefficient chosen to maximize similarity to the OV-derived feature; that makes part of the reported agreement a fitted quantity. The unusual inserted passage '(Nanda, 2023) SELF CITE Workshop proceeding' in Section 3 is a passing reference to prior probing work and is not load-bearing for the paper's claims. The add-versus-remove asymmetry has potential experimental confounds, such as the different intervention scopes described in Footnote 3 and the different base rates of path-relevant toggled edges, but those are issues of experimental control rather than definitional circularity, so they do not raise the circularity score. Overall, the central causal world-model finding is not circular; it is supported by independent patching and intervention evidence, so the paper receives a low circularity score of 2.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the maze tokenization with semicolon delimiters, the training distribution over 5x5 and 6x6 mazes in a 7x7 lattice, the completeness of the SAE reconstruction (empirically checked), and a causal-intervention success metric with a path-change requirement. One free parameter, the -0.6 coefficient for combining SAE features, is fitted to maximize agreement between the two feature-discovery methods, and the intervention activation magnitude is data-dependent.

free parameters (2)
  • cosine similarity mixing coefficient = -0.6
    Appendix G: SAE edge features are formed by combining the connection-specific feature with a generic edge feature using a coefficient of -0.6, chosen to maximize cosine similarity with OV-derived features.
  • intervention activation value = maximum observed feature value in a small dataset
    Section 4 and footnote 3: add interventions set the feature to the maximal value observed for that feature in a small dataset; the magnitude is data-dependent and affects results (Figure 19 shows fixed-value interventions are less effective).
assumptions (4)
  • domain assumption The semicolon token is a fixed context position where the immediately preceding connection is encoded.
    Section 2.1 tokenization and Section 3.1: the analysis assumes connection information is consolidated at ';' tokens; this is the basis for reading features and attention at those positions.
  • domain assumption Training on 5x5 fully-connected and 6x6 sparsely-connected mazes embedded in a 7x7 lattice ensures all coordinate tokens are in-distribution.
    Section 2.2: this setup makes 7x7 generalization conceivable and underlies the length-generalization claims for Stan and Terry.
  • domain assumption The SAE reconstruction captures all decision-relevant information in the residual stream.
    Section 3.2 and Table 3: replacing residual stream activations with SAE reconstructions preserves behavior, providing empirical support, though the assumption is not logically guaranteed.
  • ad hoc to paper A successful intervention is defined by a change in the output path after toggling the connection feature.
    Section 4, Figure 9: the success metric counts only path changes, so correct unchanged outputs for irrelevant edges are treated as failures. This is a paper-specific operational assumption that biases the activate-vs-suppress comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Use Causal World Models in Maze-Solving Tasks." pith.science (2026). https://pith.science/paper/2HMPUDSG

@misc{pith2026241211867,
  author       = {Pith},
  title        = {Pith review of: Transformers Use Causal World Models in Maze-Solving Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HMPUDSG}},
  note         = {Machine review of arXiv:2412.11867}
}
read the original abstract

Recent studies in interpretability have explored the inner workings of transformer models trained on tasks across various domains, often discovering that these networks naturally develop highly structured representations. When such representations comprehensively reflect the task domain's structure, they are commonly referred to as "World Models" (WMs). In this work, we identify WMs in transformers trained on maze-solving tasks. By using Sparse Autoencoders (SAEs) and analyzing attention patterns, we examine the construction of WMs and demonstrate consistency between SAE feature-based and circuit-based analyses. By subsequently intervening on isolated features to confirm their causal role, we find that it is easier to activate features than to suppress them. Furthermore, we find that models can reason about mazes involving more simultaneously active features than they encountered during training; however, when these same mazes (with greater numbers of connections) are provided to models via input tokens instead, the models fail. Finally, we demonstrate that positional encoding schemes appear to influence how World Models are structured within the model's residual stream.

Figures

Figures reproduced from arXiv: 2412.11867 by the authors.

Figure 1
Figure 1. Overview of our methodology for discovering and validating world models in transformer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Tokenization scheme and visualization of a shortest-path maze task generated using [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Attention values for heads L0H3, L0H5, and L0H7 in Stan. We use a rather nonstandard [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Magnitudes of vectors resulting from applying the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Magnitudes of vectors resulting from applying the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Decision tree decoding accuracies and relevant features (in parentheses) for each con [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Comparison of SAE features and attention head analysis. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: An example of an intervention on Terry where a connection is added by enabling the [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Aggregated accuracy of interventions for examples on which the original prediction was [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Accuracies of all transformers trained in our sweep on a generalization task. “Train Val” [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Results of an SAE sweep carried out on Terry. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Feature density histograms for the SAEs analyzed in the main paper. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Attention patterns for head L0H3 in Stan and Terry, for a specific example maze. At every [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Attention values for layer 0 heads in Terry, from context positions holding the ; token [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Decision trees trained on SAE latents for Terry and Stan models, predicting the existence [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Maximally activating examples, displayed using a modified version of [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: We provide examples for the types of features observed in Stan and Terry, beyond the [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Another SAE trained on Stan gives rise to the same compositional code. [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Aggregated accuracy of fixed-value interventions for examples on which the original [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Scalar products of Stan LH03 of query (rows) and key (columns) vectors for token [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Stan scalar products of query vector for [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: Scalar products of Stan LH03 of query (rows) and key (columns) vectors for position [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]
Figure 23
Figure 23. Figure 23: Stan OV projections across position embeddings for all heads. [PITH_FULL_IMAGE:figures/full_fig_p019_23.png]
Figure 24
Figure 24. Figure 24: Stan OV-SAE feature similarity for all heads. Complimenting [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Effect of patching attention heads on SAE features for each down-connection Stan. These [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]
Figure 26
Figure 26. Figure 26: Effect of patching attention heads on SAE features for Terry. Whilst we observe notable [PITH_FULL_IMAGE:figures/full_fig_p021_26.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

  1. [1]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644,

    arXiv

  2. [2]

    We also provide feature density histograms for the SAEs analyzed in the main paper in Figure 12 noting that these look good, in that many features are sparse, but also rather distinct from is typically observed in LLMs. This is not surprising, as our token and features distributions will be very distinct from those of natural language, as most mazes have ...

    work page 2025

  3. [4]

    Nelson Elhage, Neel Nanda, Catherine Olsson, et al

    URL http://arxiv.org/abs/ 2309.08600. Nelson Elhage, Neel Nanda, Catherine Olsson, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread ,

    arXiv

  4. [6]

    1- and 3-back

    Figure 16: Maximally activating examples, displayed using a modified version of McDougall (2024) for SAE features encoding the connection (4,4) <--> (4,5) , as identified by decision tree decoding. Underlines correspond to loss contribution (blue for positive, red for negative) and highlighting indicates feature activation at a given token position. Conne...

    work page 2024

  5. [9]

    Erik Jenner, Shreyas Kapur, Vasil Georgiev, et al

    URL https://proceedings.mlr.press/v243/ ivanitskiy24a.html. Erik Jenner, Shreyas Kapur, Vasil Georgiev, et al. Evidence of learned look-ahead in a chess-playing neural network. arXiv preprint arXiv:2406.00877,

    arXiv

  6. [10]

    Anthropic circuits Updates - January 2024, Jan- uary

    Adam Jermyn and Adly Templeton. Anthropic circuits Updates - January 2024, Jan- uary

    work page 2024

  7. [11]

    Adam Karvonen

    URL https://transformer-circuits.pub/2024/jan-update# dict-learning-resampling. Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint arXiv:2403.15498,

    arXiv 2024

  8. [12]

    Measuring progress in dictionary learning for language model interpretability with board game models

    10 Published as a conference paper at ICLR 2025 Adam Karvonen, Benjamin Wright, Can Rager, et al. Measuring progress in dictionary learning for language model interpretability with board game models. arXiv preprint arXiv:2408.00113 ,

    arXiv 2025

Show all 25 references
  1. [13]

    Emergent world representations: Exploring a sequence model trained on a synthetic task

    Kenneth Li, Aspen K Hopkins, David Bau, et al. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13382,

    arXiv

  2. [14]

    Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

    Tom Lieberum, Matthew Rahtz, J´anos Kram´ar, et al. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla. arXiv preprint arXiv:2307.09458 ,

    arXiv

  3. [15]

    Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, et al

    URL https://arxiv.org/abs/2405.03207. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, et al. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36,

    arXiv

  4. [16]

    In-context Learning and Induction Hea ds

    Catherine Olsson, Nelson Elhage, Neel Nanda, et al. In-context Learning and Induction Hea ds. arXiv preprint arXiv:2209.11895,

    arXiv

  5. [17]

    Future lens: Anticipating subsequent tokens from a single hidden state

    Koyena Pal, Jiuding Sun, Andrew Yuan, et al. Future lens: Anticipating subsequent tokens from a single hidden state. arXiv preprint arXiv:2311.04897,

    arXiv

  6. [18]

    Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

    Tilman R¨auker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks. In 2023 ieee conference on secure and trustworthy machine learning (satml) , pp. 464–483. IEEE,

    2023

  7. [19]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu

    URL http://arxiv.org/abs/2207.07512. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063,

    arXiv

  8. [22]

    16 Published as a conference paper at ICLR 2025 0 1 2 3 4 Maze X 4 3 2 1 0Maze Y 95% (1798,

    reveals a pattern that likely accounts for the focusing of attention from ; context positions to positions 1 and/or 3 earlier in the context. 16 Published as a conference paper at ICLR 2025 0 1 2 3 4 Maze X 4 3 2 1 0Maze Y 95% (1798,

    2025

  9. [25]

    generic edge

    These patterns account for why LH03 directs its attention to even-parity cells, while odd-parity cells are attended to by LH07 or LH05. 18 Published as a conference paper at ICLR 2025 0 50 100 1500 20 40 60 80 100 120 140 160 −0.04 −0.03 −0.02 −0.01 0 0.01 0.02 0.03 0.04 Figur...

    2025

  10. [1619]

    Decision Tree Decoding - Stan 0.90 0.92 0.94 0.96 0.98 1.00 Accuracy Figure 18: Another SAE trained on Stan gives rise to the same compositional code. 0 2 4 0 1 2 3 4 73 65 61 63 77 76 53 59 58 66 59 54 57 68 53 59 69 53 48 45 63 55 49 55 70 Add Connection, Right 0 2 4 7 78 1 ...

    2025

  11. [2016]

    Eliciting Latent Predictions from Transformers with the Tuned Lens

    Nora Belrose, Zach Furman, Logan Smith, et al. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv preprint arXiv:2303.08112,

    arXiv

  12. [2017]

    Train Val

    11 Published as a conference paper at ICLR 2025 Appendices A G ENERALIZATION AS A FUNCTION OF INPUT SEQUENCE LENGTH Train Val Mixed Maze (Max DFS 6x6) Generalization Full Maze (DFS 7x7) 0.0 0.2 0.4 0.6 0.8 1.0Accuracy Model Performance Comparison (d_model, n_heads, n_layers, p...

    2025

  13. [2019]

    Ivanitskiy, Rusheb Shah, Alex F Spies, et al

    Michael I. Ivanitskiy, Rusheb Shah, Alex F Spies, et al. A configurable library for generating and manipulating maze datasets. arXiv preprint arXiv:2309.10498,

    arXiv

  14. [2021]

    Nelson Elhage, Tristan Hume, Catherine Olsson, et al

    https://transformer- circuits.pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,

    2021 arXiv

  15. [2022]

    Dictionary learning improves patch-free circuit discov- ery in mechanistic interpretability: A case study on othello-gpt.arXiv preprint arXiv:2402.12201,

    Zhengfu He, Xuyang Ge, Qiong Tang, et al. Dictionary learning improves patch-free circuit discov- ery in mechanistic interpretability: A case study on othello-gpt.arXiv preprint arXiv:2402.12201,

    arXiv

  16. [2023]

    Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, et al

    URL https://transformer-circuits.pub/2023/monosemantic-features/ index.html. Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, et al. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. arXiv preprint arXiv:2402.11917,

    2023 arXiv

  17. [2024]

    A structural probe for finding syntax in word representa- tions

    John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short ...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.