{"id":"e141a7ac-5636-4618-b97b-6fc31abb2569","arxiv_id":"2412.11867","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Maze-solving transformers store a causal, steerable map of maze connections in sparse features, and activating these features is more effective than removing them.","lead":"Transformers trained to solve mazes encode the maze's connectivity in sparse internal features, and flipping these features changes the paths the models output. The work shows how internal 'world models' can be found and steered, which matters for interpreting and eventually controlling large language models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Intervention-success metric and asymmetric intervention scope confound the add-vs-remove finding; controls needed.","rationale":"The reader's weakest assumption targets the intervention-success metric in Section 4, and I agree that it is the most load-bearing unaddressed concern. The paper's central claim — that maze-solving transformers acquire causally efficacious world models in SAE features — is supported by multiple independent lines of evidence: SAE reconstructions preserve behavior (Table 3, Figure 8), attention-head patching confirms the circuit-feature link (Figure 7b), and specific feature toggles produce predictable path changes (Figure 8). That core claim is likely sound. The 'easier to activate than suppress' asymmetry, however, is a headline contribution that depends directly on the success metric. If an unchanged path is scored as a failure for removals, removal rates are understated; and the addition intervention's global application to all semicolon positions makes it a much larger perturbation than the local removal. These two confounds could fully explain the asymmetry without any intrinsic property of the world model. The length-generalization claim is also overstated in the abstract, as Stan's latent activation succeeded only 35% of the time, but that is secondary to the asymmetry. The proposed concrete test would settle the concern by using a path-correctness metric and controlling for path relevance and intervention scope. If the asymmetry survives, it becomes a robust finding; if not, the paper's main novel contribution is a measurement artifact. Therefore the reader's CONDITIONAL verdict remains appropriate: the causal world model claim is credible, but the asymmetry and length-generalization claims need the suggested controls before they can be accepted as stated.","tokens_in":22721,"tokens_out":8076,"duration_ms":71259,"concrete_test":"Recompute the Section 4 / Figure 9 intervention results with a path-correctness metric: for each example, determine whether the model's output equals the true shortest path in the modified maze (edge added or removed), and stratify by (i) whether the toggled edge lies on the original path, and (ii) whether the modified correct path differs from the original. Also rerun additions by toggling the feature at only the semicolon corresponding to the edge's would-be position (or at a random single semicolon) instead of all semicolons. If the asymmetry persists under path-correctness and matched intervention scope, the finding stands; otherwise it is a measurement artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline asymmetry — that activating SAE features is easier than suppressing them — rests on the intervention-success metric of Section 4 / Figure 9, which counts an intervention as successful only if the toggled connection 'leads the model to act accordingly.' As the reader notes, for a removed edge not on the original solution path, the correct behavior is to leave the path unchanged; if the metric treats an unchanged path as a failure, removal success rates are systematically deflated. The comparison is further biased by the intervention's asymmetric scope: Footnote 3 states that for additions the feature is toggled at all semicolon positions (a global perturbation), while removal only zeroes the feature where it is already active. Thus additions inject new information everywhere, whereas removals act only locally; the observed add-vs-remove gap could reflect this scope difference rather than a property of the world model. The paper does not report success rates stratified by whether the toggled edge lies on the original path or by whether the correct path should change, so the asymmetry claim is not yet controlled for.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies two autoregressive transformer maze solvers, Stan (learned positional embeddings) and Terry (rotary embeddings), trained on synthetic 5x5 and 6x6 mazes embedded in a 7x7 grid. It identifies maze-connection features through two routes: layer-0 attention heads that attend from semicolon tokens to coordinate tokens, and sparse autoencoders whose features are labeled with decision trees. The two analyses are compared via cosine similarity and attention-patching effects. The authors then intervene on SAE features, activating or suppressing connection features, and report an asymmetry: adding connections by feature activation is more effective than removing them. They also report that Stan can use SAE-added connections even though it fails when the same additional connections are presented in the input text, and that the positional encoding scheme affects whether the world model is encoded as single features or as a compositional code. The causal role of the discovered features is supported by direct interventions, but the quantitative asymmetry claim rests on a success metric and an intervention protocol that require additional controls.","tokens_in":22852,"tokens_out":6388,"duration_ms":60307,"significance":"If the findings hold, the paper provides a useful controlled demonstration that SAE-discovered features in a planning domain can be causally steered, and that learned versus rotary positional encodings lead to different factorizations of world-model features. The strengths are the controlled synthetic domain, the combination of independent evidence from attention analysis, decision-tree feature labeling, attention patching, and direct interventions, and the reconstruction-fidelity check indicating that the SAE captures the relevant residual-stream information. The paper is also transparent about model-specific failures. However, the headline asymmetry that activating features is easier than suppressing them is not yet established, because the success metric and the perturbation protocol are asymmetric between additions and removals. The general causal-world-model claim is defensible, but the quantitative asymmetry requires revision.","major_comments":[{"comment":"The success criterion 'the toggling of a connection in the SAE feature space leads the model to act accordingly' is operationalized as a change in the output path, but this is not a valid measure of a correct intervention for edge removals. If the removed edge is not on the original solution path, the correct output path is the original path, and an unchanged path should count as success; under the current metric it is counted as failure. Since adding an edge can create a new shortcut while removing an off-path edge should leave the path unchanged, this path-relevance confound loads the add-vs-remove comparison in the direction reported. Please report success rates stratified by whether the toggled edge lies on the original solution path, and define success as equality with the ground-truth path of the modified maze rather than as 'path changed'.","section":"Section 4, Figure 9"},{"comment":"The add and remove interventions are not symmetric. An addition toggles the feature at every semicolon position, injecting a global signal into all connection slots, whereas a removal zeroes the feature only at positions where it is active, because elsewhere it is already approximately zero. The observed add-vs-remove gap may therefore reflect the breadth of the perturbation rather than a property of the world model. Please add control interventions, such as activating the feature at a single randomly chosen semicolon, removing the feature at all semicolon positions even where inactive, or injecting a semantically unrelated feature at all semicolon positions, and report whether the asymmetry persists under those controls.","section":"Section 4, Footnote 3"},{"comment":"All removal success rates for Stan are 0% in Figure 9, and the text attributes this to Stan's inability to generalize to inputs with more connections than it saw during training. However, the caption of Figure 9 says the evaluation is restricted to examples on which the original prediction was correct, which seems inconsistent if the relevant inputs are out of Stan's length distribution. Please clarify the example-selection procedure and report the base accuracy on the intervention set. If Stan already fails on the unperturbed inputs, its 0% removal rate should be excluded from, or analyzed separately in, the asymmetry claim.","section":"Section 4, Stan removal panels"}],"minor_comments":[{"comment":"The citation '(Nanda, 2023) SELF CITE Workshop proceeding' contains a placeholder that should be replaced with the full citation.","section":"Section 3"},{"comment":"There is a typo in 'M1odels' in the first sentence of Section 2.1; it should read 'Models'.","section":"Section 2.1"},{"comment":"The caption and the main text should give an explicit boolean definition of an 'accurate intervention'; as written, 'leads the model to act accordingly' is ambiguous between 'changes the path' and 'produces the correct path in the modified maze'.","section":"Section 4, Figure 9 caption"},{"comment":"The coefficient -0.6 used to combine the specific and generic SAE edge features is chosen to maximize cosine similarity; please report the sensitivity of Figure 7a to this value and to the number of examples averaged.","section":"Appendix G"},{"comment":"Consider reporting the SAE reconstruction metrics from Table 3 in the main text, since the claim that the SAE faithfully reconstructs activations is central to the intervention method.","section":"Section 3.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The concerns in major comments 1 and 2 are specific and fixable, but the revised asymmetry analysis should be seen before acceptance. The Stan/Terry comparison is further complicated by the length-generalization failure; if the authors cannot stratify or add the requested controls, they should soften the asymmetry claim. The placeholder citation and the ambiguity in the success metric also suggest that the manuscript needs careful proofreading and a more precise experimental protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe headline: this is a genuinely useful mechanistic interpretability paper. It gives the first causal evidence I know of that SAE features in maze-solving transformers encode a world model and can be steered. The decision-tree labeling of SAE latents is a practical trick, and the attention-patching experiments tie the SAE features back to specific heads. The causal interventions—toggling a connection feature and watching the path change—are direct and convincing in the worked examples. This goes beyond the prior maze work, which found linear world representations but could not isolate causal features. Also new: Stan's compositional code under learned positional embeddings, and the latent-space generalization result.\n\nThe main soft spot is the add-vs-remove asymmetry. The success metric in Section 4 appears to count an intervention as successful only if the output path changes. For removing an edge that is not on the original solution path, the correct model behavior is to leave the path unchanged; counting that as failure will systematically deflate removal success. The paper does not stratify by whether the toggled edge lies on the original path, so the asymmetry claim is confounded and should be treated as suggestive. Note: the stress-test's separate concern about asymmetric intervention scope does not hold up—the text toggles at all semicolon positions for both directions, and the footnote says disabling everywhere vs locally makes little difference for removal. The real problem is the metric.\n\nTwo smaller issues. The SAE/OV cosine-similarity comparison uses a coefficient picked to maximize similarity (Appendix G), so that consistency number is partly fitted. And Stan's latent-space generalization success is 35%, which the abstract's phrasing overstates. No code or trained models are apparently released, which hurts reproducibility for a methods-oriented paper.\n\nWho this is for: mechanistic interpretability researchers interested in SAE feature validity and steering. The core causal claim is well-grounded; the asymmetry is a fixable measurement problem rather than a fatal one. I'd send this to a serious referee (it is already at ICLR 2025), and I'd cite it for the decision-tree-plus-intervention methodology.","headline":"A solid SAE-based causal-world-model paper: the core claim of causally efficacious features holds up, but the add-vs-remove asymmetry is confounded by the success metric.","tokens_in":23413,"tokens_out":4605,"would_cite":true,"duration_ms":38031,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformers trained on maze-solving encode a causal world model of maze connectivity in sparse-autoencoder features; adding a connection feature steers their behavior much more reliably than suppressing one.","keywords":["causal world models","sparse autoencoders","mechanistic interpretability","maze solving","transformer interpretability","feature intervention","positional encoding"],"falsifier":"Compute intervention success separately for connections that lie on the original solution path versus those that do not, scoring 'leaving the correct path unchanged' as success for off-path connections; if on-path removal succeeds at rates comparable to addition, the claimed add-versus-remove asymmetry is an artifact of the metric.","tokens_in":22501,"feed_emoji":"🧭","tokens_out":4046,"duration_ms":37137,"temperature":0.7,"pith_summary":"This paper sets out to show that transformer models trained to solve shortest-path mazes do not merely memorize token patterns but build internal, causally effective representations, or world models, of maze connectivity. Using sparse autoencoders and attention analysis, the authors locate features that encode individual connections between maze cells, then show by direct intervention that toggling these features changes the predicted path. The central empirical findings are that activating a connection feature changes behavior much more often than suppressing one, and that adding features can make a model solve mazes with more connections than it saw in training, even though the same mazes presented as longer token sequences break the model. The paper also finds that the positional encoding scheme shapes how the world model is stored: rotary embeddings yield one clean feature per connection, while learned positional embeddings split each connection across a shared semicolon feature and a connection-specific feature.","feed_headline":"Adding features steers maze AI; removing them does not","feed_subtitle":"SAE connection features act causally and let models handle bigger mazes than the token sequence alone allows.","key_machinery":"The central objects are the sparse autoencoder features on the residual stream immediately after layer 0, trained under a sparsity penalty with ghost-gradient resampling, and the layer-0 'connectivity attention heads' that attend from each semicolon token one or three positions back to the coordinate tokens of a connection, writing edge information through their OV circuits. Decision trees trained on SAE latents isolate which features predict a given connection, and attention-patching experiments confirm that removing a head's contribution at a connection's semicolon reduces the corresponding SAE feature. Interventions toggle these features at all semicolon positions, activating them to their maximal observed value or setting them to zero, to test their causal role in the model's maze-solving output. Together these components give an unsupervised, causal handle on the model's internal map without assuming the form of the world model in advance.","core_discovery":"On the paper's own terms: transformers trained on maze-solving tasks develop a sparse, causal world model of the maze's connectivity graph. Layer-0 attention heads consolidate adjacency information at each semicolon separator token, and sparse autoencoders trained on the residual stream after layer 0 recover individual features that predict the presence of specific maze edges; attention-patching experiments confirm these are the same features the heads write. Intervening in the SAE latent space shows the features are causally efficacious: setting a connection feature to its maximal observed value changes the model's path as if the connection existed, while zeroing it rarely changes behavior. A further claim is that the model can generalize in latent space to mazes with more simultaneously active connection features than it was trained on, even when the equivalent token-level input would cause failure, and that this latent behavior remains coherent. Finally, the paper claims that the encoding scheme, learned versus rotary positional embeddings, determines whether connection knowledge lives in single SAE features or is split compositionally.","pith_inferences":["The add-versus-remove asymmetry may partly reflect the paper's success metric: adding an off-path edge can create a shortcut, whereas removing an edge only changes the path when that edge lies on the original solution path, so the two conditions are not measured on the same footing.","If latent-space generalization to more connections is confirmed, it suggests that token-level failure on long sequences is an input-encoding bottleneck rather than a reasoning bottleneck; a testable extension is whether randomizing adjacency-list order removes the length failure.","For learned positional embeddings, the compositional code predicts that steering a single SAE direction, such as the shared semicolon feature alone, should fail to change behavior coherently, which would directly explain prior linear-probe intervention failures."],"forward_implications":["SAE-discovered connection features can serve as steering knobs: turning on an internal connection feature reroutes the model's predicted path in a high fraction of cases.","Activation is a more reliable intervention than suppression, implying that these transformers lean on the presence of connectivity cues more than on their absence.","Models can handle larger, more connected mazes through latent feature activation than through token-level input, so some capability limits are tied to positional encoding and input formatting rather than to internal reasoning.","Positional encoding choice changes how interpretable the world model is: rotary embeddings give near-monosemantic connection features, while learned embeddings produce a compositional code that single-direction interventions cannot cleanly target."],"supporting_citations":[{"why":"Introduces the sparse autoencoder dictionary-learning method used to isolate monosemantic features in the residual stream.","marker":"[Bricken et al., 2023]"},{"why":"Prior maze-solving work whose linear-probe world-model features could not be causally intervened on, providing the contrast this paper addresses.","marker":"[Ivanitskiy et al., 2024]"},{"why":"Establishes sparse autoencoders as a method for recovering interpretable features in transformer activations, the foundation of the feature extraction pipeline.","marker":"[Cunningham et al., 2023]"},{"why":"Supplies the ghost-gradient resampling technique that prevents neuron death in the trained sparse autoencoders.","marker":"[Jermyn & Templeton, 2024]"},{"why":"Provides the OV-circuit analysis framework used to identify the layer-0 attention heads that write connection features.","marker":"[Elhage et al., 2021]"},{"why":"Supplies the definition of world models as structure-preserving, causally efficacious representations that the paper adopts.","marker":"[Millière & Buckner, 2024]"}],"fun_headline_variants":["Maze AI: latent features steer, removal fails","Causal world models let maze transformers scale beyond training","Activating maze features changes path; suppressing doesn't","Transformers solve bigger mazes via latent graph, not tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The intervention metric counts an intervention as successful only when it changes the model's output path; for a connection that is not on the true solution path, the model responding correctly by leaving the path unchanged is scored as a failure, and this loads the comparison in favor of adding features over removing them.","fun_headline_variants_meta":{"raw":{"variants":["Maze AI: latent features steer, removal fails","Causal world models let maze transformers scale beyond training","Activating maze features changes path; suppressing doesn't","Transformers solve bigger mazes via latent graph, not tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1281,"prompt_tokens":921,"completion_tokens":360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":537,"tokens_out":360,"duration_ms":4649,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:30:22.290013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute intervention success separately for connections that lie on the original solution path versus those that do not, scoring 'leaving the correct path unchanged' as success for off-path connections; if on-path removal succeeds at rates comparable to addition, the claimed add-versus-remove asymmetry is an artifact of the metric.","supporting_citations":[{"cited_title":"Anthropic circuits Updates - January 2024, Jan- uary","cited_arxiv_id":null,"evidence_quote":"Supplies the ghost-gradient resampling technique that prevents neuron death in the trained sparse autoencoders."}],"review_version":1}