Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two frontier reasoning models encode the Tower of Hanoi state space almost perfectly at the prompt, then lose that representation during planning; restoring it recovers most of their lost accuracy.

desk verdict A serious mechanistic study: small-model world-model work is solid, but the large-model 'build then lose' claim needs retrained probes and a steering control before it is established. read the letter →

arxiv 2608.07077 v1 pith:DO5NSUKN submitted 2026-08-07 cs.AI cs.LG

Devin Pereira , Willem Zuidema This is my paper · ORCID
classification cs.AIcs.LG
keywords TowerofHanoiemergentworldmodelSierpińskitrianglelinearprobingactivationsteeringlargereasoningmodelschain-of-thoughtworld-modelmaintenance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large reasoning models fail at flat-to-flat Tower of Hanoi not because they lack an internal world model, but because they fail to maintain it. A 6-layer Transformer and two large reasoning models all develop a linearly decodable representation whose geometry matches the puzzle's state graph, the Sierpiński triangle, at the end of the prompt. During chain-of-thought generation this representation decays, and the models then solve only about half of instances optimally. Re-injecting the clean prompt-time activation during generation nearly doubles the solve rate for one model, so the collapse is a maintenance failure with a partial cure.

What carries the argument

The central object is the emergent world model: a linearly decodable embedding of the 81 valid four-disk configurations whose pairwise Euclidean distances track the shortest-path distances of the puzzle's state graph, whose geometry is the Sierpiński triangle. It is detected with a distance-matching linear probe, calibrated by Spearman and Pearson correlations, and its causal role is tested with activation patching at the separator token and with activation steering, which adds a cached prompt-time activation vector for the board's current configuration to the residual stream at every generated token. The same probe and steering toolkit is applied to the small Transformer and the large reasoning models, making the comparison between them direct.

What would settle it

Run the steering experiment with random, shuffled, or wrong-configuration steering vectors while keeping the external tracker: if any control recovers the same accuracy gain, the claim that the model's own representation is being restored is undermined.

Watch

Extended reading notes

Core claim

The paper claims that the reported 'illusion of thinking' collapse on the Tower of Hanoi is really a form of forgetting. Using a distance-matching linear probe, the authors find that a 6-layer Transformer trained from scratch on solution traces, and two frontier open-weight reasoning models (Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B), all encode the four-disk configuration with near-perfect fidelity and Sierpiński geometry at the final prompt token. Tracking the probe through generation shows the joint state representation degrades by the commitment point before the move list, reverting to a factored, per-disk form during move emission. Activation patching in the small model and activation steering in the large model establish causal use: substituting a donor's separator-token activation transfers state to the recipient, and steering Qwen3.6-27B with prompt-time activations throughout generation lifts optimal solves from 41% to 73% on the problems it previously failed. The paper concludes that the models build a world model and then lose it.

Load-bearing premise

The causal conclusion assumes the accuracy gain from steering comes from restoring the model's own degraded world model, not from any consistent added vector or from the external symbolic tracker that supplies the true current state at every step.

Editorial extensions

If this is right

  • On flat-to-flat Tower of Hanoi, the limiting factor for frontier reasoning models is maintaining a representation, not forming one, so accuracy is recoverable through inference-time state restoration rather than more search.
  • The same near-perfect Sierpiński encoding appears in a 6-layer Transformer and a 27B model, implying the world model is a property of the task, not of scale or training regime.
  • Activation steering nearly doubles the optimal-solve count for Qwen3.6-27B, so interventions that restore a clean state representation are a working mitigation for at least one class of planning failures.
  • For DeepSeek-R1-Distill-Qwen-32B, the same intervention fails because its errors are largely unparseable outputs, placing its failures at a stage residual-stream steering cannot reach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If representation decay scales with chain length, then longer reasoning traces may actively hurt planning accuracy; a testable prediction is that truncating or compressing the chain of thought, or periodically re-anchoring the state, improves solve rates.
  • The decay pattern may be general to planning domains with structured state spaces, but the paper tests only the Tower of Hanoi, so extending the probe-and-steer toolkit to other puzzles is the natural next check.
  • For models that fail with unparseable outputs, the right intervention is probably decoding-side constraints or output-format fine-tuning rather than world-model steering; the paper leaves this separation open.
  • The dependence on an external symbolic state tracker means the technique as designed applies where ground-truth state is cheap; for ambiguous real-world tasks, a learned state estimator would be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies the Tower of Hanoi in both a 6-layer Transformer trained from scratch on flat-to-flat solution traces and two large open-weight reasoning models (Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B). For the small model, it trains linear probes at the SEP token and reports a linearly decodable, approximately Sierpiński distance geometry, a unified-to-factored shift between planning and execution tokens, and causal involvement via whole-residual-stream activation patching. For the large models, it reports near-perfect decodability of the current configuration at the final prompt token (Position A), degraded decodability at the commitment point and during move emission (Positions B and C), and improved optimal-solving under activation steering that injects the true current state from an external symbolic tracker (Eq. 2). The authors conclude that large reasoning models build a world model and then lose it, so the 'illusion of thinking' collapse should be reframed as a maintenance failure rather than an absence of world models.

Significance. The flat-to-flat benchmark and the parallel small/large model comparison are a useful contribution, and the paper is honest about several limitations (single puzzle, two Qwen-derived models, whole-stream patching, external tracker). If the central claim were firmly established, it would be an important mechanistic explanation for the planning failures reported by Shojaee et al. (2025), and it would strengthen the case that supervised probing, when combined with causal interventions, can identify task-relevant internal structure in large reasoning models. However, the current evidence is not yet sufficient: the probe results lack held-out evaluation and retraining at later token positions, and the steering experiment lacks the control conditions needed to support a causal maintenance story. These gaps are fixable and are the main reason for my recommendation.

major comments (4)
  1. [Section 4.1, Eq. (1), Table 2] The distance-matching probe is trained and evaluated on the same 81 configurations, and the manuscript does not describe a held-out state split. Eq. (1) minimizes exactly the squared mismatch between the probe's pairwise distances and the graph distances over all states, so the reported Spearman correlations and perfect nearest-state retrieval are in-sample fits of the probe objective; they show that the residual stream can be mapped to the Sierpiński geometry, but not that the model generalizes this geometry. The same issue affects the Position A results for the large models in Section 5, where the 81 configurations are also used for probing. Please report k-fold or held-out probe evaluations, for both the distance-matching and per-disk probes, and state the probe training/evaluation split explicitly.
  2. [Section 5, Table 4] The degradation at Positions B and C is measured with probes fit at Position A and applied to later token positions; no retraining or distribution-matched baseline at B/C is described. Linear probes are sensitive to covariate shift, and the activations at B/C differ in token position, context length, and model state, so lower Spearman correlations and near-chance per-disk accuracies are expected even if the same state information remains linearly decodable in the new distribution. Table 4 therefore does not by itself establish that the world model decays. Please retrain the distance-matching and per-disk probes from scratch at Positions B and C, with train/test splits over states and with chance-level comparisons, and report both transfer and retrained results.
  3. [Section 5, Eq. (2), Table 5] The steering experiment injects the true current configuration, maintained by an external symbolic tracker, at every generated token, but includes no control condition with a random vector of the same norm or with a wrong-state steering vector. Without these controls, the accuracy gains in Table 5 could be explained by a generic effect of adding any consistent activation vector, or by the oracle state information itself, rather than by restoration of the model's own degraded world model. Please add no-steer, random-vector, and wrong-state steering controls, and ideally a tracker-free or delayed-steering ablation, before claiming that maintenance, not absence, is the bottleneck.
  4. [Abstract, Section 5, Table 5] The title and abstract generalize the 'build then lose' conclusion to both large models, but the causal steering evidence supports only Qwen3.6-27B; for DeepSeek-R1-Distill-Qwen-32B the best steering converts only 6 of 72 failures to optimal, and the authors attribute most of its failures to unparseable output. Because the paper's central claim is about large reasoning models generally, the conclusion should either be narrowed to models that produce parseable move lists, or supported by an additional evaluation for DeepSeek that separates formatting failure from planning failure (for example, scoring the intended plan before format collapse).
minor comments (6)
  1. [Throughout] Notation: the number of disks is N in Section 2 but n in Table 1 and Section 4; unify these symbols.
  2. [Section 5, Eq. (2)] The steering vector is written h_prompt(s_t) without the layer superscript ell that appears on the left-hand side; make the layer dependence explicit.
  3. [Table 1] The column headers such as '3 /34' are easy to misread; clarify that each cell is correct/total sampled instances for that disk size.
  4. [Section 5] State explicitly how Position B, the 'commitment point', is detected in the two models' free-form output, since this determines the probe comparison and is needed for reproducibility.
  5. [Limitations] The Limitations section begins with a sentence fragment ('single puzzle at a single size (n=4) Tower of Hanoi's...'); add the missing punctuation.
  6. [Table 4] At Position A the per-disk accuracies are well below the 'near-perfect' joint retrieval (for example, D0=0.48 at layer 48); explain the relationship between the distance-matching probe and the per-disk probes, since the reader might otherwise see a contradiction.

Circularity Check

1 steps flagged · score 3.0 of 10

Distance-matching probe reports its own training objective as discovered geometry; causal interventions keep the central claim largely independent.

  1. fitted input called prediction [Section 4.1, Eq. (1), Table 2; echoed in Section 5 Position A results]
    "To test whether the Sierpiński state space is (linearly) encoded, we train a linear probe fφ : R^d → R^2 that maps a hidden state to a 2D embedding whose pairwise Euclidean distances match the graph distances dG between configurations: Lprobe(φ) = 1/|S|^2 Σ (||fφ(hs)−fφ(hs′)||2 − dG(s, s′))^2. ... Probe results are also reported quantitatively by the Spearman (ρ) and Pearson (r) correlations between predicted and true pairwise distances, and by nearest-state retrieval accuracy..."

    The probe is trained to minimize exactly the distance mismatch that is then reported as the discovered geometry: Eq. (1) optimizes ||fφ(hs)−fφ(hs′)||2 against dG(s,s′) over all 81 configurations, and Table 2's ρ/r/nearest-state metrics are computed between these same predicted and true pairwise distances on the same 81 states. A high correlation therefore largely reflects successful optimization of the probe's own target, not an independent measurement of the model's geometry. The phrase 'geometrically faithful representation' is imposed by the probe's target space (a 2D Sierpiński embedding) rather than read off the activations.

full rationale

I find one partial circularity. The distance-matching probe's reported geometric fidelity is an in-sample fit to the same distance labels used in its loss, so the specific Sierpiński-geometry claim is partly constructed by the probe objective. This does not sink the paper's central derivation: per-disk probes, principal-angle analyses, activation patching, and activation steering provide independent evidence that some state representation is present and causally used. The steering experiment is not circular; it demonstrates that supplying oracle state information improves performance, though without a random/wrong-vector control it does not by itself isolate 'maintenance' as the mechanism. The B/C degradation measurement would be cleaner with probes retrained at each position and held-out states; the paper does not explicitly describe such retraining. The paper itself concedes in Limitations: 'A supervised probe can fit structure the model does not use; we mitigate this with patching and steering, which test causal use directly.' Overall, the derivation chain is largely self-contained and the central claim has independent causal support, so the circularity burden is moderate, not maximal.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rely on accepted interpretability assumptions (linear probes and causal interventions), on correct symbolic traces and parsing, and on hand-chosen steering hyperparameters. No new entities are introduced. The main ledger cost is the probe objective itself, which builds the reported geometry into the fitted function.

free parameters (2)
  • Steering strength alpha = swept 0.5, 1, 2, 5, 10; best 2 for Qwen3.6-27B, 1 for DeepSeek-R1-Distill
    The reported recovery of optimal solutions depends on this magnitude; it is selected on the same failed problems used for the headline result, with no held-out validation.
  • Steering layer ell = 28 for Qwen3.6-27B; 44 for DeepSeek-R1-Distill
    The intervention layer is chosen by hand; results may depend on this choice and no selection rule or sensitivity analysis across layers is given.
assumptions (5)
  • standard math The 4-disk Tower of Hanoi state graph is the Sierpinski triangle, and graph distances define the correct geometry.
    Invoked in Section 2 and used as the target in the distance-matching probe loss (Eq. 1).
  • domain assumption Linear decodability by a probe implies the model internally represents the state in that geometry.
    Sections 4.1 and 5 rely on the linear representation hypothesis; the Limitations section acknowledges probes can fit structure the model does not use.
  • domain assumption Activation patching and steering interventions reveal causal use of the probed representation.
    Sections 4.3 and 5 interpret patching and steering as causal evidence, which assumes the residual-stream components intervened on are causally upstream of the output.
  • domain assumption The symbolic state tracker correctly parses the model's chain of thought and maintains the true board state.
    Section 5's always-steer regime recomputes the current configuration with a regex matcher and replay; if parsing or state tracking fails, the injected steering target is wrong.
  • domain assumption The precomputed flat-to-flat solution traces used for training are optimal and complete.
    Section 4 trains on symbolically generated traces; correctness of these traces is not independently verified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking." pith.science (2026). https://pith.science/paper/DO5NSUKN

@misc{pith2026260807077,
  author       = {Pith},
  title        = {Pith review of: Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DO5NSUKN}},
  note         = {Machine review of arXiv:2608.07077}
}
read the original abstract

The Tower of Hanoi is a simple planning puzzle that in prior work has proven challenging for large reasoning models (LRMs). Current models solve the standard formulation of the puzzle, but still struggle with the flat-to-flat variant (where initial and goal states are not restricted to have all rings on a single peg). This paper presents an in-depth study of how both small, in-house Transformers and large, third-party LRMs solve this task. To understand the failures mechanistically, we first train small Transformers from scratch on precomputed solution traces. Using a variety of interpretability techniques, we show that these Transformers develop an emergent world model: a linearly decodable, geometrically faithful representation of the puzzle's state space (the Sierpinski triangle), that is causally involved in solving the puzzles. Second, we return to the large LLMs and apply our techniques to two frontier reasoning models, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B, that attempt to solve the task through extended chain-of-thought. Surprisingly, we find that both models encode the Sierpinski world model near-perfectly at the end of the prompt, and yet fail at the majority of tasks when there are more than 3 rings. We locate the source of this failure in the decaying representation of the world model. We probe for the representation at different stages during planning, and establish causality by showing that performance can be improved by injecting the prompt-time representation at inference. The failure of the models is thus one of maintenance of the required representations, not their absence, and performance is at least partially recoverable. These results thus reframe the reported collapse in performance from prior work: current Large Reasoning Models build a world model, and then lose it.

Figures

Figures reproduced from arXiv: 2608.07077 by the authors.

Figure 1
Figure 1. Tower-to-tower (top) and flat-to-flat (bottom) vari [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The state space of the 4-disk Tower of Hanoi forms [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. 2D output of the distance-matching probe on the raw residual stream of the SEP token at layer 5, over training (from left [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 13 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    On the planning abilities of large language models-a critical investigation , author=. Advances in neural information processing systems , volume=

    work page

  2. [2]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  3. [3]

    https://www.anthropic.com/research/reasoning-models-dont-say-think , author =

    Reasoning models don't always say what they think , url =. https://www.anthropic.com/research/reasoning-models-dont-say-think , author =. 2025 , note =

    work page 2025

  4. [4]

    arXiv preprint arXiv:2506.06941 , year=

    The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity , author=. arXiv preprint arXiv:2506.06941 , year=

    arXiv

  5. [5]

    Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    On the dangers of stochastic parrots: Can language models be too big? , author=. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    2021

  6. [6]

    arXiv preprint arXiv:2210.13382 , year=

    Emergent world representations: Exploring a sequence model trained on a synthetic task , author=. arXiv preprint arXiv:2210.13382 , year=

    arXiv

  7. [7]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Chess as a testbed for language model state tracking , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    work page

  8. [8]

    ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling , year=

    Transformers Use Causal World Models in Maze-Solving Tasks , author=. ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling , year=

    work page 2025

Show all 33 references
  1. [9]

    Computational Linguistics , volume=

    Probing classifiers: Promises, shortcomings, and advances , author=. Computational Linguistics , volume=

  2. [10]

    Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies , pages=

    Linguistic regularities in continuous space word representations , author=. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies , pages=

    2013

  3. [11]

    arXiv preprint arXiv:2311.03658 , year=

    The linear representation hypothesis and the geometry of large language models , author=. arXiv preprint arXiv:2311.03658 , year=

    arXiv

  4. [12]

    Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=

    Recurrent neural networks learn to store and generate sequences using non-linear representations , author=. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=

  5. [13]

    International Conference on Learning Representations , volume=

    Not all language model features are one-dimensionally linear , author=. International Conference on Learning Representations , volume=

  6. [14]

    Mathematics of computation , volume=

    Numerical methods for computing angles between linear subspaces , author=. Mathematics of computation , volume=

  7. [15]

    Advances in neural information processing systems , volume=

    Investigating gender bias in language models using causal mediation analysis , author=. Advances in neural information processing systems , volume=

  8. [16]

    Advances in neural information processing systems , volume=

    Locating and editing factual associations in gpt , author=. Advances in neural information processing systems , volume=

  9. [17]

    arXiv preprint arXiv:2211.00593 , year=

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small , author=. arXiv preprint arXiv:2211.00593 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2308.10248 , year=

    Steering language models with activation engineering , author=. arXiv preprint arXiv:2308.10248 , year=

    arXiv

  11. [19]

    Findings of the Association for Computational Linguistics: ACL 2022 , pages=

    Extracting latent steering vectors from pretrained language models , author=. Findings of the Association for Computational Linguistics: ACL 2022 , pages=

    2022

  12. [20]

    arXiv preprint arXiv:2304.00740 , year=

    Inspecting and editing knowledge representations in language models , author=. arXiv preprint arXiv:2304.00740 , year=

    arXiv

  13. [21]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  14. [22]

    Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=

    Emergent linear representations in world models of self-supervised sequence models , author=. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages=

  15. [23]

    Cognitive Psychology , volume=

    The functional equivalence of problem solving skills , author=. Cognitive Psychology , volume=. 1975 , publisher=

    1975

  16. [24]

    Assessment , volume=

    The Development and Preliminary Validation of the Tower of Hanoi-Revised , author=. Assessment , volume=. 2001 , publisher=

    2001

  17. [25]

    2025 , eprint=

    Rethinking the Illusion of Thinking , author=. 2025 , eprint=

    2025

  18. [26]

    2025 , eprint=

    Solving a Million-Step LLM Task with Zero Errors , author=. 2025 , eprint=

    2025

  19. [27]

    2026 , eprint=

    Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens , author=. 2026 , eprint=

    2026

  20. [28]

    DeepSeek-AI and Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Zhang, Ruoyu and Xu, Runxin and Zhu, Qihao and Ma, Shirong and Wang, Peiyi and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhu...

    doi arXiv

  21. [29]

    Chain-of-

    Arcuschin, Iván and Janiak, Jett and Krzyzanowski, Robert and Rajamanoharan, Senthooran and Nanda, Neel and Conmy, Arthur , month = jun, year =. Chain-of-. doi:10.48550/arXiv.2503.08679 , abstract =

    doi arXiv

  22. [30]

    Reasoning models don't always say what they think , url =

  23. [31]

    TMLR , author =

    A. TMLR , author =. 2025 , note =. doi:10.48550/arXiv.2504.09037 , abstract =

    2025 doi

  24. [32]

    https://openai.com/index/harness-engineering/ , author =

    Harness engineering: leveraging. https://openai.com/index/harness-engineering/ , author =

  25. [33]

    https://www.anthropic.com/engineering/multi-agent-research-system , author =

    How we built our multi-agent research system , url =. https://www.anthropic.com/engineering/multi-agent-research-system , author =

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.