Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Model-free reinforcement learning yields decision-time planning when each environment state is bound to its own hidden cell and cells exchange messages along learned relations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:55 UTC pith:2CVWDJBD

load-bearing objection Attention-core demonstration is solid and new; the 'binding is deciding factor' claim outruns the evidence because the slot control is confounded by underperformance and an extra slot-attention module. the 3 major comments →

arxiv 2607.18589 v1 pith:2CVWDJBD submitted 2026-07-20 cs.LG cs.AI

Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States

Armin Sommer This is my paper
classification cs.LG cs.AI
keywords emergent planningmodel-free reinforcement learningrelational hidden statesstate-graph bindingdecision-time planningattentionSokobanmechanistic interpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning is usually divided into model-based methods that plan by simulating a learned world model and model-free methods that react. This paper argues that planning can emerge from model-free reward maximization alone, and that the deciding factor is the hidden-state structure of the network: a pool of cells, each bound to one environment state, exchanging messages along relations that come to respect the transition graph. An attention-based agent trained on Sokoban discovers this binding and routing from reward alone, and shows planning: a decodable goal-directed plan, decisions that improve with added thinking time, and a routing that concentrates on true one-step successors. A matched control whose slots must also learn which cell represents which state fails to bind, learns no transition graph, and shows no planning — leading the paper to conclude that the relational bias over state-bound cells, not any built-in graph, produces planning.

Core claim

Emergent planning in model-free RL is driven by a relational hidden state: a set of neural cells, each anchored to a distinct environment state, exchanging messages along learned pairwise relations. With this anchoring, the relations settle into the environment's transition graph, and each thinking step moves information one hop along that graph, so K updates implement a K-step lookahead planner without a world model. The paper tests this in an attention-based Sokoban agent, which discovers the binding and routing from reward, decodes a goal-directed plan, and improves with thinking time. A matched control whose slots must also learn the cell-to-state mapping forms no binding, learns no tran

What carries the argument

A relational hidden state is a recurrent update in which each cell h(i) is updated by messages from other cells via a learned relation F_φ, instantiated by convolutions or attention. State-graph binding adds a map σ from environment states to cells such that each state's statistic is carried by a unique cell, and the relations respect the transition graph. Together they turn the K-fold thinking recursion into a horizon-K decision-time planner: each step moves information one hop along the learned transition graph, and the readout at the current state's cell returns the best successor. The SlotLSTM control replaces the fixed binding with a slot-attention competition that must discover the cor

Load-bearing premise

The load-bearing premise is that the slot control is otherwise matched to the attention core; its failure to plan is blamed on the missing fixed cell-to-state binding, but it trains to a lower solve rate (65–68% vs 91.9%) at a fixed thinking depth K=4, so the failure could instead be underperformance or optimization difficulty — an openness the paper itself notes in §6.

What would settle it

Train the slot-based control with variable thinking depth and for enough steps to match the attention core's ~92% training solve rate, then probe for state-graph binding, graph-respecting routing, and plan refinement; if the slot agent now binds and plans, the central claim that a supplied binding is load-bearing is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any architecture that supplies a cheap one-state-per-cell correspondence — such as grid layouts or spatially indexed fields — should exhibit planning gains under model-free training and extra test-time computation.
  • The learned attention routing is itself an amortized transition model; no separate predictive loss is needed to fit the world model.
  • Architectures without a stable binding, such as transformers whose positions mix many features, are predicted to show little or no emergent planning unless a binding is realized elsewhere.
  • The four mechanistic signatures used here (localized decision statistics, graph-respecting routing, decodable plan, refinement with thinking) provide a reusable test battery for detecting decision-time planning in other trained RL agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: train the slot control with variable thinking depth and a training budget matched to the attention core's ~92% training solve rate; if planning signatures then appear, the claim that a supplied binding is load-bearing is falsified.
  • Implicit prediction: a transformer that first binds states to dedicated tokens (e.g., via a slot-attention encoder) before relational reasoning should show the same planning signatures on Sokoban, whereas a vanilla residual-stream transformer should not.
  • The soft geometric decay of attention (ρ≈0.61 per graph hop) suggests the effective planning horizon is graded rather than a hard K-step cutoff; gains from thinking time should track how strongly routing concentrates on true successors, which could be optimized directly.
  • If the brain hypothesis is right, it yields a concrete prediction: hippocampal-like structures that bind items to dedicated representations are necessary for reward-driven planning, and degrading that binding should selectively destroy deliberative but not reactive behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that emergent decision-time planning in model-free RL is caused by an architectural prior: relational hidden states whose cells are bound one-to-one to environment states, with learned relations that come to respect the environment's transition graph. The authors train an attention-based recurrent agent (AttnLSTM) on Boxoban with IMPALA and no auxiliary model loss, and present four mechanistic signatures: (i) the action statistic is localized to the bound cell; (ii) the attention routing is an amortized transition model, decaying with graph distance and causally sensitive to walls; (iii) a goal-directed plan is decodable over the cell field; and (iv) additional thinking steps refine the decision. To isolate the role of the supplied binding, they train a SlotLSTM free-slot control and report that no stable binding, no transition-respecting routing, and no planning emerge. They conclude that the relational bias operating over state-bound cells, not a built-in graph or generic relational computation, is the deciding factor for emergent planning.

Significance. If the central claim holds, the paper would make a valuable contribution: it identifies a concrete architectural prior—state-graph binding plus learned relational message passing—under which model-free RL can yield decision-time planning, and it connects the DRC/ConvLSTM results of prior work to a new attention-based instantiation. The empirical work is multi-pronged and carefully reported: three independently trained AttnLSTM and SlotLSTM seeds, per-seed tables, mean±sd bands, faithful recomputation of the forward pass for probing, and causal wall interventions. The paper is also commendably explicit about its own limitations in §6. The significance is, however, conditional on the SlotLSTM control being an interpretable matched ablation; if that control is confounded, the central causal claim is not established.

major comments (3)
  1. [§5.5, App. A, Table 1] The negative control is confounded by training performance. SlotLSTM reaches only a 65–68% training solve rate versus 91.9% for AttnLSTM, so its failure to form a binding or plan could be due to underperformance, optimization difficulty, or insufficient training rather than the absence of a supplied cell-to-state correspondence. The fixed-K AttnLSTM comparison in §5.5 addresses the depth schedule confound, but the paper does not report that variant's training solve rate, so it does not eliminate the underperformance confound. To support the 'deciding factor' claim, the authors should either train SlotLSTM to a comparable training solve rate, report the AttnLSTM-K=4 training solve rate and show it matches SlotLSTM, or otherwise demonstrate that the negative result is not a capacity/optimization artifact.
  2. [§5.5, Eq. (22)] SlotLSTM differs from AttnLSTM not only in lacking a fixed binding but also in containing an additional slot-attention competition module (Eq. 22). This module introduces a different optimization landscape and an extra mechanism whose known training difficulty could explain the absence of planning, independent of whether binding is load-bearing. The control is therefore not a matched removal of the binding; it is a different architecture. A cleaner ablation would keep the AttnLSTM core and replace the fixed cell-to-square indexing with a learned assignment or soft permutation, or keep SlotLSTM but supply a fixed one-to-one assignment at initialization, so that the only difference is whether the binding is given rather than discovered.
  3. [Abstract, Introduction, §6] The abstract and introduction state that the hidden-state structure of the architecture is 'the deciding factor' and that 'it is therefore the relational bias operating over state-bound cells, not any built-in graph, that produces planning.' This overstates what the evidence supports, because the only direct evidence against learned binding is the confounded SlotLSTM result. Section 6 itself acknowledges that a different learned-binding mechanism, longer training, or greater scale might stabilize σ. The central claim should be tempered to what the experiments establish, e.g., 'in the tested training regime and architectures, we find no evidence that a slot-attention-based learned binding suffices,' and the abstract should carry this caveat rather than stating the deciding factor without qualification.
minor comments (5)
  1. [§4.3, Prop. 4.1] Proposition 4.1 is an existence result: it shows that some parameters (φ, ρπ) can implement decision-time planning under Assumption 4.1. The surrounding text in §4.3 says SGD 'can discover' planning, but the theoretical argument does not establish learnability; that claim rests entirely on the empirical results. This distinction should be stated explicitly to avoid over-reading the proposition.
  2. [Fig. 2 caption] The caption correctly notes that the horizontal axes are different interventions, but the figure places the two curves side by side without a shared axis. Consider labeling the axes more prominently or separating the panels clearly so readers do not compare slopes directly.
  3. [Eq. (20), App. A] The entmax1.5 operation is used but not defined or cited. A brief definition or reference would help readers unfamiliar with sparse attention.
  4. [§5.1] The phrase 'the policy statistic πt(s)' is slightly vague; it is later clarified as the model's own next-action readout. Consider defining it at first use.
  5. [Throughout] There are occasional formatting issues, e.g., 'equation 4' in §3.2 and the odd caption in Fig. 3a ('Q .24 ≈0 ⋆'). A careful proofread would improve readability.

Circularity Check

0 steps flagged

No significant circularity; the core claims are empirical and the negative control, though confounded, is not a definitional reduction.

full rationale

The derivation chain is not circular. Definition 3.2 and Assumption 4.1 are explicit assumptions; Proposition 4.1 is an asserted equivalence between message passing over transition-respecting relations and a horizon-K lookahead, but the paper does not use it as a fitted constant or rename an input as a prediction. The empirical sections test the assumptions independently: attention weights are compared against the environment's true transition graph, action localization is measured with a probe, plan refinement is measured by policy readouts, and the SlotLSTM control is a separate architecture. No self-citations by the sole author appear; the cited DRC planning results are external. The main weakness is the SlotLSTM control's lower training solve rate (65–68% vs 91.9%, App. A) and its extra slot-attention module, which is a validity/correctness confound for the 'deciding factor' claim, not a circularity: the absence of planning in SlotLSTM is not guaranteed by construction, and the paper itself leaves longer training and alternative learned-binding mechanisms open in §6. Prop. 4.1 is stated without proof, a rigor gap, but the empirical claim does not reduce to it by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 2 invented entities

The paper introduces no physical free parameters; its hyperparameters (thinking depth K, entmax α=1.5, D=3, etc.) are standard architecture choices, not fitted to make the central claim. The load-bearing assumptions are the deterministic MDP setting, the localized-readout hypothesis, the architectural supply of binding, the existence claim in Proposition 4.1, and the behavioral definition of planning. Invented constructs are operational and falsifiable, so they do not function as uncontrolled degrees of freedom.

axioms (5)
  • domain assumption The environment is a deterministic finite MDP with transition map T (Section 2.1).
    The Sokoban environment is deterministic, and the theory uses P(s'|s) and successor sets N(s) that require determinism (Eq. 6).
  • ad hoc to paper Assumption 4.1 (Localized readout): after K thinking steps, the policy statistic π(·|s_t) is recoverable from the hidden cell σ(s_t).
    This is a hypothesis the paper tests empirically (Section 5.1), but it is needed for Proposition 4.1 to connect message passing to the policy.
  • domain assumption State-graph binding σ exists and is supplied by the convolutional encoder in the attention core (Section 3.2, App. A).
    The AttnLSTM's one-cell-per-square structure is architectural, not learned; the paper's comparison is between this supplied binding and the slot core's learned binding.
  • ad hoc to paper Proposition 4.1 asserts existence of parameters (ϕ, ρπ) under which message passing transports information for planning.
    The proposition is an existence argument relying on network capacity; no training guarantee is proven, and the paper does not show gradient descent will find these parameters.
  • domain assumption Improvement in solve rate with added test-time computation is a diagnostic of planning (from Guez et al. [9]).
    Used to interpret the behavioral gains in Figures 2 and the pondering protocol in Section 5.5; adopted from prior work.
invented entities (2)
  • State-graph binding (σ map) independent evidence
    purpose: A correspondence between environment states and hidden cells that the paper argues is necessary for planning to emerge.
    Defined in Def. 3.2 and operationally probed in Section 5.5 (binding read mass, stability, top-square coverage); it is a falsifiable construct with measurable failure modes.
  • Amortized transition model read from routing weights independent evidence
    purpose: The claim that attention weights ω constitute an implicit transition model p̂ without a separate learned dynamics model.
    Measured by comparing attention mass to graph distance and by wall-perturbation experiments; the causal intervention provides an outside handle.

pith-pipeline@v1.3.0-alltime-deepseek · 15571 in / 10493 out tokens · 114945 ms · 2026-08-01T14:55:50.444312+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning is conventionally divided into model-based and model-free methods. In this taxonomy, model-based methods perform lookahead planning over a learned world model, whereas model-free methods learn a reactive state-action mapping. Recent work, however, has shown that planning can emerge from model-free reinforcement learning alone. The conditions under which this behavior emerges from a pure reward-maximization objective have so far remained unclear. In this paper, we present evidence that, in the observed cases, the hidden-state structure of the neural architecture is the deciding factor. We find that a network of relational hidden states, each anchored to an environment state and exchanging messages along learned relations, acquires a planning mechanism. These hidden states recover the environment's transition structure in their learned relations, and improve the policy at decision time by planning over the learned graph. In a matched control agent that must additionally discover which cells represent which states, no such binding arises, and no planning follows from it. We argue that this explains the observed phenomenon of emergent planning in model-free reinforcement learning and raises the question of how common such emergent planning might be more generally. Finally, we hypothesize that the discovered mechanism could describe how planning emerges from pure reward maximization in the human brain through a neural architectural prior.

Figures

Figures reproduced from arXiv: 2607.18589 by Armin Sommer.

Figure 1
Figure 1. Figure 1: State-graph binding. Each Sokoban board square (left) is bound to one relational hidden cell (right) of the neural network via a fixed indexing (color-matched arrows and cells). 3.2 HIDDEN-STATE BINDING A relational core (Def. 3.1) carries content over a fixed pool of cells, but nothing in Eq. equation 4 forces those cells to mean anything. Suppose the network computes some statistic g(s) of a state, which… view at source ↗
Figure 2
Figure 2. Figure 2: Additional test-time computation improves solving. Solve rate on held-out Boxoban valid_medium. (a) For the pretrained DRC(3,3) of Taufeeque et al. [33], additional computation is provided by forced no-op pondering steps, following Guez et al. [9]. (b) For our AttnLSTM, additional computation is provided by increasing the number of inner recurrent thinking ticks before acting. Both agents improve with extr… view at source ↗
Figure 3
Figure 3. Figure 3: The attention routing is an amortized transition model. (a) A cell Q’s settled attention concentrates on feasible one-step successors (≈ 0.24 at one hop) and decays geometrically with graph distance (ρ ≈ 0.61). Its attention to cells made unreachable by a wall is approximately zero. (b) Turning a floor tile on the agent’s route into a wall shifts the current position’s hidden cell 2.3× more than turning an… view at source ↗
Figure 4
Figure 4. Figure 4: Thinking improves the decision only in the bound core. At each thinking step k, the greedy action is read from the intermediate hidden state through the model’s own trained actor head, on the initial observation of each of 512 held-out boards, and scored by whether the first move is goalward. We restrict to hard levels with agent-to-goal distance of 8–12 hops, where lookahead matters most. Here, the attent… view at source ↗
Figure 5
Figure 5. Figure 5: Each thinking step refines the decision. At every thinking step k of the AttnLSTM we read the policy from the intermediate state h k t through the trained actor head. The margin on the chosen action, the top-1 minus top-2 action logit, is shown normalized to its converged value (blue) and rises monotonically to 1, while the fraction of boards whose greedy action has not yet reached its settled value (grey)… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [1]

    Battaglia, Jessica B

    Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra...

    Pith/arXiv arXiv 2018

  2. [2]

    Interpreting emergent planning in model-free reinforcement learning

    Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga-Alonso, and David Krueger. Interpreting emergent planning in model-free reinforcement learning. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=DzGe40glxs

    2025

  3. [3]

    Cohen and Howard Eichenbaum.Memory, Amnesia, and the Hippocampal System

    Neal J. Cohen and Howard Eichenbaum.Memory, Amnesia, and the Hippocampal System. MIT Press, Cambridge, MA, 1993

    1993

  4. [4]

    Emergent response planning in LLMs, 2025

    Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. Emergent response planning in LLMs, 2025. URLhttps://arxiv.org/abs/2502.06258

    Pith/arXiv arXiv 2025

  5. [5]

    On the integration of space, time, and memory.Neuron, 95(5):1007–1018,

    Howard Eichenbaum. On the integration of space, time, and memory.Neuron, 95(5):1007–1018,

  6. [6]

    IM- PALA: Scalable distributed deep-RL with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of...

    2018

  7. [7]

    Hybrid computing using a neural network with dynamic external memory.Nature, 538(7626): 471–476, 2016

    Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwi´nska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hyb...

    doi:10.1038/nature20101 2016

  8. [8]

    Srivastava, and Jürgen Schmidhuber

    Klaus Greff, Rupesh K. Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation, 2017. URL https://arxiv.org/abs/1612.07771

    Pith/arXiv arXiv 2017

  9. [9]

    An investigation of model-free planning

    Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Re...

    2019

  10. [10]

    World models, 2018

    David Ha and Jürgen Schmidhuber. World models, 2018. URL https://arxiv.org/ abs/1803.10122

    Pith/arXiv arXiv 2018

  11. [11]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2555–2565. PMLR, 2019. URL https://proceedings.mlr. press/v97/hafne...

    2019

  12. [12]

    Mastering diverse domains through world models, 2023

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023. URLhttps://arxiv.org/abs/2301.04104

    Pith/arXiv arXiv 2023

  13. [13]

    Evidence of learned look-ahead in a chess-playing neural network

    Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network. InAdvances in Neural Information Processing Systems, volume 37, 2024. URL https://arxiv.org/abs/ 2406.00877. 11 Preprint

    Pith/arXiv arXiv 2024

  14. [14]

    An empirical exploration of recurrent network architectures

    Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. InProceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 2342–2350. PMLR,

  15. [15]

    Transformers are RNNs: Fast autoregressive transformers with linear attention, 2020

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv. org/abs/2006.16236

    Pith/arXiv arXiv 2020

  16. [16]

    Neural rela- tional inference for interacting systems

    Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural rela- tional inference for interacting systems. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 2688–2697. PMLR, 2018. URLhttps://proceedings.mlr.press/v80/kipf18a.html

    2018

  17. [17]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. InAdvances in Neural Information Processing Systems, volume 12, 1999. URL https: //proceedings.neurips.cc/paper_files/paper/1999/file/ 6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

    1999

  18. [18]

    Alex Konkel and Neal J. Cohen. Relational memory and the hippocampus: Representations and methods.Frontiers in Neuroscience, 3:166–174, 2009. doi: 10.3389/neuro.01.023.2009. URL https://www.frontiersin.org/journals/neuroscience/articles/ 10.3389/neuro.01.023.2009

    work page doi:10.3389/neuro.01.023.2009 2009

  19. [19]

    Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra

    Nathan O. Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. InProceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 ofProceedings of Machine Learning Research, pages 761–

  20. [20]

    Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg

    Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=DeG07_TcZvT

    2023

  21. [21]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Tom Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire...

    2025

  22. [22]

    Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming

    Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York, NY , 1994. doi: 10.1002/9780470316887

    doi:10.1002/9780470316887 1994

  23. [23]

    Hopfield networks is all you need, 2020

    Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi´c, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need, 2020. URL https://arxiv.org/abs/2008.02217

    Pith/arXiv arXiv 2020

  24. [24]

    Graph networks as learnable physics engines for inference and control

    Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Ried- miller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. InProceedings of the 35th International Conference on Machine Learn- ing, volume 80 ofProceedings of Machine Learning Research, pages 4470–4479. PMLR, 2...

    2018

  25. [25]

    Mastering Atari, Go, chess and shogi by planning with a learned model.Nature, 588:604–609, 2020

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model.Nature, 588:604–609, 2020. doi: 10.1038/s41586-020-03051-4. 12 Preprint

    doi:10.1038/s41586-020-03051-4 2020

  26. [26]

    High- dimensional continuous control using generalized advantage estimation, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2015. URL https: //arxiv.org/abs/1506.02438

    Pith/arXiv arXiv 2015

  27. [27]

    Convolutional LSTM network: A machine learning approach for pre- cipitation nowcasting

    Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang- chun Woo. Convolutional LSTM network: A machine learning approach for pre- cipitation nowcasting. InAdvances in Neural Information Processing Systems, vol- ume 28, 2015. URL https://proceedings.neurips.cc/paper/2015/hash/ 07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html

    2015

  28. [28]

    Mastering the game of Go without human knowledge.Nature, 550(7676):354–359,

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge.Nature, 550(7676):354–359,

  29. [29]

    Stachenfeld, Matthew M

    Kimberly L. Stachenfeld, Matthew M. Botvinick, and Samuel J. Gershman. The hippocampus as a predictive map.Nature Neuroscience, 20(11):1643–1653, 2017. doi: 10.1038/nn.4650

    doi:10.1038/nn.4650 2017

  30. [30]

    Richard S. Sutton. Learning to predict by the methods of temporal differences.Machine Learning, 3(1):9–44, 1988. doi: 10.1007/BF00115009

    doi:10.1007/bf00115009 1988

  31. [31]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2 edition, 2018

    2018

  32. [32]

    doi: 10.1038/nature24270

    doi:10.1038/nature24270

  33. [33]

    Planning in a recurrent neural network that plays Sokoban,

    Mohammad Taufeeque, Philip Quirke, Maximilian Li, Chris Cundy, Aaron David Tucker, Adam Gleave, and Adrià Garriga-Alonso. Planning in a recurrent neural network that plays Sokoban,

  34. [34]

    Path channels and plan extension kernels: A mechanistic description of planning in a Sokoban RNN

    Mohammad Taufeeque, Aaron David Tucker, Adam Gleave, and Adrià Garriga-Alonso. Path channels and plan extension kernels: A mechanistic description of planning in a Sokoban RNN. InMechanistic Interpretability Workshop at NeurIPS 2025, 2025. URL https: //openreview.net/forum?id=z025sIH4Nq

    2025

  35. [35]

    Edward C. Tolman. Cognitive maps in rats and men.Psychological Review, 55(4):189–208,

  36. [36]

    Sutton, David McAllester, Satinder Singh, and Yishay Mansour

    Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxima- tion. InAdvances in Neural Information Processing Systems, volume 12, pages 1057–1063. MIT Press, 2000. URL https://papers.nips.cc/paper/ 1713-policy-gradient-methods-for-reinforcement-learning-with-function-...

    2000

  37. [37]

    thinking

    Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Relational deep reinforcement learning, 2018. URLhttps://arxiv.org/abs/1806.01830. 13 Preprint A A...

    Pith/arXiv arXiv 2018

  38. [42]

    James C. R. Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy E. J. Behrens. The Tolman-Eichenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation.Cell, 183(5): 1249–1263.e23, 2020. doi: 10.1016/j.cell.2020.10.024

    doi:10.1016/j.cell.2020.10.024 2020

  39. [770]

    URL https://proceedings.mlr.press/v120/lambert20a

    PMLR, 2020. URL https://proceedings.mlr.press/v120/lambert20a. html

    2020

  40. [1948]

    doi: 10.1037/h0061626

    doi:10.1037/h0061626

  41. [2015]

    URLhttps://proceedings.mlr.press/v37/jozefowicz15.html

  42. [2017]

    doi: 10.1016/j.neuron.2017.06.036

    doi:10.1016/j.neuron.2017.06.036 2017

  43. [2024]

    URLhttps://arxiv.org/abs/2407.15421

    Pith/arXiv arXiv