Pith. sign in

REVIEW 4 major objections 4 minor 16 references

Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that fixed echo state networks make trainable recurrent memory unnecessary for memory-based reinforcement learning tasks, and that a locally connected reservoir performs best across all four tested benchmarks.

desk verdict A plausible and clearly written empirical case for fixed reservoirs in memory-based RL, but the fair-comparison claim is unverifiable until hyperparameters, code, and statistics are released. read the letter →

arxiv 2412.13093 v1 pith:BLGWF5ZY submitted 2024-12-17 cs.LG

Kevin McKee This is my paper · ORCID
classification cs.LG
keywords reservoircomputingechostatenetworkreinforcementlearningpartialobservabilitymemoryrecurrentneuralmeta-learningPOMDP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On four benchmark tasks that require holding past observations in memory—symbol recall, probabilistic bandits, sequential action planning, and a navigational water maze—this paper compares reinforcement learning agents whose recurrent memory is either a trainable RNN, GRU, LSTM, or a fixed echo state network (ESN). The paper finds that the ESNs, especially a locally connected variant, learn faster and with less variability than any trainable recurrent module, and that the ranking of models tracks the number of free parameters in the recurrent module. Because the reservoir's weights are never trained, the agent avoids backpropagation through time and instead learns only a feed-forward decoder from the reservoir state to actions and values. The author concludes that trainable recurrent memory provides no measurable benefit on these tasks and that fixed reservoirs are a promising memory substrate for meta-reinforcement learning at scale.

What carries the argument

The reservoir is an echo state network: a fixed recurrent layer with tanh units, sparse random input and recurrent weights, and a spectral radius set to 1.0 so that past inputs leave decaying traces without blowing up. The locally connected ESN variant (ESNLG) partitions hidden nodes into 'unique' nodes dedicated to each input and 'shared' nodes that integrate neighboring inputs, then sparsifies local recurrent weights and adds a small amount of global connectivity. This fixed dynamical system transforms the input stream into a high-dimensional nonlinear impulse response, and a trainable two-layer MLP decoder maps the reservoir state, together with the previous action and reward, to action probabilities and state values. The mechanism's work is to replace gradient-through-time learning of what to remember with a fixed, task-agnostic memory that the decoder can read directly.

What would settle it

Run the same four tasks with the RNN, GRU, and LSTM after an extensive hyperparameter sweep, reporting best-found configurations and matched parameter counts; if a tuned GRU reaches the ESNLG's learning speed and variability, the claim that trainable recurrent memory provides no benefit over fixed reservoirs would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a fixed, randomly wired echo state network, whose recurrent weights are sparsified and scaled to a spectral radius of 1.0, can serve as the entire memory system of a reinforcement learning agent on partially observable tasks, and that a locally connected version of this reservoir performs best in every comparison: it trains fastest on recall match, multi-armed bandit, water maze, and sequential bandit, and shows the least run-to-run variability. The paper interprets this as evidence that gradient-based training of recurrent connections is not just costly but actively harmful: gated units must first discover what to remember, whereas the reservoir already presents a compressed history of all recent inputs to a simple feed-forward decoder. The paper therefore claims that for these memory tasks, trainable RNNs offer no justification, since the simple RNN merely learned to approximate the same sparse, unit-spectral-radius configuration that the reservoir is given for free.

Load-bearing premise

The fairness of the model comparison: the trainable RNN, GRU, and LSTM were given roughly the same number of parameters as the ESNs and were not handicapped by poor hyperparameter choices, so the performance gap reflects the architectures rather than tuning effort.

Editorial extensions

If this is right

  • If the central claim holds, memory-based RL training can drop backpropagation through time entirely, reducing the training update to a short feed-forward step through the decoder.
  • The locally connected reservoir's consistent advantage suggests that spatial locality in the reservoir connectivity is a useful inductive bias for RL memory tasks, not just a speed optimization.
  • Because the ranking by training speed matches the number of free parameters in the recurrent module, the results imply that gating's extra parameters are a source of training inefficiency on these tasks rather than a benefit.
  • The results position fixed reservoirs as a practical memory module for large-scale meta-reinforcement learning, where an agent must explore many distributional tasks and needs a fast, general-purpose memory system.
  • The MLP's failure on recall match and slower bandit learning confirms that without a memory trace the tasks are not solvable, so the observed ESN performance is due to memory, not to the decoder alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not test: whether the ESN advantage persists on tasks with much longer memory horizons, where the reservoir's fixed decay timescale set by the spectral radius might become a bottleneck compared to gated units that can learn to hold information indefinitely.
  • A testable fairness check: rerun the comparison with a hyperparameter sweep over the GRU and LSTM, matching the reservoir's parameter count and giving them the same tuning effort; if a tuned gated unit closes the gap, the conclusion that trainable memory is unjustified would need qualification.
  • The parameter-count ranking suggests a simpler hypothesis: the advantage comes less from reservoir dynamics than from avoiding learned recurrent parameters; this could be tested by training a GRU with frozen, spectral-radius-normalized recurrent weights and comparing it to the ESN.
  • If the results scale, they imply that reservoir computing could be paired with non-recurrent transformer-style decoders for long-horizon RL memory, outsourcing memory to fixed dynamics and letting the trained module focus on policy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper compares fixed, untrained echo state networks (a standard ESN and a locally connected variant, ESNLG) against trainable recurrent modules (MLP baseline, RNN, GRU, LSTM) on four memory-based RL tasks: recall match, multi-armed bandit, sequential bandits, and water maze. All agents use the same actor-critic training algorithm; the recurrent module differs. The central claim is that reservoir computing greatly simplifies and speeds up RL on memory tasks, with the locally connected ESN performing best in every comparison, to the point that the author states "we cannot provide any justification for training RNNs." Results are reported from 8 runs per model using min-max intervals and means.

Significance. If the reported advantages are real, the paper would provide a practically valuable simplification for RL in partially observable environments and for meta-reinforcement learning, where memory modules are a bottleneck. The task set is well chosen to probe different memory demands, and the comparison of fixed reservoirs to trainable recurrent modules is a legitimate and underexplored question in this setting. However, the evidence as presented is preliminary: only 8 runs per model, no statistical significance tests, no code or data, and incomplete reporting of hyperparameters for the trainable baselines. The central claim is therefore plausible but not yet established at the level needed for a definitive publication.

major comments (4)
  1. [Appendix A, Table 1] Table 1 lists hyperparameters for the actor-critic, the ESN, and the locally connected ESN, but it does not list any hyperparameters for the RNN, GRU, or LSTM, such as hidden size, initialization scheme, or whether any sparsity or spectral regularization was applied. The paper states in the Discussion that "the hyperparameters were chosen per model to all have roughly the same number of parameters," but this claim is unverifiable without the recurrent hidden sizes for the trainable modules. Because the central comparison is between architectures, the fairness of this comparison is load-bearing and must be documented.
  2. [Results, Figure 3] All conclusions are based on 8 runs per model with min-max intervals and means, with no significance tests or confidence intervals. Statements such as "the ESNLG performed best in every comparison" and "the ESNLG showed the least variability" are stronger than the data support, especially when min-max intervals of different models overlap on several tasks. The authors should either add statistical testing (e.g., bootstrap confidence intervals or a paired test across seeds) or temper the claims to what the evidence supports.
  3. [Discussion, paragraph 2] The sentence "we cannot provide any justification for training RNNs" overreaches beyond the evidence. The paper itself acknowledges in the next sentence that the RNN "was not initialized to be sparse or to have a spectral radius of 1.0," which is an asymmetry in initialization rather than a demonstrated property of the architecture. Since the RNN hyperparameters are not reported, it is impossible to distinguish an inherent disadvantage of trainable RNNs from an undertuned baseline. This conclusion needs to be softened or supported by additional experiments with properly tuned and reported baselines.
  4. [Limitations, final paragraph] The Limitations section concedes that the ESNs "required many more nodes than the trainable recurrent modules" and that the study "did not seek to undertake extensive search." This is consistent with my concern that the parameter-count matching is not established and that the comparison may not reflect equal training effort. Without reporting the actual number of trainable parameters for every model and demonstrating that each baseline was reasonably tuned, the core claim that fixed reservoirs outperform trainable recurrent modules remains unconvincing.
minor comments (4)
  1. [Discussion, paragraph 3] The statement that "the results perfectly rank by the number of free parameters in the recurrent module" is an interesting observation, but it is not demonstrated quantitatively; consider reporting the parameter counts in a table and showing the ranking explicitly.
  2. [Appendix B] The connectivity specifications are clear, but the phrase "randomizing and sparsifying an RNN layer" in the Models section is vague; Appendix B does clarify it, so consider moving a pointer to the appendix earlier.
  3. [Figure 3] The caption says "Colored intervals show the minimum to maximum reward per time step," but the text refers to "episodes" and "training time"; please unify the x-axis terminology (steps vs. episodes) across the figures and text.
  4. [Introduction, paragraph 1] The phrase "rely on chance initialization of weights to produce an initial memory signal" is an oversimplification; gated units also rely on learned gates, not only initial weights. Consider rewording to avoid seeming to dismiss the trained components.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported ESN advantage is an empirical benchmark result, not a quantity derived from fitted inputs or self-citation.

full rationale

The paper is an empirical comparison of memory architectures on POMDP-style RL tasks. There is no derivation chain in which a predicted quantity is constructed from the data it is supposed to explain. The ESN and locally connected ESN weight matrices are fixed random constructions with specified spectral radius and connectivity, and those settings are hyperparameters chosen before training; they are not fitted to the task outcomes. The reported training-speed and variability advantages are measured results from repeated runs, not quantities forced by definition. References to Lukoševičius and Jaeger for reservoir computing principles are external methodological citations, not self-citations carrying the central claim. The paper's assertion that hyperparameters were chosen so models have roughly the same number of parameters is not demonstrated in Appendix A, and the RNN, GRU, and LSTM hyperparameters are not reported; however, this is a fairness and reporting concern about the benchmark, not a circularity in the argument. The Limitations section explicitly acknowledges that only a few reasonable hyperparameter values were considered and no extensive search was undertaken, which further confirms that the comparison is empirical rather than definitional. No prediction in the paper reduces by construction to a fitted parameter, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

No derivation is attempted; the claim is empirical. The listed free parameters are hand-selected hyperparameters that influence the reported performance ranking. The axioms are domain assumptions about task representativeness and fair model comparison.

free parameters (7)
  • Spectral radius (phi) = 1.0
    Hand-chosen to set reservoir's dynamic regime and memory; affects all ESN results.
  • Global connection probability PG(W) = 40% (ESN), 1% (local ESN)
    Determines reservoir connectivity; selected by hand, not tuned.
  • Local connection probability PL(W) = 50%
    Controls sparsification of local recurrent weights.
  • Input connection probability PI(W) = 40%/50%
    Masks input weights to reservoir; hand-set.
  • NUnique / NShared = 20 / 10
    Defines local input projection structure; hand-set.
  • Learning rate = 0.0003
    Actor-critic learning rate; common default, not task-tuned.
  • Hidden units per MLP layer = 32-62
    Decoder size; chosen to match parameter counts approximately.
assumptions (3)
  • domain assumption The four diagnostic tasks are representative of a 'wide variety of memory-based problems' (Introduction).
    Used to generalize from task-specific results to a broad claim about memory tasks and meta-learning.
  • domain assumption Hyperparameter choices give a fair comparison across models, with 'roughly the same number of parameters' (Discussion).
    The paper asserts parity but does not provide hyperparameters for RNN, GRU, and LSTM, so the fairness is unverifiable.
  • standard math ESN with spectral radius 1.0 and sparsified connectivity satisfies the echo state property.
    Implicitly relies on known reservoir-computing theory (Jaeger) that these settings yield stable dynamics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks." pith.science (2026). https://pith.science/paper/BLGWF5ZY

@misc{pith2026241213093,
  author       = {Pith},
  title        = {Pith review of: Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLGWF5ZY}},
  note         = {Machine review of arXiv:2412.13093}
}
read the original abstract

Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable recurrent hidden layers, often with gated memory. Reservoir computing presents an alternative, in which a recurrent layer is not trained, but rather has a set of fixed, sparse recurrent weights. The weights are scaled to produce stable dynamical behavior such that the reservoir state contains a high-dimensional, nonlinear impulse response function of the inputs. An output decoder network can then be used to map the compressive history represented by the reservoir's state to any outputs, including agent actions or predictions. In this study, we find that reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks by (1) eliminating the need for backpropagation of gradients through time, (2) presenting all recent history simultaneously to the downstream network, and (3) performing many useful and generic nonlinear computations upstream from the trained modules. In particular, these findings offer significant benefit to meta-learning that depends primarily on efficient and highly general memory systems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning.nature, 518 (7540):529–533, 2015

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Os- trovski, et al. Human-level control through deep reinforcement learning.nature, 518 (7540):529–533, 2015

    work page 2015

  2. [2]

    Learning phrase representa- tions using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representa- tions using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    arXiv 2014

  3. [3]

    Long short-term memory.Neural compu- tation, 9(8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural compu- tation, 9(8):1735–1780, 1997

    work page 1997

  4. [4]

    Reservoir computing approaches to recurrent neural network training.Computer science review, 3(3):127–149, 2009

    Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural network training.Computer science review, 3(3):127–149, 2009

    work page 2009

  5. [5]

    Recent advances in physical reservoir computing: A review.Neural Networks, 115:100–123, 2019

    Gouhei Tanaka, Toshiyuki Yamane, Jean Benoit Héroux, Ryosho Nakane, Naoki Kanazawa, Seiji Takeda, Hidetoshi Numata, Daiju Nakano, and Akira Hirose. Recent advances in physical reservoir computing: A review.Neural Networks, 115:100–123, 2019

    2019

  6. [6]

    Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication.science, 304(5667):78–80, 2004

    Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication.science, 304(5667):78–80, 2004

    2004

  7. [7]

    Short term memory in echo state networks

    Herbert Jaeger. Short term memory in echo state networks. 2001. REINFORCEMENT LEARNING WITH RESERVOIR COMPUTERS 8

    work page 2001

  8. [8]

    echo state

    Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note.Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13, 2001

    work page 2001

Show all 16 references
  1. [9]

    Real-time computing withoutstablestates: Anewframeworkforneuralcomputationbasedonperturbations

    Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing withoutstablestates: Anewframeworkforneuralcomputationbasedonperturbations. Neural computation, 14(11):2531–2560, 2002

    2002

  2. [10]

    Pattern recognition in a bucket

    Chrisantha Fernando and Sampsa Sojakka. Pattern recognition in a bucket. InAd- vances in Artificial Life: 7th European Conference, ECAL 2003, Dortmund, Germany, September 14-17, 2003. Proceedings 7, pages 588–597. Springer, 2003

    2003

  3. [11]

    Popgym: Benchmarking partially observable reinforcement learning

    StevenMorad, RyanKortvelesy, MatteoBettini, StephanLiwicki, andAmandaProrok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.01859, 2023

    2023 arXiv

  4. [12]

    Reservoirs learn to learn

    Anand Subramoney, Franz Scherr, and Wolfgang Maass. Reservoirs learn to learn. Reservoir Computing: Theory, Physical Implementations, and Applications, pages 59– 76, 2021

    2021

  5. [13]

    Place navigation impaired in rats with hippocampal lesions.Nature, 297(5868):681–683, 1982

    Richard GM Morris, Paul Garrud, JNP al Rawlins, and John O’Keefe. Place navigation impaired in rats with hippocampal lesions.Nature, 297(5868):681–683, 1982

    1982

  6. [14]

    MIT press, 2018

    Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction. MIT press, 2018

    2018

  7. [15]

    Prefrontal cortex as a meta-reinforcement learning system.Nature neuroscience, 21(6):860–868, 2018

    Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system.Nature neuroscience, 21(6):860–868, 2018

    2018

  8. [16]

    Pytorch: An imperative style, high-performance deep learning library.Advances in neural in- formation processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library.Advances in neural in- formation processing systems, 32...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.