{"id":"37183d61-1623-4291-b1a1-ab2ff119ae36","arxiv_id":"2412.13093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fixed, untrained reservoirs outperform trained recurrent networks on four memory-based reinforcement learning tasks, with local connectivity giving the fastest learning.","lead":"This paper tests whether fixed, untrained recurrent networks called reservoirs can replace trainable memory cells in reinforcement learning agents. It finds that reservoirs, especially a locally connected version, learn memory tasks faster and more reliably than GRUs and LSTMs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fair-comparison claim is unverified and possibly violated: ESNs are initialized with spectral radius 1.0 and custom connectivity, while RNN/GRU/LSTM hyperparameters are unreported and likely untuned, so the reported performance gap may be an artifact of initialization asymmetry.","rationale":"I read the paper in good faith as an empirical comparison of fixed reservoir memory against trainable recurrent modules for RL in POMDPs. The experiments are clearly described and the consistent pattern of ESNs training faster is plausible given prior reservoir computing literature. However, the central claim requires that the comparison be fair, and the manuscript does not establish this. The reader's weakest assumption identified exactly this: missing hyperparameters for RNN/GRU/LSTM and no evidence of tuning. My concern sharpens it: the ESN is constructed with a deliberately chosen spectral radius of 1.0 and custom sparse local connectivity, while the trainable recurrent modules are said not to have such initialization. That is a concrete asymmetry that could explain the performance gap independently of the fixed-vs-trainable distinction. The claim 'we cannot provide any justification for training RNNs' is an overgeneralization from this setup. I also note the absence of statistical tests and the small number of seeds, which supports the need for a conditional verdict pending additional evidence. I do not see an internal inconsistency or a statistical error in the reported figures; the concern is about external validity and experimental control. Therefore the appropriate verdict remains CONDITIONAL, as the reader recommended, and I see no reason to change it. The concrete test I propose—a hyperparameter and initialization sweep with matched parameter counts and adequate seeds—would directly settle whether the reported advantage is robust.","tokens_in":6041,"tokens_out":2710,"duration_ms":27693,"concrete_test":"Release the full hyperparameter table for all models and the training code. Then run a controlled sweep for RNN, GRU, and LSTM over hidden size (matched to the ESN's effective recurrent parameters), initialization scheme (including orthogonal and spectral-radius-scaled initializations), and learning rate, using at least 30 seeds per configuration and reporting a significance test (e.g., bootstrap or Mann-Whitney U) comparing the best tuned baseline to the ESN and ESNLG on all four tasks. If a tuned trainable baseline matches or beats the ESN on any task, the central claim is an artifact of undertuning; if the ESN still dominates after the sweep, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that fixed ESNs, especially the locally connected variant, outperform trainable recurrent modules on memory-based RL tasks, to the point that training RNNs is unjustified. This claim depends on the comparisons being fair across architectures. Appendix A lists hyperparameters only for the actor-critic, ESN, and locally connected ESN; it does not report hidden sizes, initialization, or other settings for the RNN, GRU, and LSTM. The author states that hyperparameters were chosen so models have 'roughly the same number of parameters,' but without the recurrent hidden sizes for trainable modules this is unverifiable. More concretely, the ESN recurrent weights are deliberately constructed with spectral radius 1.0, sparse connectivity, and local structure, whereas the paper admits the RNN 'was not initialized to be sparse or to have a spectral radius of 1.0.' This asymmetry is not a neutral property of the architectures: the ESN starts at a well-tuned dynamical operating point, while trainable models start from default random initializations and must simultaneously learn both the memory dynamics and the policy. The observed speed advantage of ESNs could therefore reflect unequal starting conditions and untuned baselines rather than an intrinsic benefit of reservoir computing. The claim 'we cannot provide any justification for training RNNs' overreaches beyond the evidence, especially with only 8 runs per model, min-max intervals rather than significance tests, and no code or data release. This is the load-bearing concern because if the trainable baselines were properly tuned and parameter-matched, the headline result could weaken or disappear.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares fixed, untrained echo state networks (a standard ESN and a locally connected variant, ESNLG) against trainable recurrent modules (MLP baseline, RNN, GRU, LSTM) on four memory-based RL tasks: recall match, multi-armed bandit, sequential bandits, and water maze. All agents use the same actor-critic training algorithm; the recurrent module differs. The central claim is that reservoir computing greatly simplifies and speeds up RL on memory tasks, with the locally connected ESN performing best in every comparison, to the point that the author states \"we cannot provide any justification for training RNNs.\" Results are reported from 8 runs per model using min-max intervals and means.","tokens_in":6404,"tokens_out":1990,"duration_ms":19422,"significance":"If the reported advantages are real, the paper would provide a practically valuable simplification for RL in partially observable environments and for meta-reinforcement learning, where memory modules are a bottleneck. The task set is well chosen to probe different memory demands, and the comparison of fixed reservoirs to trainable recurrent modules is a legitimate and underexplored question in this setting. However, the evidence as presented is preliminary: only 8 runs per model, no statistical significance tests, no code or data, and incomplete reporting of hyperparameters for the trainable baselines. The central claim is therefore plausible but not yet established at the level needed for a definitive publication.","major_comments":[{"comment":"Table 1 lists hyperparameters for the actor-critic, the ESN, and the locally connected ESN, but it does not list any hyperparameters for the RNN, GRU, or LSTM, such as hidden size, initialization scheme, or whether any sparsity or spectral regularization was applied. The paper states in the Discussion that \"the hyperparameters were chosen per model to all have roughly the same number of parameters,\" but this claim is unverifiable without the recurrent hidden sizes for the trainable modules. Because the central comparison is between architectures, the fairness of this comparison is load-bearing and must be documented.","section":"Appendix A, Table 1"},{"comment":"All conclusions are based on 8 runs per model with min-max intervals and means, with no significance tests or confidence intervals. Statements such as \"the ESNLG performed best in every comparison\" and \"the ESNLG showed the least variability\" are stronger than the data support, especially when min-max intervals of different models overlap on several tasks. The authors should either add statistical testing (e.g., bootstrap confidence intervals or a paired test across seeds) or temper the claims to what the evidence supports.","section":"Results, Figure 3"},{"comment":"The sentence \"we cannot provide any justification for training RNNs\" overreaches beyond the evidence. The paper itself acknowledges in the next sentence that the RNN \"was not initialized to be sparse or to have a spectral radius of 1.0,\" which is an asymmetry in initialization rather than a demonstrated property of the architecture. Since the RNN hyperparameters are not reported, it is impossible to distinguish an inherent disadvantage of trainable RNNs from an undertuned baseline. This conclusion needs to be softened or supported by additional experiments with properly tuned and reported baselines.","section":"Discussion, paragraph 2"},{"comment":"The Limitations section concedes that the ESNs \"required many more nodes than the trainable recurrent modules\" and that the study \"did not seek to undertake extensive search.\" This is consistent with my concern that the parameter-count matching is not established and that the comparison may not reflect equal training effort. Without reporting the actual number of trainable parameters for every model and demonstrating that each baseline was reasonably tuned, the core claim that fixed reservoirs outperform trainable recurrent modules remains unconvincing.","section":"Limitations, final paragraph"}],"minor_comments":[{"comment":"The statement that \"the results perfectly rank by the number of free parameters in the recurrent module\" is an interesting observation, but it is not demonstrated quantitatively; consider reporting the parameter counts in a table and showing the ranking explicitly.","section":"Discussion, paragraph 3"},{"comment":"The connectivity specifications are clear, but the phrase \"randomizing and sparsifying an RNN layer\" in the Models section is vague; Appendix B does clarify it, so consider moving a pointer to the appendix earlier.","section":"Appendix B"},{"comment":"The caption says \"Colored intervals show the minimum to maximum reward per time step,\" but the text refers to \"episodes\" and \"training time\"; please unify the x-axis terminology (steps vs. episodes) across the figures and text.","section":"Figure 3"},{"comment":"The phrase \"rely on chance initialization of weights to produce an initial memory signal\" is an oversimplification; gated units also rely on learned gates, not only initial weights. Consider rewording to avoid seeming to dismiss the trained components.","section":"Introduction, paragraph 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and potentially useful question, but the empirical evidence is not yet at the level required for publication. The missing RNN/GRU/LSTM hyperparameters and the lack of significance testing are fixable with additional experiments and reporting, which is why I recommend major revision rather than rejection. The author should also consider providing code and seeds to enable reproducibility, and should temper the concluding claim about RNNs being unjustifiable unless the baselines are demonstrably well tuned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kevin McKee's paper is a straightforward empirical comparison of fixed echo state networks against trainable recurrent modules (RNN, GRU, LSTM) on four memory-based RL tasks. The core idea is not new, and the paper says so. What is new is the systematic comparison across four diagnostic tasks and the locally connected ESN variant, which consistently trains faster and with less variability. That result is plausible and worth taking seriously.\n\nThe paper does several things well. The tasks are well chosen to isolate different memory requirements, the agent architecture is shared across all conditions, and the limitations section is honest about what is not tested. The claim that ESNs eliminate backprop-through-time and present history in a compressed form is accurate, and the empirical pattern across tasks is consistent.\n\nThe soft spots are real and load-bearing. Appendix A lists hyperparameters only for the actor-critic and the ESNs; the RNN, GRU, and LSTM hidden sizes, initializations, and tuning are not reported. The author says models were set to 'roughly the same number of parameters' but this is unverifiable. More importantly, the ESN recurrent weights are deliberately initialized with spectral radius 1.0 and sparse connectivity, while the paper admits the RNN 'was not initialized to be sparse or to have a spectral radius of 1.0.' That is an unequal starting point. The ESN starts at a well-tuned dynamical operating point; the trainable models start at random and must learn both memory dynamics and policy. The observed speed advantage could partly reflect this asymmetry rather than an intrinsic property of reservoirs.\n\nAlso, each model ran only 8 times, with min-max intervals and no significance tests. No code or data are provided. The conclusion that 'we cannot provide any justification for training RNNs' overreaches beyond this evidence. The paper's own limitations section acknowledges the hyperparameter search was limited, but the discussion does not carry that caution into the central claim.\n\nStill, I do not think this is a bad paper. The experiments are clean enough to be reproducible if the missing details are supplied, and the consistent ordering across tasks is suggestive. The right venue for this is peer review with mandatory code/data release and full hyperparameter tables. I would not desk-reject it. A serious referee could push for proper statistical analysis and a fairer baseline tuning protocol, and the result—if it survives that—would be a useful default recommendation for memory modules in RL.\n\nWho should read it: researchers working on memory-based RL or meta-RL who are deciding between gated recurrent units and cheaper alternatives. I would not cite it in its current form because the comparison is not yet verified, but I would bring it to our reading group and want to see the revision.","headline":"A plausible and clearly written empirical case for fixed reservoirs in memory-based RL, but the fair-comparison claim is unverifiable until hyperparameters, code, and statistics are released.","tokens_in":6910,"tokens_out":895,"would_cite":false,"duration_ms":10400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that fixed echo state networks make trainable recurrent memory unnecessary for memory-based reinforcement learning tasks, and that a locally connected reservoir performs best across all four tested benchmarks.","keywords":["reservoir computing","echo state network","reinforcement learning","partial observability","memory","recurrent neural network","meta-learning","POMDP"],"falsifier":"Run the same four tasks with the RNN, GRU, and LSTM after an extensive hyperparameter sweep, reporting best-found configurations and matched parameter counts; if a tuned GRU reaches the ESNLG's learning speed and variability, the claim that trainable recurrent memory provides no benefit over fixed reservoirs would be falsified.","tokens_in":5750,"feed_emoji":"🧠","tokens_out":7081,"duration_ms":60080,"temperature":0.7,"pith_summary":"On four benchmark tasks that require holding past observations in memory—symbol recall, probabilistic bandits, sequential action planning, and a navigational water maze—this paper compares reinforcement learning agents whose recurrent memory is either a trainable RNN, GRU, LSTM, or a fixed echo state network (ESN). The paper finds that the ESNs, especially a locally connected variant, learn faster and with less variability than any trainable recurrent module, and that the ranking of models tracks the number of free parameters in the recurrent module. Because the reservoir's weights are never trained, the agent avoids backpropagation through time and instead learns only a feed-forward decoder from the reservoir state to actions and values. The author concludes that trainable recurrent memory provides no measurable benefit on these tasks and that fixed reservoirs are a promising memory substrate for meta-reinforcement learning at scale.","feed_headline":"Fixed reservoirs outlearn trained RNNs on memory tasks","feed_subtitle":"No backprop through time: a fixed reservoir learns memory tasks faster and with less variance.","key_machinery":"The reservoir is an echo state network: a fixed recurrent layer with tanh units, sparse random input and recurrent weights, and a spectral radius set to 1.0 so that past inputs leave decaying traces without blowing up. The locally connected ESN variant (ESNLG) partitions hidden nodes into 'unique' nodes dedicated to each input and 'shared' nodes that integrate neighboring inputs, then sparsifies local recurrent weights and adds a small amount of global connectivity. This fixed dynamical system transforms the input stream into a high-dimensional nonlinear impulse response, and a trainable two-layer MLP decoder maps the reservoir state, together with the previous action and reward, to action probabilities and state values. The mechanism's work is to replace gradient-through-time learning of what to remember with a fixed, task-agnostic memory that the decoder can read directly.","core_discovery":"The central discovery is that a fixed, randomly wired echo state network, whose recurrent weights are sparsified and scaled to a spectral radius of 1.0, can serve as the entire memory system of a reinforcement learning agent on partially observable tasks, and that a locally connected version of this reservoir performs best in every comparison: it trains fastest on recall match, multi-armed bandit, water maze, and sequential bandit, and shows the least run-to-run variability. The paper interprets this as evidence that gradient-based training of recurrent connections is not just costly but actively harmful: gated units must first discover what to remember, whereas the reservoir already presents a compressed history of all recent inputs to a simple feed-forward decoder. The paper therefore claims that for these memory tasks, trainable RNNs offer no justification, since the simple RNN merely learned to approximate the same sparse, unit-spectral-radius configuration that the reservoir is given for free.","pith_inferences":["A direct extension the paper does not test: whether the ESN advantage persists on tasks with much longer memory horizons, where the reservoir's fixed decay timescale set by the spectral radius might become a bottleneck compared to gated units that can learn to hold information indefinitely.","A testable fairness check: rerun the comparison with a hyperparameter sweep over the GRU and LSTM, matching the reservoir's parameter count and giving them the same tuning effort; if a tuned gated unit closes the gap, the conclusion that trainable memory is unjustified would need qualification.","The parameter-count ranking suggests a simpler hypothesis: the advantage comes less from reservoir dynamics than from avoiding learned recurrent parameters; this could be tested by training a GRU with frozen, spectral-radius-normalized recurrent weights and comparing it to the ESN.","If the results scale, they imply that reservoir computing could be paired with non-recurrent transformer-style decoders for long-horizon RL memory, outsourcing memory to fixed dynamics and letting the trained module focus on policy."],"forward_implications":["If the central claim holds, memory-based RL training can drop backpropagation through time entirely, reducing the training update to a short feed-forward step through the decoder.","The locally connected reservoir's consistent advantage suggests that spatial locality in the reservoir connectivity is a useful inductive bias for RL memory tasks, not just a speed optimization.","Because the ranking by training speed matches the number of free parameters in the recurrent module, the results imply that gating's extra parameters are a source of training inefficiency on these tasks rather than a benefit.","The results position fixed reservoirs as a practical memory module for large-scale meta-reinforcement learning, where an agent must explore many distributional tasks and needs a fast, general-purpose memory system.","The MLP's failure on recall match and slower bandit learning confirms that without a memory trace the tasks are not solvable, so the observed ESN performance is due to memory, not to the decoder alone."],"supporting_citations":[{"why":"Supplies the echo state network construction principles—sparsification and spectral radius scaling—that the paper's fixed reservoirs are built from.","marker":"[4]"},{"why":"Defines the GRU, the gated recurrent baseline the ESNs are compared against.","marker":"[2]"},{"why":"Defines the LSTM, the other gated recurrent baseline in the comparison.","marker":"[3]"},{"why":"The prior benchmark study that concluded GRUs performed best on partially observable RL tasks; the paper's comparison targets that conclusion.","marker":"[11]"},{"why":"Provides the actor-critic training algorithm and entropy regularization used uniformly across all models.","marker":"[14]"},{"why":"Frames meta-reinforcement learning as memory-based exploration, which motivates the paper's claim that efficient reservoir memory benefits meta-learners.","marker":"[15]"},{"why":"Defines the water-maze navigation task, one of the four benchmark tasks the paper uses to compare memory modules.","marker":"[13]"}],"fun_headline_variants":["Skip backprop: fixed reservoir crushes memory RL","Reservoir memory: train only outputs, not recurrent weights","Echo state net: free memory for RL agents","Sparse fixed weights beat trained RNNs on memory tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fairness of the model comparison: the trainable RNN, GRU, and LSTM were given roughly the same number of parameters as the ESNs and were not handicapped by poor hyperparameter choices, so the performance gap reflects the architectures rather than tuning effort.","fun_headline_variants_meta":{"raw":{"variants":["Skip backprop: fixed reservoir crushes memory RL","Reservoir memory: train only outputs, not recurrent weights","Echo state net: free memory for RL agents","Sparse fixed weights beat trained RNNs on memory tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1701,"prompt_tokens":914,"completion_tokens":787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":720}},"tokens_in":530,"tokens_out":787,"duration_ms":7774,"temperature":1.0,"reasoning_tokens":720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:25:41.770965+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four tasks with the RNN, GRU, and LSTM after an extensive hyperparameter sweep, reporting best-found configurations and matched parameter counts; if a tuned GRU reaches the ESNLG's learning speed and variability, the claim that trainable recurrent memory provides no benefit over fixed reservoirs would be falsified.","supporting_citations":[{"cited_title":"Reservoir computing approaches to recurrent neural network training.Computer science review, 3(3):127–149, 2009","cited_arxiv_id":null,"evidence_quote":"Supplies the echo state network construction principles—sparsification and spectral radius scaling—that the paper's fixed reservoirs are built from."},{"cited_title":"Long short-term memory.Neural compu- tation, 9(8):1735–1780, 1997","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM, the other gated recurrent baseline in the comparison."},{"cited_title":"Prefrontal cortex as a meta-reinforcement learning system.Nature neuroscience, 21(6):860–868, 2018","cited_arxiv_id":null,"evidence_quote":"Frames meta-reinforcement learning as memory-based exploration, which motivates the paper's claim that efficient reservoir memory benefits meta-learners."},{"cited_title":"Place navigation impaired in rats with hippocampal lesions.Nature, 297(5868):681–683, 1982","cited_arxiv_id":null,"evidence_quote":"Defines the water-maze navigation task, one of the four benchmark tasks the paper uses to compare memory modules."}],"review_version":1}