{"id":"fc3a99a9-2040-489f-8b99-cb48670d9da8","arxiv_id":"2412.15292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Deep RL agents with a log-compressed, scale-invariant memory (CogRNN) learn at similar rates across rescaled time intervals, unlike LSTM and RNN baselines.","lead":"The authors add a neuroscience-inspired, scale-invariant memory module called CogRNN to deep reinforcement learning agents and test whether it learns equally well across different timing scales. The result suggests AI agents can handle rescaled time intervals without per-scale retuning, which matters for real-world tasks with variable timing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Discounting-induced gradient scale differences make the reported scale-invariant learning speed implausible unless an unreported normalization is used.","rationale":"The paper's central claim is that the CogRNN memory makes learning speed invariant to temporal scale. The formal derivation of the memory's translation-covariance is sound, and the comparison of neural activity to time cells is a useful independent contribution. However, the empirical claim about learning speed depends on the full RL update, and the reported hyperparameters appear to make the long-scale task's reward signal exponentially smaller than the short-scale task's. The reader's weakest_assumption identified the RL algorithm's scale-breaking components; I sharpen this to the concrete numerical consequence of γ=0.98 on return magnitudes. If the released code contains no normalization, the observed similarity in learning curves likely stems from an undocumented mechanism or from learning-rate selection that masks the difference. This is not a rejection of the idea, but a demand for transparency: the claim should be reproducible from the stated hyperparameters alone. The concrete test directly inspects the training signal and would settle whether the concern lands. I keep the CONDITIONAL verdict because the issue is testable and the theoretical framework remains plausible; no change from the reader's verdict is needed at this stage.","tokens_in":15134,"tokens_out":9069,"duration_ms":68094,"concrete_test":"Run the released implementation (github.com/cogneuroai/RL-with-scale-invariant-memory) for the 1D interval timing task at step sizes 10 and 100 using the reported γ=0.98, λ=0.95 and the selected learning rates, and log the mean absolute GAE advantage over the first 10k trials. If the long-scale advantages are more than an order of magnitude smaller yet the learning curves overlap, identify the compensating mechanism in the code; if the curves separate by a factor matching the return ratio, the central claim is not supported. As an auxiliary check, retrain with advantage normalization and see if the across-scale curves collapse for both CogRNN and LSTM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"With step size 10 the intervals are 300–480 steps; with step size 100 they are 30–48 steps. The fixed discount factor γ=0.98 makes the reward at the end of a trial discounted by γ^delay, giving mean returns of ~0.002–0.0001 for the long scale versus ~0.3–0.38 for the short scale. Policy-gradient updates are proportional to the advantage magnitude, so the long-scale effective learning signal is 2–3 orders of magnitude smaller. The paper does not report any advantage normalization, reward scaling, or per-scale learning-rate adjustment, and the Discussion merely acknowledges that 'RL algorithms such as A2C contain components that break scale invariance.' Therefore the observation that learning speed is 'similar at different temporal scales' (Fig. 4) is surprising and, as presented, internally inconsistent with the stated hyperparameters. The CogRNN memory changes the observation representation but cannot change the scale of the scalar return. Unless some compensating mechanism exists in the code (e.g., reward normalization or advantage whitening), the central claim may be an artifact of unreported training details.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes integrating a computational neuroscience model of scale-invariant memory (CogRNN) into deep RL agents. The memory is built from an approximate Laplace transform with log-spaced time constants, and the paper provides a theoretical argument that temporal rescaling becomes a translation in the memory representation; adding convolution and pooling makes the representation invariant to rescaling. The authors compare CogRNN with LSTM and RNN cores on interval timing (1D and 3D), interval discrimination, delayed-match-to-sample, and interval reproduction tasks, varying task-relevant intervals via a step-size parameter. They report that CogRNN agents reach high performance and learn at similar speeds across temporal scales, while LSTM/RNN agents show scale-dependent learning. They also show zero-shot transfer of a CogRNN-based policy to rescaled environments and report that CogRNN time-cell-like activity exhibits scale-invariant width growth.","tokens_in":15334,"tokens_out":3075,"duration_ms":24413,"significance":"If the central empirical claim holds, the paper would make a valuable contribution by demonstrating that a biologically inspired, log-compressed memory representation can decouple RL learning difficulty from temporal scale within a designed range, addressing a real limitation of standard recurrent cores like LSTM and RNN. The theoretical derivation of scale covariance is clear and the zero-shot transfer experiment is a concrete, falsifiable demonstration of the designed property. The paper also provides open-source code. However, the significance currently hinges on a qualitative reading of learning curves and on the assumption that no unreported normalization compensates for discounting-induced gradient scale differences; until those points are resolved, the strength of the claim is not fully established.","major_comments":[{"comment":"The paper reports a fixed discount factor gamma=0.98 for all agents and scales. In the interval timing task, with step size 10 the intervals are 300-480 steps, while with step size 100 they are 30-48 steps. For a correct-trial reward of 1.0 at the end of the trial, the discounted return is roughly gamma^delay: about 0.3-0.5 for the short scale and below 0.002 for the long scale. Since A2C policy-gradient updates scale with advantage magnitude, one would expect the long-scale learning signal to be orders of magnitude smaller, making similar learning speeds implausible unless advantage normalization, reward scaling, or per-scale learning-rate adjustment is applied. The paper does not report any such normalization or scaling. Please specify whether advantages are normalized (e.g., by subtracting the mean and dividing by the standard deviation), whether rewards are scaled, or whether any other mechanism compensates for the discount-induced gradient scale difference. If none is used, please provide a concrete explanation for the observed similarity in learning speeds.","section":"Hyperparameters and training"},{"comment":"The central claim that 'the speed of learning was similar at different temporal scales' for CogRNN agents is based on visual inspection of learning curves in Fig. 4. The manuscript does not provide a quantitative measure such as trials-to-threshold, time-to-criterion, or a statistical comparison (e.g., confidence intervals for learning speed, or a test of whether final performance differs across scales). Please add a quantitative analysis that supports the claim, including error bars and a statistical test if possible.","section":"Results (Fig. 4)"},{"comment":"The text states that the authors 'selected the best-performing learning rates for each agent' among 0.001, 0.0001, and 0.00001, but the selected values are not reported. If different learning rates were used for different architectures or different temporal scales, the comparison in Fig. 4 could be confounded. Please report the selected learning rates for each architecture and each environment/scale, or state explicitly that the same learning rate was used across scales for each architecture.","section":"Hyperparameters and training"},{"comment":"The Discussion acknowledges that A2C contains components that break scale invariance, including exponential discounting, horizon, and rollout length, but the paper does not attempt to isolate how the CogRNN memory representation overcomes the scale dependence of the RL algorithm itself. This is important because the central claim is about learning speed, not just representational invariance. I suggest adding an experiment or at least a more detailed analysis of why the memory's invariance dominates the RL algorithm's scale dependence, for example by comparing with a CogRNN agent using power-law discounting, or by ablating the normalization mechanism if one exists.","section":"Discussion"}],"minor_comments":[{"comment":"In Eq. (5), the impulse response formula includes a factor 1/t and a power (t/tau*)^(k+1); the notation is a bit dense. It would help to state explicitly that the peak time equals tau* after the derivative computation, since the current text says 'equate it with 0' without showing the intermediate step.","section":"Model (Scale invariant memory network)"},{"comment":"The interval reproduction results are only presented in the Supplemental Information (Table S1, S2, Fig. S11, S12). Since the reproduction task supports the multi-scale learning claim, consider summarizing at least one key quantitative result in the main text.","section":"Results"},{"comment":"The sentence 'We updated the model parameters after each trial instead of backpropagating the gradients after horizon number of steps' is ambiguous: does this mean the effective horizon is the full trial length, or that gradients are accumulated over a trial? Please clarify the update schedule and its relation to the 'horizon' hyperparameter in Fig. S10.","section":"Hyperparameters and training"},{"comment":"In the 3D interval timing panel, the CogRNN and LSTM curves appear to plateau at different levels; adding a legend or annotation clarifying the final performance values would improve readability.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is appealing and the engineering of scale-invariant memory is clearly presented. However, the main empirical claim currently rests on visual inspection and on an implicit assumption about gradient scaling that is contradicted by the stated hyperparameters. I would need to see either a clear statement of advantage normalization/per-scale learning-rate adjustment or a direct measurement of learning speed before endorsing publication. The paper is within scope for the journal and likely to be of interest to the AI and cognitive-science communities if these issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid, useful integration of a known scale-invariant memory model into online deep RL, with a clean multi-task comparison against LSTM and RNN. The main claim—CogRNN gives roughly scale-invariant learning speed—is plausible but not fully nailed down, because the paper does not report reward normalization or advantage whitening, and the fixed discount factor should make long-scale returns two to three orders of magnitude smaller than short-scale ones. That gap needs to be closed before I'd trust the headline.\n\nWhat's actually new: the memory model itself is prior work (Shankar and Howard 2012; Howard et al. 2014), and the log-compression/translation property is well understood. The new piece is putting it inside A2C-style agents and showing, across four timing tasks, that CogRNN learns at similar trial counts when intervals are rescaled, while LSTM and RNN do not. The ablation studies, psychometric curves, and the time-cell activity analysis are reasonable and add texture. The zero-shot transfer with convolution and pooling is a clean demonstration of the designed invariance, even if it is not a surprise.\n\nSoft spots, in order of importance. First, the discounting issue. With gamma=0.98, a trial of 480 steps has a terminal reward discounted to roughly 0.0001–0.002, while a 30-step trial sees 0.3–0.38. Without some reward normalization or advantage scaling, policy-gradient updates at the long scale should be orders of magnitude smaller, and \"similar learning speed\" across scales would be very surprising. The paper says A2C contains scale-breaking components and reports the results anyway, but it never explains why the RL side does not dominate. Either the code has a normalization step (which should be disclosed) or the result needs a quantitative analysis of gradient magnitudes. Second, the selected learning rates per agent are mentioned but not reported; the reader cannot tell whether the comparison is apples-to-apples. Third, the zero-shot transfer experiment uses REINFORCE and lacks an LSTM baseline, so the \"RNN fails\" comparison is narrow.\n\nThese are addressable with additional analysis, not reasons to reject. The core idea is sound and the evidence is substantial. I would send it to review with a request for the missing details, and I would want the authors to either report the normalization or provide gradient-scale data.\n\nFor who: anyone working on memory in RL, or on scale invariance in biologically inspired AI, will find something useful. It is a niche contribution, not a field-changer. My recommendation: engage with it, and ask for revision.","headline":"A useful integration of a known scale-invariant memory into RL, with a plausible but under-supported claim about scale-invariant learning speed due to missing normalization details.","tokens_in":15877,"tokens_out":4237,"would_cite":true,"duration_ms":32682,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A log-compressed memory core lets deep RL agents learn timing tasks at the same speed whether intervals last tens or hundreds of steps, while LSTM- and RNN-based agents slow down at longer scales.","keywords":["scale-invariant memory","deep reinforcement learning","temporal credit assignment","Laplace transform","time cells","interval timing","LSTM recurrent networks","cognitive neuroscience"],"falsifier":"Train the same CogRNN agents on the simplest interval timing task with all intervals rescaled so that they fall outside the designed 1-to-1000-step memory window, for example every interval longer than 1000 steps, and compare the learning curves across scales; if the speed of learning varies as much as it does for LSTM agents at those scales, the claimed scale invariance holds only inside the manually chosen range rather than as a property of the representation.","tokens_in":14931,"feed_emoji":"⏱️","tokens_out":14481,"duration_ms":112337,"temperature":0.7,"pith_summary":"This paper argues that the reason deep reinforcement learning agents handle time poorly is the way they remember it, and that replacing the learned recurrent core with a neuroscience-inspired, log-compressed memory removes the problem. The proposed core, called CogRNN, first runs input through a recurrent layer that approximates a Laplace transform of recent history, then through a dense layer that approximates the inverse transform; the result is a timeline of sequentially active units, similar to biological time cells, spaced uniformly on a logarithmic time axis. Because rescaling a task in time shifts this timeline rather than stretching it, the learning problem has the same difficulty at every temporal scale inside the memory's range. In interval timing, interval discrimination, delayed-match-to-sample, and interval reproduction tasks, CogRNN agents reached high reward at similar trial counts across a tenfold range of interval durations, while LSTM- and RNN-based agents learned at visibly different speeds; with convolution and max-pooling added on top, agents trained at one scale generalized to two- and four-times-faster versions without retraining.","feed_headline":"Log-compressed memory keeps RL learning speed fixed across time scales","feed_subtitle":"A neuroscience-inspired recurrent core keeps trial counts constant when task intervals are rescaled tenfold.","key_machinery":"The load-bearing object is a two-layer recurrent core with analytically fixed weights. The first layer is a recurrent network with diagonal connectivity matrix $L=e^{-S\\Delta t}$, where $S$ is a diagonal matrix of positive decay rates $s$; this computes a running approximation of the modified Laplace transform $\\mathcal{F}(s;t)$ of the input history. The second layer is a dense matrix that approximates the inverse Laplace transform using derivatives of $\\mathcal{F}$ with respect to $s$, producing units $\\tilde{f}^{*}_{\\tau;t}$ whose impulse responses are unimodal functions peaking at $\\tau^*$, with width proportional to peak time. Choosing the $\\tau^*$ values log-spaced between $\\tau^*_{\\min}=1$ and $\\tau^*_{\\max}=1000$ places the units equidistant and equally wide on a log-time axis, so temporal rescaling $t\\to at$ becomes a translation by $\\log(a)$ along the neuron index. Convolution and max-pooling over that translated activity make the representation invariant to rescaling, up to edge effects at the ends of the $\\tau^*$ grid.","core_discovery":"The central claim is that scale invariance of learning is a property of the memory representation, not of the learning rule, and that a fixed, analytically constructed memory can give deep RL agents the same scale-invariant learning seen in animals. The CogRNN core approximates the modified Laplace transform $\\mathcal{F}(s;t)=\\int_0^t e^{-s(t-t')} f(t')\\,dt'$ with a recurrent layer whose diagonal connection matrix is $e^{-S\\Delta t}$, then applies a dense layer that approximates the inverse Laplace transform. The resulting units $\\tilde{f}^{*}_{\\tau;t}$ activate sequentially, each peaking at a log-spaced $\\tau^*$ and having temporal field width that grows linearly with peak time. When task intervals are rescaled by a factor $a$, this memory pattern translates by $\\log(a)$ rather than stretching, and the paper reports that as a result the number of trials needed to learn stays roughly constant across a tenfold range of interval durations. In the fully rescaled environment, adding convolution and max-pooling over the translated representation makes the policy invariant, so agents trained at scale 1 earned perfect scores at scales 2 and 4 with no additional training.","pith_inferences":["If the memory representation is the true carrier of scale invariance, combining this core with power-law (scale-invariant) temporal discounting should extend the effect beyond the tested window and reduce the edge effects at the ends of the $\\tau^*$ grid; the paper suggests this combination as a direction but does not test it.","Because rescaling becomes translation, other translation-equivariant architectures over the log-time axis, including attention, should inherit the same scale invariance, not only the convolution-and-max-pooling readout tested here.","The same Laplace/inverse-Laplace core could be inserted into non-reinforcement-learning sequence models to make their temporal judgments invariant to the speed of the input; this is a direct but untested extension of the paper's mechanism."],"forward_implications":["Agents built on CogRNN can be trained once at one temporal scale and applied at other scales without retuning learning rate, discount factor, or horizon, provided task intervals stay inside the memory's 1-to-1000-step support.","Temporal memory need not be learned: a fixed, neuroscience-derived core can replace a trainable recurrent core and reach the same or better performance while training in roughly half the time because it has fewer trainable parameters.","Adding translation-invariant readout over the log-time axis yields policies that transfer to unseen time scales with zero additional gradient updates.","Time-cell-like neurons with field width growing linearly with peak time are a signature of scale-invariant representation; this signature appears in CogRNN but not in RNN or LSTM agents."],"supporting_citations":[{"why":"Supplies the scale-invariant Laplace-transform memory model that the CogRNN core implements.","marker":"(Shankar and Howard 2012)"},{"why":"Provides the optimally fuzzy temporal memory formulation behind the inverse-transform layer.","marker":"(Shankar and Howard 2013)"},{"why":"Documents the behavioral scale invariance (constant trials to learn under rescaling) that the experiments aim to reproduce.","marker":"(Gallistel and Gibbon 2000)"},{"why":"Provides the six-duration interval timing task design that the environments are built on.","marker":"(Kim et al. 2013)"},{"why":"Establishes the interval reproduction task for deep RL agents that this paper extends to multiple scales.","marker":"(Deverett et al. 2019)"},{"why":"Defines the LSTM baseline whose scale-dependent learning is contrasted with CogRNN.","marker":"(Hochreiter and Schmidhuber 1997)"},{"why":"Supplies the A3C actor-critic update underlying the agents.","marker":"(Mnih et al. 2016)"},{"why":"Supplies the generalized advantage estimator used for policy gradients.","marker":"(Schulman et al. 2016)"},{"why":"REINFORCE is the algorithm used in the full-environment rescaling generalization experiment.","marker":"(Williams 1992)"}],"fun_headline_variants":["Scale-free RL memory: same trials for tenfold time changes","Brain-inspired memory fixes RL across time scales","Time-scale invariant RL from log-spaced memory","RL learns equally fast at any time scale with this memory","Neuroscience memory design makes RL scale-invariant"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that the memory's scale invariance, not the reinforcement learning update's exponential discounting, finite rollout, or horizon, sets the learning difficulty, and that every task interval stays inside the hand-selected 1-to-1000-step memory range.","fun_headline_variants_meta":{"raw":{"variants":["Scale-free RL memory: same trials for tenfold time changes","Brain-inspired memory fixes RL across time scales","Time-scale invariant RL from log-spaced memory","RL learns equally fast at any time scale with this memory","Neuroscience memory design makes RL scale-invariant"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1803,"prompt_tokens":924,"completion_tokens":879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":803}},"tokens_in":540,"tokens_out":879,"duration_ms":6135,"temperature":1.0,"reasoning_tokens":803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:05:30.046468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same CogRNN agents on the simplest interval timing task with all intervals rescaled so that they fall outside the designed 1-to-1000-step memory window, for example every interval longer than 1000 steps, and compare the learning curves across scales; if the speed of learning varies as much as it does for LSTM agents at those scales, the claimed scale invariance holds only inside the manually chosen range rather than as a property of the representation.","supporting_citations":[{"cited_title":"H.; and Howard, M","cited_arxiv_id":null,"evidence_quote":"Supplies the scale-invariant Laplace-transform memory model that the CogRNN core implements."},{"cited_title":"H.; and Howard, M","cited_arxiv_id":null,"evidence_quote":"Provides the optimally fuzzy temporal memory formulation behind the inverse-transform layer."},{"cited_title":"R.; and Gibbon, J","cited_arxiv_id":null,"evidence_quote":"Documents the behavioral scale invariance (constant trials to learn under rescaling) that the experiments aim to reproduce."},{"cited_title":"H.; and Jung, M","cited_arxiv_id":null,"evidence_quote":"Provides the six-duration interval timing task design that the environments are built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the interval reproduction task for deep RL agents that this paper extends to multiple scales."},{"cited_title":"I.; and Abbeel, P","cited_arxiv_id":null,"evidence_quote":"Supplies the generalized advantage estimator used for policy gradients."}],"review_version":1}