Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Deep reinforcement learning with time-scale invariant memory

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A log-compressed memory core lets deep RL agents learn timing tasks at the same speed whether intervals last tens or hundreds of steps, while LSTM- and RNN-based agents slow down at longer scales.

desk verdict A useful integration of a known scale-invariant memory into RL, with a plausible but under-supported claim about scale-invariant learning speed due to missing normalization details. read the letter →

arxiv 2412.15292 v1 pith:JO5MQ7XS submitted 2024-12-19 cs.AI cs.LG

Md Rysul Kabir , James Mochizuki-Freeman , Zoran Tiganj This is my paper · ORCID
classification cs.AIcs.LG
keywords scale-invariantmemorydeepreinforcementlearningtemporalcreditassignmentLaplacetransformtimecellsintervaltimingLSTMrecurrentnetworkscognitiveneuroscience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reason deep reinforcement learning agents handle time poorly is the way they remember it, and that replacing the learned recurrent core with a neuroscience-inspired, log-compressed memory removes the problem. The proposed core, called CogRNN, first runs input through a recurrent layer that approximates a Laplace transform of recent history, then through a dense layer that approximates the inverse transform; the result is a timeline of sequentially active units, similar to biological time cells, spaced uniformly on a logarithmic time axis. Because rescaling a task in time shifts this timeline rather than stretching it, the learning problem has the same difficulty at every temporal scale inside the memory's range. In interval timing, interval discrimination, delayed-match-to-sample, and interval reproduction tasks, CogRNN agents reached high reward at similar trial counts across a tenfold range of interval durations, while LSTM- and RNN-based agents learned at visibly different speeds; with convolution and max-pooling added on top, agents trained at one scale generalized to two- and four-times-faster versions without retraining.

What carries the argument

The load-bearing object is a two-layer recurrent core with analytically fixed weights. The first layer is a recurrent network with diagonal connectivity matrix $L=e^{-S\Delta t}$, where $S$ is a diagonal matrix of positive decay rates $s$; this computes a running approximation of the modified Laplace transform $\mathcal{F}(s;t)$ of the input history. The second layer is a dense matrix that approximates the inverse Laplace transform using derivatives of $\mathcal{F}$ with respect to $s$, producing units $\tilde{f}^{*}_{\tau;t}$ whose impulse responses are unimodal functions peaking at $\tau^*$, with width proportional to peak time. Choosing the $\tau^*$ values log-spaced between $\tau^*_{\min}=1$ and $\tau^*_{\max}=1000$ places the units equidistant and equally wide on a log-time axis, so temporal rescaling $t\to at$ becomes a translation by $\log(a)$ along the neuron index. Convolution and max-pooling over that translated activity make the representation invariant to rescaling, up to edge effects at the ends of the $\tau^*$ grid.

What would settle it

Train the same CogRNN agents on the simplest interval timing task with all intervals rescaled so that they fall outside the designed 1-to-1000-step memory window, for example every interval longer than 1000 steps, and compare the learning curves across scales; if the speed of learning varies as much as it does for LSTM agents at those scales, the claimed scale invariance holds only inside the manually chosen range rather than as a property of the representation.

Watch

Extended reading notes

Core claim

The central claim is that scale invariance of learning is a property of the memory representation, not of the learning rule, and that a fixed, analytically constructed memory can give deep RL agents the same scale-invariant learning seen in animals. The CogRNN core approximates the modified Laplace transform $\mathcal{F}(s;t)=\int_0^t e^{-s(t-t')} f(t')\,dt'$ with a recurrent layer whose diagonal connection matrix is $e^{-S\Delta t}$, then applies a dense layer that approximates the inverse Laplace transform. The resulting units $\tilde{f}^{*}_{\tau;t}$ activate sequentially, each peaking at a log-spaced $\tau^*$ and having temporal field width that grows linearly with peak time. When task intervals are rescaled by a factor $a$, this memory pattern translates by $\log(a)$ rather than stretching, and the paper reports that as a result the number of trials needed to learn stays roughly constant across a tenfold range of interval durations. In the fully rescaled environment, adding convolution and max-pooling over the translated representation makes the policy invariant, so agents trained at scale 1 earned perfect scores at scales 2 and 4 with no additional training.

Load-bearing premise

The result rests on the assumption that the memory's scale invariance, not the reinforcement learning update's exponential discounting, finite rollout, or horizon, sets the learning difficulty, and that every task interval stays inside the hand-selected 1-to-1000-step memory range.

Editorial extensions

If this is right

  • Agents built on CogRNN can be trained once at one temporal scale and applied at other scales without retuning learning rate, discount factor, or horizon, provided task intervals stay inside the memory's 1-to-1000-step support.
  • Temporal memory need not be learned: a fixed, neuroscience-derived core can replace a trainable recurrent core and reach the same or better performance while training in roughly half the time because it has fewer trainable parameters.
  • Adding translation-invariant readout over the log-time axis yields policies that transfer to unseen time scales with zero additional gradient updates.
  • Time-cell-like neurons with field width growing linearly with peak time are a signature of scale-invariant representation; this signature appears in CogRNN but not in RNN or LSTM agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the memory representation is the true carrier of scale invariance, combining this core with power-law (scale-invariant) temporal discounting should extend the effect beyond the tested window and reduce the edge effects at the ends of the $\tau^*$ grid; the paper suggests this combination as a direction but does not test it.
  • Because rescaling becomes translation, other translation-equivariant architectures over the log-time axis, including attention, should inherit the same scale invariance, not only the convolution-and-max-pooling readout tested here.
  • The same Laplace/inverse-Laplace core could be inserted into non-reinforcement-learning sequence models to make their temporal judgments invariant to the speed of the input; this is a direct but untested extension of the paper's mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes integrating a computational neuroscience model of scale-invariant memory (CogRNN) into deep RL agents. The memory is built from an approximate Laplace transform with log-spaced time constants, and the paper provides a theoretical argument that temporal rescaling becomes a translation in the memory representation; adding convolution and pooling makes the representation invariant to rescaling. The authors compare CogRNN with LSTM and RNN cores on interval timing (1D and 3D), interval discrimination, delayed-match-to-sample, and interval reproduction tasks, varying task-relevant intervals via a step-size parameter. They report that CogRNN agents reach high performance and learn at similar speeds across temporal scales, while LSTM/RNN agents show scale-dependent learning. They also show zero-shot transfer of a CogRNN-based policy to rescaled environments and report that CogRNN time-cell-like activity exhibits scale-invariant width growth.

Significance. If the central empirical claim holds, the paper would make a valuable contribution by demonstrating that a biologically inspired, log-compressed memory representation can decouple RL learning difficulty from temporal scale within a designed range, addressing a real limitation of standard recurrent cores like LSTM and RNN. The theoretical derivation of scale covariance is clear and the zero-shot transfer experiment is a concrete, falsifiable demonstration of the designed property. The paper also provides open-source code. However, the significance currently hinges on a qualitative reading of learning curves and on the assumption that no unreported normalization compensates for discounting-induced gradient scale differences; until those points are resolved, the strength of the claim is not fully established.

major comments (4)
  1. [Hyperparameters and training] The paper reports a fixed discount factor gamma=0.98 for all agents and scales. In the interval timing task, with step size 10 the intervals are 300-480 steps, while with step size 100 they are 30-48 steps. For a correct-trial reward of 1.0 at the end of the trial, the discounted return is roughly gamma^delay: about 0.3-0.5 for the short scale and below 0.002 for the long scale. Since A2C policy-gradient updates scale with advantage magnitude, one would expect the long-scale learning signal to be orders of magnitude smaller, making similar learning speeds implausible unless advantage normalization, reward scaling, or per-scale learning-rate adjustment is applied. The paper does not report any such normalization or scaling. Please specify whether advantages are normalized (e.g., by subtracting the mean and dividing by the standard deviation), whether rewards are scaled, or whether any other mechanism compensates for the discount-induced gradient scale difference. If none is used, please provide a concrete explanation for the observed similarity in learning speeds.
  2. [Results (Fig. 4)] The central claim that 'the speed of learning was similar at different temporal scales' for CogRNN agents is based on visual inspection of learning curves in Fig. 4. The manuscript does not provide a quantitative measure such as trials-to-threshold, time-to-criterion, or a statistical comparison (e.g., confidence intervals for learning speed, or a test of whether final performance differs across scales). Please add a quantitative analysis that supports the claim, including error bars and a statistical test if possible.
  3. [Hyperparameters and training] The text states that the authors 'selected the best-performing learning rates for each agent' among 0.001, 0.0001, and 0.00001, but the selected values are not reported. If different learning rates were used for different architectures or different temporal scales, the comparison in Fig. 4 could be confounded. Please report the selected learning rates for each architecture and each environment/scale, or state explicitly that the same learning rate was used across scales for each architecture.
  4. [Discussion] The Discussion acknowledges that A2C contains components that break scale invariance, including exponential discounting, horizon, and rollout length, but the paper does not attempt to isolate how the CogRNN memory representation overcomes the scale dependence of the RL algorithm itself. This is important because the central claim is about learning speed, not just representational invariance. I suggest adding an experiment or at least a more detailed analysis of why the memory's invariance dominates the RL algorithm's scale dependence, for example by comparing with a CogRNN agent using power-law discounting, or by ablating the normalization mechanism if one exists.
minor comments (4)
  1. [Model (Scale invariant memory network)] In Eq. (5), the impulse response formula includes a factor 1/t and a power (t/tau*)^(k+1); the notation is a bit dense. It would help to state explicitly that the peak time equals tau* after the derivative computation, since the current text says 'equate it with 0' without showing the intermediate step.
  2. [Results] The interval reproduction results are only presented in the Supplemental Information (Table S1, S2, Fig. S11, S12). Since the reproduction task supports the multi-scale learning claim, consider summarizing at least one key quantitative result in the main text.
  3. [Hyperparameters and training] The sentence 'We updated the model parameters after each trial instead of backpropagating the gradients after horizon number of steps' is ambiguous: does this mean the effective horizon is the full trial length, or that gradients are accumulated over a trial? Please clarify the update schedule and its relation to the 'horizon' hyperparameter in Fig. S10.
  4. [Figure 4] In the 3D interval timing panel, the CogRNN and LSTM curves appear to plateau at different levels; adding a legend or annotation clarifying the final performance values would improve readability.

Circularity Check

2 steps flagged · score 4.0 of 10

Secondary results are designed features presented as findings, but the central learning-speed claim remains independent.

  1. self definitional [Results, 'Combining scale invariant memory with translation-invariance results in invariance to temporal rescaling' (Fig. 5)]
    "When the objective is to build agents that are invariant to temporal rescaling, we apply convolution and pooling over ˜f. The output of convolution and pooling is translation invariant, making the network invariant to temporal rescaling ... We then rescaled the time in the observation space to 2x and 4x the initial scale. Without additional training, our agents reached perfect performance (Fig. 5B)."

    The invariance is constructed, not discovered: the model section proves that log-spaced tau makes temporal rescaling equivalent to a translation of the internally represented timeline, and convolution plus pooling is translation invariant by definition. Consequently, the observation fed to the policy is identical at scales 1, 2, and 4 (up to edge effects), so the policy trained at scale 1 reaches perfect performance at other scales by construction. The experiment is a demonstration of the architecture's definition, not an independent prediction.

  2. self definitional [Results, 'Neural activity is scale invariant and resembles time cells recorded from mammalian brains' (Fig. S4)]
    "For a representation to be scale invariant, the relationship between the peak time and the standard deviation should be linear. This was the case only for CogRNN neurons."

    The model section derives from Eq. 5 that the impulse response of the inverse-Laplace layer has coefficient of variation c = 1/sqrt(k+1), which does not depend on t or tau*, implying that the width of the unimodal basis functions increases linearly with their peak time. Fitting Gaussians to CogRNN neurons and recovering the linear peak-time/standard-deviation relation therefore confirms the analytically designed constant-c property of the basis functions, rather than providing independent neural evidence. The paper even notes that fitted CogRNN deviations are due to the same numerical derivative and edge effects that the model section predicted.

full rationale

The central empirical claim that CogRNN agents learn at similar speeds across temporal scales (Fig. 4) has independent content: although the CogRNN representation is constructed to translate under temporal rescaling, the A2C/GAE update rules include exponential discounting, fixed horizon, and rollout length, which are not scale invariant, so the similar learning-speed observation is not forced by the architecture alone. The paper acknowledges this in the Discussion. The self-citations to Shankar and Howard and Howard et al. supply the memory formalism, but the invariance proof is re-derived in Eqs. 1-5, so those citations are not load-bearing. However, two headline results do reduce by construction. The zero-shot transfer experiment (Fig. 5) is a direct restatement of the design: log-spaced Laplace/inverse-Laplace memory converts rescaling into translation, and convolution plus max-pooling is translation invariant, so the trained policy transfers by construction. The claim that CogRNN time-cell peak-time versus standard deviation is linear (Fig. S4) restates the derivation that the coefficient of variation c=1/sqrt(k+1) is constant, meaning width increases linearly with peak time; this is a property engineered into Eq. 5, not an emergent empirical discovery. These constructed results do not invalidate the central learning-speed finding, but they inflate the apparent number of independent confirmations.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The paper's scale-invariant behavior is largely engineered: the CogRNN weights are analytic, and the range of effective scales is set by hand-picked tau_min, tau_max, k, and the number of units. The RL comparison also depends on several hand-chosen hyperparameters, some of which are not reported. No new physical or conceptual entities are introduced; the 'time cells' observed in CogRNN are emergent activities of the designed memory, not independently verified entities.

free parameters (9)
  • tau_min = 1
    Minimum log-spaced peak time for the CogRNN memory units; chosen by hand to match the shortest task-relevant intervals. Determines the lower bound of the scale-invariance range.
  • tau_max = 1000
    Maximum log-spaced peak time; chosen to cover the longest intervals (up to 480 steps). The paper's scale-invariance claim only holds within [tau_min, tau_max].
  • k = 8
    Order of Post's inversion derivative in the inverse Laplace layer; sets the basis function width (c = 1/sqrt(k+1)). Chosen by hand; ablation shown in Fig. S7.
  • number_of_memory_units = 8
    Number of log-spaced units in CogRNN for the 3D environment; the same CogRNN is used for simple environments. The coarse 8-unit grid still covers two decades.
  • learning_rate = not reported (searched 0.001, 0.0001, 0.00001)
    The authors selected the best-performing learning rate for each architecture but do not report the selected values, weakening transparency.
  • entropy_coefficient = not reported (explored in SI)
    Hyperparameter explored in the Supplemental Information; the final value used in the main experiments is not stated.
  • horizon = not reported (explored in SI)
    Rollout/horizon length explored in Fig. S10; the exact value used for the main results is not specified in the main text.
  • discount_factor_gamma = 0.98
    Chosen by hand for the RL algorithm; exponential discounting is one of the RL components that breaks scale invariance, and its value affects all agents equally.
  • gae_lambda = 0.95
    Chosen by hand for the generalized advantage estimator; affects bias-variance tradeoff but is fixed across agents.
assumptions (5)
  • standard math Post's inversion formula with a finite k (Eq. 2) provides a sufficiently accurate inverse Laplace transform for recovering the temporal history.
    The inverse layer is built on Post 1930; accuracy depends on k, and the authors acknowledge edge effects from the finite numerical derivative.
  • domain assumption The log-spaced basis-function memory from Shankar and Howard 2012 is a valid scale-invariant temporal representation.
    The paper builds CogRNN on this prior model without re-deriving its validity; the central demonstration inherits this assumption.
  • domain assumption Temporal rescaling of the environment maps to a translation of the log-compressed memory representation, making the learning problem equally difficult across scales inside [tau_min, tau_max].
    Shown analytically in the Model section, but only approximate due to finite range and edge effects; the empirical claims depend on this.
  • domain assumption The scale-dependence of the RL algorithm (A2C/GAE with discounting, horizon, rollout length) is negligible compared to the scale-invariance of the memory representation.
    The Discussion acknowledges these components break scale invariance but offers no proof that the memory dominates; the robustness results may depend on the chosen values.
  • domain assumption The discrete environment time step is treated as the unit of time (dt = 1) in the memory update, so the hand-set tau_min and tau_max directly match environment steps.
    The paper states 'we assume that the duration of a discrete-time step dt = 1' in the Model section. If the environment step size were different, the memory range would need to be rescaled accordingly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep reinforcement learning with time-scale invariant memory." pith.science (2026). https://pith.science/paper/JO5MQ7XS

@misc{pith2026241215292,
  author       = {Pith},
  title        = {Pith review of: Deep reinforcement learning with time-scale invariant memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JO5MQ7XS}},
  note         = {Machine review of arXiv:2412.15292}
}
read the original abstract

The ability to estimate temporal relationships is critical for both animals and artificial agents. Cognitive science and neuroscience provide remarkable insights into behavioral and neural aspects of temporal credit assignment. In particular, scale invariance of learning dynamics, observed in behavior and supported by neural data, is one of the key principles that governs animal perception: proportional rescaling of temporal relationships does not alter the overall learning efficiency. Here we integrate a computational neuroscience model of scale invariant memory into deep reinforcement learning (RL) agents. We first provide a theoretical analysis and then demonstrate through experiments that such agents can learn robustly across a wide range of temporal scales, unlike agents built with commonly used recurrent memory architectures such as LSTM. This result illustrates that incorporating computational principles from neuroscience and cognitive science into deep neural networks can enhance adaptability to complex temporal dynamics, mirroring some of the core properties of human learning.

Figures

Figures reproduced from arXiv: 2412.15292 by the authors.

Figure 1
Figure 1. A. The six temporal intervals used in the task. At each trial, a random interval is selected and the agent has to indicate whether the interval was long or short. B. Schematic of the environment. After the agent crosses the start line, one of six delay intervals is presented. portion of the T-maze and without a bound at the end. Once the agent reaches the goal positions, it receives a reward of either 10 for a corre… view at source ↗
Figure 2
Figure 2. Architecture of the (RL) agent. Observations from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A. Response of the CogRNN to δ pulses. Neurons in Fs;t decay exponentially at a spectrum of time constants s implementing a discrete approximation of a real-domain Laplace transform. Neurons in ˜f∗ τ;s activate sequentially, resembling time cells. B. Log-compressed memory (bottom) of three signals that are rescaled versions of each other (top) at time t = 250. Each circle represents the activity of individual neuron… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The performance (mean with standard error over five runs) across the four tasks for CogRNN and LSTM agents. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: A. Output of convolution and pooling operations for three signals from Fig. 3B. B. Performance of CogRNN ( ˜f) and RNN agents trained on the 1D interval timing task. The agents were trained on scale 1 and evaluated on scales 1, 2 and 4. tive agent from RNN, LSTM and Co…
Figure 6
Figure 6. Figure 6: Normalized activity of neurons that resemble time cells from three representative agents. Neurons are sorted by peak [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prediction Is Not Memory: Dual-Timescale Gated Profile Writing for Persistent User Modeling

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A lightweight write-risk gate reduces harmful persistent-profile updates from 22.45% to about 14.5% on MicroLens-100K, and next-item ranking confidence is a poor substitute for write-risk scoring.

Reference graph

Works this paper leans on

72 extracted references · 65 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Balci, F.; and Freestone, D. 2020. The peak interval procedure in rodents: a tool for studying the neurobiological basis of interval timing and its alterations in models of human disease. Bio-protocol, 10(17): e3735--e3735

    work page 2020

  4. [4]

    Balci, F.; and Simen, P. 2016. A decision model of timing. Current opinion in behavioral sciences, 8: 94--101

    work page 2016

  5. [5]

    D.; and Gallistel, C

    Balsam, P. D.; and Gallistel, C. R. 2009. Temporal maps and informativeness in associative learning. Trends in neurosciences, 32(2): 73--78

    work page 2009

  6. [6]

    J.; Degris, T.; Modayil, J.; et al

    Banino, A.; Barry, C.; Uria, B.; Blundell, C.; Lillicrap, T.; Mirowski, P.; Pritzel, A.; Chadwick, M. J.; Degris, T.; Modayil, J.; et al. 2018. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705): 429--433

    work page 2018

  7. [7]

    V.; Aziz, D.; Winslow, D.; Carter, R

    Buhusi, C. V.; Aziz, D.; Winslow, D.; Carter, R. E.; Swearingen, J. E.; and Buhusi, M. C. 2009. Interval timing accuracy and scalar timing in C57BL/6 mice. Behavioral neuroscience, 123(5): 1102

    work page 2009

  8. [8]

    V.; and Meck, W

    Buhusi, C. V.; and Meck, W. H. 2005. What makes us tick? Functional and neural mechanisms of interval timing. Nature reviews neuroscience, 6(10): 755--765

    work page 2005

Show all 72 references
  1. [9]

    H.; Charczynski, S

    Cao, R.; Bladon, J. H.; Charczynski, S. J.; Hasselmo, M. E.; and Howard, M. W. 2022. Internally generated time in the rodent hippocampus is logarithmically compressed. Elife, 11: e75353

    2022

  2. [10]

    Coumans, E.; and Bai, Y. 2021. PyBullet, a Python module for physics simulation for games, robotics and machine learning. http://pybullet.org

    2021

  3. [11]

    A.; Tiganj, Z.; Brincat, S

    Cruzado, N. A.; Tiganj, Z.; Brincat, S. L.; Miller, E. K.; and Howard, M. W. 2020. Conjunctive representation of what and when in monkey hippocampus and lateral prefrontal cortex during an associative memory task. Hippocampus, 30(12): 1332--1346

    2020

  4. [12]

    J.; Saez, A.; Marcos, E.; Genovesio, A.; Jazayeri, M.; Romo, R.; Salzman, C

    Cueva, C. J.; Saez, A.; Marcos, E.; Genovesio, A.; Jazayeri, M.; Romo, R.; Salzman, C. D.; Shadlen, M. N.; and Fusi, S. 2020. Low-dimensional dynamics for working memory and time encoding. Proceedings of the National Academy of Sciences, 117(37): 23021--23032

    2020

  5. [13]

    Deverett, B.; Faulkner, R.; Fortunato, M.; Wayne, G.; and Leibo, J. Z. 2019. Interval timing in deep reinforcement learning agents. Advances in Neural Information Processing Systems, 32

    2019

  6. [14]

    A.; Paulsen, O.; Robbins, T

    Donnelly, N. A.; Paulsen, O.; Robbins, T. W.; and Dalley, J. W. 2015. Ramping single unit activity in the medial prefrontal cortex and ventral striatum reflects the onset of waiting but not imminent impulsive actions. European Journal of Neuroscience, 41(12): 1524--1537

    2015

  7. [15]

    Eichenbaum, H. 2014. Time cells in the hippocampus: a new dimension for mapping memories. Nature Reviews Neuroscience, 15(11): 732--744

    2014

  8. [16]

    B.; De Corte, B

    Emmons, E. B.; De Corte, B. J.; Kim, Y.; Parker, K. L.; Matell, M. S.; and Narayanan, N. S. 2017. Rodent medial frontal control of temporal processing in the dorsomedial striatum. Journal of Neuroscience, 37(36): 8718--8733

    2017

  9. [17]

    1860/1912

    Fechner, G. 1860/1912. Elements of Psychophysics. Vol. I. Houghton Mifflin

    1912

  10. [18]

    R.; and Gibbon, J

    Gallistel, C. R.; and Gibbon, J. 2000. Time, rate, and conditioning. Psychological Review, 107(2): 289--344

    2000

  11. [19]

    R.; and Shahan, T

    Gallistel, C. R.; and Shahan, T. A. 2024. Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement. Proceedings of the National Academy of Sciences, 121(30): e2405451121

    2024

  12. [20]

    Genovesio, A.; Tsujimoto, S.; and Wise, S. P. 2009. Feature- and Order-Based Timing Representations in the Frontal Cortex. Neuron, 63(2): 254--266

    2009

  13. [21]

    Gibbon, J. 1977. Scalar expectancy theory and Weber's law in animal timing. Psychological review, 84(3): 279

    1977

  14. [22]

    S.; Monteiro, T.; Motiwala, A.; Soares, S.; Machens, C.; and Paton, J

    Gouv \^e a, T. S.; Monteiro, T.; Motiwala, A.; Soares, S.; Machens, C.; and Paton, J. J. 2015. Striatal dynamics explain duration judgments. Elife, 4: e11386

    2015

  15. [23]

    Grossberg, S.; and Schmajuk, N. A. 1989. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural networks, 2(2): 79--102

    1989

  16. [24]

    Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8): 1735--1780

    1997

  17. [25]

    A.-D.; and Telek, M

    Horv \'a th, G.; Horv \'a th, I.; Almousa, S. A.-D.; and Telek, M. 2020. Numerical inverse Laplace transformation using concentrated matrix exponential distributions. Performance Evaluation, 137: 102067

    2020

  18. [26]

    W.; MacDonald, C

    Howard, M. W.; MacDonald, C. J.; Tiganj, Z.; Shankar, K. H.; Du, Q.; Hasselmo, M. E.; and Eichenbaum, H. 2014. A unified mathematical framework for coding time, space, and sequences in the hippocampal region. Journal of Neuroscience, 34(13): 4692--4707

    2014

  19. [27]

    W.; Shankar, K

    Howard, M. W.; Shankar, K. H.; Aue, W. R.; and Criss, A. H. 2015. A distributed representation of internal time. Psychological review, 122(1): 24

    2015

  20. [28]

    W.; and Sederberg, P

    Jacques, B.; Tiganj, Z.; Howard, M. W.; and Sederberg, P. B. 2021. DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales. Advances in neural information processing system

    2021

  21. [29]

    G.; Tiganj, Z.; Sarkar, A.; Howard, M.; and Sederberg, P

    Jacques, B. G.; Tiganj, Z.; Sarkar, A.; Howard, M.; and Sederberg, P. 2022. A deep convolutional neural network that is invariant to time rescaling. In International Conference on Machine Learning, 9729--9738. PMLR

    2022

  22. [30]

    Jazayeri, M.; and Shadlen, M. N. 2015. A neural mechanism for sensing and reproducing a time interval. Current Biology, 25(20): 2599--2609

    2015

  23. [31]

    Z.; Fujii, N.; and Graybiel, A

    Jin, D. Z.; Fujii, N.; and Graybiel, A. M. 2009. Neural representation of time in cortico-basal ganglia circuits. Proceedings of the National Academy of Sciences, 106(45): 19156--19161

    2009

  24. [32]

    H.; and Jung, M

    Kim, J.; Ghim, J.-W.; Lee, J. H.; and Jung, M. W. 2013. Neural correlates of interval timing in rodent prefrontal cortex. Journal of Neuroscience, 33(34): 13834--13847

    2013

  25. [33]

    L.; Ruggiero, R

    Kim, Y.-C.; Han, S.-W.; Alberico, S. L.; Ruggiero, R. N.; De Corte, B.; Chen, K.-H.; and Narayanan, N. S. 2017. Optogenetic stimulation of frontal D1 neurons compensates for impaired temporal control of action in dopamine-depleted mice. Current biology, 27(1): 39--47

    2017

  26. [34]

    Lin, D.; and Richards, B. A. 2021. Time cell encoding in deep reinforcement learning agents depends on mnemonic demands. bioRxiv, 2021--07

    2021

  27. [35]

    A.; Sutton, R

    Ludvig, E. A.; Sutton, R. S.; and Kehoe, E. J. 2008. Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural computation, 20(12): 3034--3054

    2008

  28. [36]

    time cells

    MacDonald, C. J.; Lepage, K. Q.; Eden, U. T.; and Eichenbaum, H. 2011. Hippocampal “time cells” bridge the gap in memory for discontiguous events. Neuron, 71(4): 737--749

    2011

  29. [37]

    J.; and Tonegawa, S

    MacDonald, C. J.; and Tonegawa, S. 2021. Crucial role for CA2 inputs in the sequential organization of CA1 time cells supporting memory. Proceedings of the National Academy of Sciences, 118(3): e2020698118

    2021

  30. [38]

    S.; Mochizuki-Freeman, J.; Indi, C

    Maini, S. S.; Mochizuki-Freeman, J.; Indi, C. S.; Jacques, B. G.; Sederberg, P. B.; Howard, M. W.; and Tiganj, Z. 2023. Representing latent dimensions using compressed number lines. In 2023 international joint conference on neural networks (ijcnn), 1--10. IEEE

    2023

  31. [39]

    R.; Malik, A

    Masset, P.; Tano, P.; Kim, H. R.; Malik, A. N.; Pouget, A.; and Uchida, N. 2023. Multi-timescale reinforcement learning in the brain. bioRxiv

    2023

  32. [40]

    S.; and Meck, W

    Matell, M. S.; and Meck, W. H. 2004. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Cognitive brain research, 21(2): 139--170

    2004

  33. [41]

    B.; Soares, S.; and Paton, J

    Mello, G. B.; Soares, S.; and Paton, J. J. 2015. A scalable population code for time in the striatum. Current Biology, 25(9): 1113--1122

    2015

  34. [42]

    K.; Erickson, C

    Miller, E. K.; Erickson, C. A.; and Desimone, R. 1996. Neural Mechanisms of Visual Working Memory in Prefrontal Cortex of the Macaque. Journal of Neuroscience, 16(16): 5154--5167

    1996

  35. [43]

    P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K

    Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, 1928--1937. PMLR

    2016

  36. [44]

    R.; and Tiganj, Z

    Mochizuki-Freeman, J.; Kabir, M. R.; and Tiganj, Z. 2024. Incorporating a cognitive model for evidence accumulation into deep reinforcement learning agents. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46

    2024

  37. [45]

    J.; de la Rocha, J.; et al

    Molano-Mazon, M.; Barbosa, J.; Pastor-Ciurana, J.; Fradera, M.; Zhang, R.-Y.; Forest, J.; del Pozo Lerida, J.; Ji-An, L.; Cueva, C. J.; de la Rocha, J.; et al. 2022. NeuroGym: An open resource for developing and sharing neuroscience tasks

    2022

  38. [46]

    Momennejad, I.; and Howard, M. W. 2018. Predicting the future with multi-scale successor representations. bioRxiv, 449470

    2018

  39. [47]

    Namboodiri, V. M. K. 2022. How do real animals account for the passage of time during associative learning? Behavioral Neuroscience

    2022

  40. [48]

    Narayanan, N. S. 2016. Ramping activity is a cortical mechanism of temporal control of action. Current opinion in behavioral sciences, 8: 226--230

    2016

  41. [49]

    O'Keefe, J. 1976. Place units in the hippocampus of the freely moving rat. Experimental neurology, 51(1): 78--109

    1976

  42. [50]

    L.; Chen, K.-H.; Kingyon, J

    Parker, K. L.; Chen, K.-H.; Kingyon, J. R.; Cavanagh, J. F.; and Narayanan, N. S. 2014. D1-dependent 4 Hz oscillations and ramping activity in rodent medial frontal cortex during interval timing. Journal of Neuroscience, 34(50): 16774--16783

    2014

  43. [51]

    Pastalkova, E.; Itskov, V.; Amarasingham, A.; and Buzsaki, G. 2008. Internally generated cell assembly sequences in the rat hippocampus. Science, 321(5894): 1322--1327

    2008

  44. [52]

    P \'e rez, O.; and Merchant, H. 2018. The synaptic properties of cells define the hallmarks of interval timing in a recurrent neural network. Journal of Neuroscience, 38(17): 4186--4199

    2018

  45. [53]

    A.; Gershman, S

    Petter, E. A.; Gershman, S. J.; and Meck, W. H. 2018. Integrating models of interval timing and reinforcement learning. Trends in cognitive sciences, 22(10): 911--922

    2018

  46. [54]

    Post, E. 1930. Generalized Differentiation. Transactions of the American Mathematical Society, 32: 723--781

    1930

  47. [55]

    Raphan, T.; Dorokhin, E.; and Delamater, A. R. 2019. Modeling interval timing by recurrent neural nets. Frontiers in integrative neuroscience, 13: 46

    2019

  48. [56]

    D.; and Kurth-Nelson, Z

    Redish, A. D.; and Kurth-Nelson, Z. 2010. Neural models of delay discounting

    2010

  49. [57]

    M.; Tiganj, Z.; Khasnabish, S.; Kohley, A.; Sheehan, D.; Howard, M

    Salz, D. M.; Tiganj, Z.; Khasnabish, S.; Kohley, A.; Sheehan, D.; Howard, M. W.; and Eichenbaum, H. 2016. Time cells in hippocampal area CA3. Journal of Neuroscience, 36(28): 7476--7484

    2016

  50. [58]

    Schaeffer, R.; Khona, M.; and Fiete, I. 2022. No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit. bioRxiv, 2022--08

    2022

  51. [59]

    I.; and Abbeel, P

    Schulman, J.; Moritz, P.; Levine, S.; Jordan, M. I.; and Abbeel, P. 2016. High-Dimensional Continuous Control Using Generalized Advantage Estimation. In Bengio, Y.; and LeCun, Y., eds., 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico,...

    2016

  52. [60]

    H.; and Howard, M

    Shankar, K. H.; and Howard, M. W. 2012. A scale-invariant internal representation of time. Neural Computation, 24(1): 134--193

    2012

  53. [61]

    H.; and Howard, M

    Shankar, K. H.; and Howard, M. W. 2013. Optimally fuzzy temporal memory. Journal of Machine Learning Research, 14: 3753--3780

    2013

  54. [62]

    Sorscher, B.; Mel, G.; Ganguli, S.; and Ocko, S. 2019. A unified theory for the origin of grid cells through the lens of pattern formation. Advances in neural information processing systems, 32

    2019

  55. [63]

    Tano, P.; Dayan, P.; and Pouget, A. 2020. A Local Temporal Difference Code for Distributional Reinforcement Learning. Advances in Neural Information Processing Systems, 33

    2020

  56. [64]

    A.; Roy, J

    Tiganj, Z.; Cromer, J. A.; Roy, J. E.; Miller, E. K.; and Howard, M. W. 2018. Compressed timeline of recent experience in monkey lateral prefrontal cortex. Journal of cognitive neuroscience, 30(7): 935--950

    2018

  57. [65]

    J.; Sederberg, P

    Tiganj, Z.; Gershman, S. J.; Sederberg, P. B.; and Howard, M. W. 2019. Estimating scale-invariant future in continuous time. Neural computation, 31(4): 681--709

    2019

  58. [66]

    W.; Kim, J.; and Howard, M

    Tiganj, Z.; Jung, M. W.; Kim, J.; and Howard, M. W. 2017. Sequential firing codes for time in rodent medial prefrontal cortex. Cerebral Cortex, 27(12): 5663--5671

    2017

  59. [67]

    Tiganj, Z.; Tang, W.; and Howard, M. 2021. A computational model for simulating the future using a memory timeline. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43

    2021

  60. [68]

    N.; Kaiser, L

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017. Attention is All you Need. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Infor...

    2017

  61. [69]

    A.; and Jazayeri, M

    Wang, J.; Narain, D.; Hosseini, E. A.; and Jazayeri, M. 2018. Flexible timing by temporal scaling of cortical responses. Nature neuroscience, 21: 102--110

    2018

  62. [70]

    C.; Warren, J.; and Behrens, T

    Whittington, J. C.; Warren, J.; and Behrens, T. E. 2021. Relating transformers to models and neural representations of the hippocampal formation. arXiv preprint arXiv:2112.04035

    2021 arXiv

  63. [71]

    Wilkes, J. T. 2015. Reverse first principles: Weber's law and optimality in different senses. Ph.D. thesis, UNIVERSITY OF CALIFORNIA, SANTA BARBARA

    2015

  64. [72]

    Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 229--256

    1992

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.