Pith. sign in

REVIEW 3 major objections 5 minor 68 references

Dividing neural drive by a gain signal preserves continuous memory

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Divisive normalization in a recurrent network yields low-rank slow manifolds that support continuous working memory, whereas subtractive inhibition shatters them under time-varying input.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A genuinely useful empirical study of divisive normalization in trained RNNs, but the 'essentiality' claim overreaches and the key ablation is confounded by an extra rectification. the 3 major comments →

arxiv 2608.01947 v1 pith:3TMVUMHT submitted 2026-08-03 q-bio.NC cs.AIcs.NE

Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

Zhaotian Gu , Jie Su , Weiwei Wang , Chang Liu , Tianyi Qian , Dahui Wang This is my paper
classification q-bio.NC cs.AIcs.NE
keywords divisive normalizationworking memoryslow manifoldscontinuous attractorslow-rank recurrent networksgradient scalingbackpropagation through timeattractor shattering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that divisive normalization——dividing a neuron's drive by a dynamic gain signal from the surrounding population——is a computational mechanism for continuous working memory, not merely a sensory gain-control side effect. The authors introduce the Recurrent Divisive Normalization Network (RDNN), a minimal recurrent circuit whose excitatory state is updated as (Jf(R)+I)/(η+G) while an inhibitory pool G tracks the population activity Wf(R). They show that on two canonical tasks——angular velocity integration and memory-guided saccade——the RDNN converges to smooth, low-dimensional slow manifolds, whereas GRUs and LSTMs either fail to converge or shatter the state space into discrete point attractors. They trace this to an activity-dependent gradient scaling during backpropagation, which suppresses parameter updates along highly active directions and compresses the network's effective rank without explicit low-rank factorization. An ablation indicates that the divisive, multiplicative form of inhibition is what prevents manifold shattering under time-varying input, while subtractive inhibition can hold static memories but not integrate continuous drive.

Core claim

The central discovery the paper reports is that a single algebraic operation——division by a dynamic, activity-tracking pool——transforms the learning of continuous representations in recurrent networks. Trained RDNNs form slow manifolds whose Jacobian spectra are bimodal with a strict gap in the autonomous task and a structured tripartite spectrum in the input-driven task, while GRU/LSTM spectra are diffuse. The same divisor enters the BPTT gradient as α/(η+G), an activity-dependent attenuator that suppresses weight updates along highly active directions, correlating with an effective-rank self-compression that scales sub-linearly with hidden size. Replacing division with subtraction preserve

What carries the argument

The central object is the divisive normalization gate: an auxiliary inhibitory state G that tracks the excitatory population's activity via Wf(R) and divides the recurrent input, so the excitatory update reads approximately R ← (1−αR)R + αR (Jf(R)+I)/(η+G). The paper's load-bearing mechanism is that this divisor appears in the BPTT gradient as α/(η+G), an activity-dependent local gradient attenuator. That scaling is what the authors credit for producing flat, normally hyperbolic slow manifolds (via state-dependent shrinking of off-manifold Jacobian eigenvalues toward 0) and for the emergent low effective rank.

Load-bearing premise

The load-bearing premise is that the trained discrete-time update is dynamically equivalent to the continuous-time system the paper analyzes; if the leak placement or the noise structure (multiplicative vs. additive) differs between the two as written, the theoretical slow-manifold and gradient-scaling results may not describe the network that was actually trained.

What would settle it

Retrain the RDNN on angular velocity integration with the divisor frozen at its trial-averaged value——constant gain instead of dynamic η+G——and keep everything else identical; if the ring manifold survives intact, dynamic divisive normalization is not the essential ingredient. Alternatively, replace the additive noise in the discrete update with the multiplicative noise of Eq. (1); if the spectral gap disappears, the theory is analyzing a different system than the one trained.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On both canonical tasks, RDNN instances form topologically reliable ring/line manifolds in roughly 90% of seeds, versus under 50% for GRU and LSTM, which tend to discretize the state space into alternating stable points and saddles.
  • The 99%-energy effective rank of the RDNN's recurrent weights scales sub-linearly with hidden size (about 35–38 at H=256), while GRU/LSTM use essentially all available dimensions (>240); explicit low-rank factorization achieves even lower ranks but introduces a higher-rank convergence anomaly——wider bottlenecks train worse——that the unconstrained network avoids.
  • Replacing divisive inhibition with subtractive inhibition preserves the static ring attractor for the memory-guided saccade task but shatters it into stable fixed points under time-varying angular velocity input, supporting the claim that the multiplicative form is the essential ingredient for input-driven integration.
  • The same qualitative slow-manifold behaviour appears in ORGaNICs and in a two-dimensional double-integration task, while continuous-time Neural ODEs without division fail to maintain normal hyperbolicity, indicating the mechanism is the divisive normalization itself rather than continuous time or a particular gating scheme.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A cleaner ablation than the paper's subtractive control would clamp the divisor to a constant (static gain) during training on angular velocity integration; if the ring manifold survived, the 'essential' ingredient would be positive scaling rather than dynamic division, and if it shattered, the dynamic tracking would be confirmed as the active mechanism.
  • If the activity-dependent gradient attenuation is the true driver of low-rank self-compression, similar effects should appear in any architecture that divides its pre-activation by an activity-tracking signal——suggesting divisive normalization as a generic inductive bias for continuous latent-state models in machine learning, beyond the two cognitive tasks tested.
  • The paper's own long-horizon analysis shows the flat divisive manifold diffuses over time while subtractive discretization 'locks in' bounded error; this implies a natural design for robust circuits: divisive tracking for smooth input-driven integration plus a subtractive or homeostatic correction for long-timescale stability, a combination the paper does not test.
  • The authors explicitly leave two questions open: BPTT is not a biologically plausible learning rule, and the predicted eigenvalue spectra and rank compression await confirmation from large-scale neural population recordings; both are acknowledged limits, not established results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes the Recurrent Divisive Normalization Network (RDNN), an RNN in which a recurrent excitatory population is divided elementwise by an auxiliary inhibitory population, and trains it on angular velocity integration and memory-guided saccade tasks. The authors report that the RDNN learns low-rank, nearly continuous slow manifolds with a bimodal eigenvalue spectrum, whereas GRU/LSTM baselines discretize the state space, and that an explicitly factorized low-rank variant suffers from a higher-rank convergence anomaly. A central claim is that divisive normalization is 'mathematically essential' to prevent manifold shattering under time-varying inputs, supported by an ablation against a subtractive-inhibition variant. The paper also provides gradient-based explanations for the spectral separation and low-rank compression.

Significance. If the central claims hold, the paper would make a valuable contribution to computational neuroscience and recurrent-network design: it identifies a biologically grounded operation that induces robust, low-dimensional continuous attractors without explicit low-rank constraints, and it provides a rich set of dynamical systems diagnostics (fixed-point topology, eigenvalue spectra, effective rank, long-horizon stability). The empirical scope is substantial, including ORGaNICs, Neural ODE, and a double-integration task, and the low-rank factorization analysis is an interesting optimization finding. However, the manuscript's strongest claim of mathematical necessity rests on a confounded ablation and on a continuous-time analysis that is not shown to govern the discretized network actually trained. The mechanistic gradient account is evocative but not formally derived. These issues are fixable, but they are load-bearing for the paper's headline conclusions.

major comments (3)
  1. [§6 and Appendix E.1, Eq. (A-10)] The ablation is confounded. The Subtractive Network differs from the RDNN in two ways: division is replaced by subtraction, and every state update is passed through a hard ReLU rectification. The paper's own Appendix E.5.1 attributes attractor shattering to the discontinuous Heaviside gating induced by that rectification. Consequently, the observed shattering may be caused entirely by the extra threshold, not by the absence of division. A 2×2 control is needed: subtractive without rectification and RDNN with rectification. Without it, the abstract's 'mathematically essential' claim is unsupported; at most a sufficiency claim survives.
  2. [Eq. (1) vs Appendix B, Eqs. (A-1)/(A-9)] The continuous-time SDE in Eq. (1) uses multiplicative noise terms σ_R R dW_R and σ_G G dW_G and a continuous leak, while the discrete update used for training, Eq. (A-9), injects additive Gaussian noise ξ_R,t, ξ_G,t and does not multiply by R or G. The dynamical systems analysis in Sections 4 and Appendix E.3 is written for the continuous system, but the trained and evaluated network is the discrete one. No equivalence or error bound is shown. This matters because spectral properties and gradient-flow claims are transferred from one system to the other. The authors should either train with a discretization that matches Eq. (1), or re-run the analysis on the discrete map and state the theory accordingly.
  3. [Appendix E.3, Eqs. (A-15)–(A-19)] The derivation assumes the conclusion it claims to establish. The text states that BPTT 'mathematically drives' tangent eigenmodes to |λ|≈1 and normal eigenmodes to 0, but no optimization theorem is provided; the argument is that gradients would vanish or explode unless these conditions hold, which presumes the optimizer can and does enforce them. Similarly, the noise-covariance argument presumes that the optimizer successfully suppresses off-manifold variance. As written, this is a post-hoc heuristic consistent with the spectra, not a derivation. The paper should either state precise assumptions and prove the claimed convergence, or explicitly label this subsection as an interpretative mechanism supported by the empirical spectra.
minor comments (5)
  1. [Abstract and general text] There are multiple typographical issues, e.g., 'convergetorobust' in the abstract and inconsistent spacing in equations. A careful proofread is needed.
  2. [Eq. (1)] The equation is ambiguous as typeset: 'τ_R dR = (-R + Jf(R)+I(t)/η+G)dt' should be written as τ_R dR = (-R + (Jf(R)+I(t))/(η+G))dt + σ_R R dW_R, with parentheses made explicit.
  3. [Appendix B/Equation numbering] Appendix B labels its updates as (A-1), but later appendices refer to (A-9) and (A-10); the equation numbering should be made consistent throughout.
  4. [Figure A14 and Section 6] Figure A14 shows that the Subtractive Network achieves comparable or lower test NMSE than the RDNN on at least some configurations. The main text says 'competitive task performance,' but this deserves explicit discussion given the later functional-niche argument.
  5. [Reproducibility] No code or data availability statement is included. Given the detailed empirical analyses, releasing training and analysis code would substantially strengthen reproducibility.

Circularity Check

1 steps flagged

No central circularity; one self-confirming flatness metric and a confounded ablation weaken specific claims.

specific steps
  1. self definitional [Appendix D.2 (Identification of the Slow Manifold / Uniform Norm of the Manifold Flow); Section 4.2]
    "Within each angular bin, the state exhibiting the minimum drift speed was selected. ... we calculated the uniform norm (L∞ norm) of the vector field restricted to the slow manifold. Specifically, for all states h∈M on the identified manifold, we computed the maximum 1-step drift magnitude ... The RDNN achieves an exceptionally low uniform norm (on the order of 10−3 to 10−4), indicating an extremely flat energy landscape."

    The slow manifold M is constructed by selecting, in each angular bin, the state with the minimum drift speed among points below a 10^-3 threshold. The 'uniform norm' metric then reports the maximum of exactly that drift quantity over the selected states. Therefore a reported value of 10^-3–10^-4 is largely a restatement of the selection criterion, not an independent measurement of how flat the learned dynamics are. The absolute claim of an 'extremely flat energy landscape' is self-confirming, although relative comparisons to baselines using the same procedure are not invalidated.

full rationale

The paper's central empirical comparisons (RDNN vs GRU/LSTM/Neural-ODE, and RDNN vs Subtractive Network on static memory) are self-contained and externally checkable; the effective-rank measurements are computed from trained weights and are not fitted to the conclusions. The gradient-scaling analysis in Appendix E.2–E.4 is a straightforward differentiation of the RDNN update (A-9) and is appropriately hedged as correlational rather than presented as a rank-minimization theorem. Appendix E.3's spectral argument ('BPTT forces |λ|≈1', 'noise suppression drives |λ|→0') is a post-hoc consistency narrative rather than a rigorous derivation, but it is not circular because the observed spectrum is not used as a premise in a formal derivation. The main load-bearing weakness is the ablation: Appendix E.1 Eq. A-10 adds a hard rectification max(0,·) to the Subtractive Network that is absent from Eq. A-9, and Appendix E.5.1 attributes the shattering to the discontinuous Heaviside gating Θ_t introduced by that rectification. Thus the conclusion that divisive normalization is 'mathematically essential' is not established by the reported two-way comparison; a 2×2 control (division vs subtraction, with and without rectification) would be required. That is a confound or inference gap, not circularity. The one genuine circular measurement is the uniform-norm flatness metric: the manifold is selected as the lowest-drift states, and the uniform norm then reports the maximum drift of those same states, so the low absolute flatness values are partly by construction. This does not affect the relative baseline comparisons but does weaken the absolute 'extremely flat energy landscape' language. Overall, the paper does not reduce its central predictions to its inputs by construction; score 3 reflects one self-confirming secondary metric rather than a load-bearing circular derivation.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 1 invented entities

All network parameters (J, W, eta, tau, projections) are learned from the task data; none are independently measured from biology. The mechanistic story also assumes certain optimization outcomes and ignores cross-coupling terms. The empirical claims about trained networks are well specified, but the theoretical transfer from continuous SDE to implemented discrete dynamics is assumed.

free parameters (7)
  • Recurrent excitatory weight matrix J = learned, Softplus(J_raw)
    Determines the recurrent drive that forms the manifold; estimated from task data, no independent biological constraint.
  • Inhibitory projection matrix W = learned, Softplus(W_raw)
    Projects excitatory activity to G; controls the divisor and thus the gradient scaling.
  • Semi-saturation vector eta = learned, Softplus(eta_raw)+1e-5
    Sets baseline responsiveness and prevents division by zero; directly shapes the scale of the divisive term.
  • Time constants / leak rates alpha_R, alpha_G = sigmoid of learned raw parameters; initialized approximately 0.1
    Set the integration timescale and the effective leak of both populations.
  • Input/output projection weights = learned
    Map task inputs to R and decode outputs; influence what slow manifold is learnable.
  • State noise intensity sigma = 0.1 during training
    Chosen by hand; the noise-induced regularization argument in Appendix E.3 depends on it.
  • Initialization of leak rates and recurrent weights = alpha_raw mean -2.2, J_raw ~ N(-ln H, 1/sqrt(H))
    Biases the network toward slow integration; hand-chosen inductive biases that affect what dynamics are reachable.
axioms (6)
  • domain assumption Divisive normalization is a canonical cortical computation that provides dynamic gain control.
    Motivation from Carandini and Heeger (2012); the paper assumes this biological relevance transfers to learned memory circuits.
  • domain assumption A one-dimensional angular variable is adequately represented by a ring slow manifold and read out linearly from R.
    Task design and manifold identification assume the ring is the correct structure for these tasks.
  • ad hoc to paper BPTT on MSE together with injected state noise drives tangent eigenvalues of the learned dynamics to |lambda| approximately 1 and normal eigenvalues to 0.
    Used in Appendix E.3 to explain the observed spectral gap; this is an assumed optimization outcome, not proven.
  • ad hoc to paper The linearized Jacobian that ignores cross-coupling through G is sufficient for the gradient-flow analysis.
    Eq. A-15 explicitly ignores indirect cross-coupling 'for clarity'; the derivation depends on that approximation.
  • ad hoc to paper The discrete Euler-Maruyama update is a faithful discretization of the continuous SDE in Eq. (1).
    The paper's theoretical framing uses the continuous SDE, but the implemented update differs in the leak and noise terms; this equivalence is assumed rather than shown.
  • domain assumption Effective rank of the merged recurrent weight matrix reflects the dimensionality of the learned dynamics.
    Section 5 and Appendix J treat 99% energy rank of concatenated weight matrices as a proxy for dynamic dimensionality.
invented entities (1)
  • Auxiliary inhibitory gain-control population G no independent evidence
    purpose: Tracks network activity and divides the excitatory drive, implementing divisive normalization.
    G is a model component, not an empirically identified population; the paper does not provide a separate falsifiable handle on G itself beyond the behavioral predictions of the full network.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory." pith.science (2026). https://pith.science/paper/3TMVUMHT

@misc{pith2026260801947,
  author       = {Pith},
  title        = {Pith review of: Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TMVUMHT}},
  note         = {Machine review of arXiv:2608.01947}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a minimal and algebraically isolated model of dynamic division. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that this biophysical constraint allows the network to converge to robust, high-fidelity slow manifolds. Furthermore, we analyze the gradient dynamics of divisive normalization during Backpropagation Through Time (BPTT), showing that it introduces an activity-dependent local gradient scaling. This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.

Figures

Figures reproduced from arXiv: 2608.01947 by Chang Liu, Dahui Wang, Jie Su, Tianyi Qian, Weiwei Wang, Zhaotian Gu.

Figure 1
Figure 1. Figure 1: Overview of the proposed RDNN and its performance on continuous working memory tasks. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dynamical topology and spectral properties of the learned slow manifolds. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effective rank scaling and explicit low-rank training dynamics. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of dynamical topology between divisive and subtractive networks. Statistical counts of [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The 99% energy effective rank of the merged recurrent weight matrices is plotted against hidden [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 16 canonical work pages

  1. [1]

    Implicit Regularization in Matrix Factorization , booktitle =

    Gunasekar, Suriya and Woodworth, Blake and Bhojanapalli, Srinadh and Neyshabur, Behnam and Srebro, Nathan , year = 2017, series =. Implicit Regularization in Matrix Factorization , booktitle =

    2017

  2. [2]

    Implicit Regularization in Deep Matrix Factorization , booktitle =

    Arora, Sanjeev and Cohen, Nadav and Hu, Wei and Luo, Yuping , year = 2019, publisher =. Implicit Regularization in Deep Matrix Factorization , booktitle =

    2019

  3. [3]

    How Do Neurons Operate on Sparse Distributed Representations?

    Ahmad, Subutai and Hawkins, Jeff , year = 2016, publisher =. How Do Neurons Operate on Sparse Distributed Representations?. doi:10.48550/ARXIV.1601.00720 , urldate =

    Pith/arXiv arXiv doi:10.48550/arxiv.1601.00720 2016

  4. [4]

    , year = 2009, journal =

    Ayaz, Asli and Chance, Frances S. , year = 2009, journal =. Gain. doi:10.1152/jn.90547.2008 , urldate =

    arXiv 2009

  5. [5]

    Capacity-

    Battista, Aldo and Monasson, R. Capacity-. Physical Review Letters , volume =. doi:10.1103/PhysRevLett.124.048302 , urldate =

    work page doi:10.1103/physrevlett.124.048302

  6. [6]

    , author =

    Theory of Orientation Tuning in Visual Cortex. , author =. Proceedings of the National Academy of Sciences , volume =. doi:10.1073/pnas.92.9.3844 , urldate =

    doi:10.1073/pnas.92.9.3844

  7. [7]

    PLOS Computational Biology , volume =

    Coding with Transient Trajectories in Recurrent Neural Networks , author =. PLOS Computational Biology , volume =. doi:10.1371/journal.pcbi.1007655 , urldate =

    work page doi:10.1371/journal.pcbi.1007655

  8. [8]

    Current Opinion in Neurobiology , volume =

    Basic Mechanisms for Graded Persistent Activity: Discrete Attractors, Continuous Attractors, and Dynamic Representations , author =. Current Opinion in Neurobiology , volume =. doi:10.1016/S0959-4388(03)00050-3 , urldate =

    work page doi:10.1016/s0959-4388(03)00050-3

  9. [9]

    PLOS Computational Biology , volume =

    Learning Divisive Normalization in Primary Visual Cortex , author =. PLOS Computational Biology , volume =. doi:10.1371/journal.pcbi.1009028 , urldate =

    doi:10.1371/journal.pcbi.1009028

  10. [10]

    Nature Reviews Neuroscience , volume =

    Normalization as a Canonical Neural Computation , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/nrn3136 , urldate =

    doi:10.1038/nrn3136

  11. [11]

    Nature Neuroscience , volume =

    The Intrinsic Attractor Manifold and Population Dynamics of a Canonical Cognitive Circuit across Waking and Sleep , author =. Nature Neuroscience , volume =. doi:10.1038/s41593-019-0460-x , urldate =

    work page doi:10.1038/s41593-019-0460-x

  12. [12]

    Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David K , editor =. Neural Ordinary Differential Equations , booktitle =

  13. [13]

    , year = 2000, journal =

    Compte, A. , year = 2000, journal =. Synaptic. doi:10.1093/cercor/10.9.910 , urldate =

    doi:10.1093/cercor/10.9.910 2000

  14. [14]

    and Qi, Xue-Lian and Wang, Min and Arnsten, Amy F.T

    Constantinidis, Christos and Funahashi, Shintaro and Lee, Daeyeol and Murray, John D. and Qi, Xue-Lian and Wang, Min and Arnsten, Amy F.T. , year = 2018, journal =. Persistent. doi:10.1523/JNEUROSCI.2486-17.2018 , urldate =

    work page doi:10.1523/jneurosci.2486-17.2018 2018

  15. [15]

    doi:10.48550/ARXIV.1804.10200 , urldate =

    The Loss Landscape of Overparameterized Neural Networks , author =. doi:10.48550/ARXIV.1804.10200 , urldate =

    Pith/arXiv arXiv doi:10.48550/arxiv.1804.10200

  16. [16]

    Gradient Descent Finds Global Minima of Deep Neural Networks , booktitle =

    Du, Simon and Lee, Jason and Li, Haochuan and Wang, Liwei and Zhai, Xiyu , editor =. Gradient Descent Finds Global Minima of Deep Neural Networks , booktitle =

  17. [17]

    Physical Review Research , volume =

    Lyapunov Spectra of Chaotic Recurrent Neural Networks , author =. Physical Review Research , volume =. doi:10.1103/PhysRevResearch.5.043044 , urldate =

    doi:10.1103/physrevresearch.5.043044

  18. [18]

    Ermentrout, G. B. and Kopell, N. , year = 1986, journal =. Parabolic. doi:10.1137/0146017 , urldate =

    doi:10.1137/0146017 1986

  19. [19]

    Nature Reviews Neuroscience , volume =

    Noise in the Nervous System , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/nrn2258 , urldate =

    doi:10.1038/nrn2258

  20. [20]

    Nature Machine Intelligence , volume =

    Gradient-Based Learning Drives Robust Representations in Recurrent Neural Networks by Balancing Compression and Expansion , author =. Nature Machine Intelligence , volume =. doi:10.1038/s42256-022-00498-0 , urldate =

    work page doi:10.1038/s42256-022-00498-0

  21. [21]

    Journal of Differential Equations , volume =

    Geometric Singular Perturbation Theory for Ordinary Differential Equations , author =. Journal of Differential Equations , volume =. doi:10.1016/0022-0396(79)90152-9 , urldate =

    doi:10.1016/0022-0396(79)90152-9

  22. [22]

    and Lund, J

    Fitzpatrick, D. and Lund, J. S. and Schmechel, D. E. and Towles, A. C. , year = 1987, journal =. Distribution of. doi:10.1002/cne.902640107 , urldate =

    work page doi:10.1002/cne.902640107 1987

  23. [23]

    PLOS Computational Biology , volume =

    Slow Manifolds within Network Dynamics Encode Working Memory Efficiently and Robustly , author =. PLOS Computational Biology , volume =. doi:10.1371/journal.pcbi.1009366 , urldate =

    work page doi:10.1371/journal.pcbi.1009366

  24. [24]

    and Koutnik, Jan and Steunebrink, Bas R

    Greff, Klaus and Srivastava, Rupesh K. and Koutnik, Jan and Steunebrink, Bas R. and Schmidhuber, Jurgen , year = 2017, journal =. doi:10.1109/TNNLS.2016.2582924 , urldate =

    arXiv 2017

  25. [25]

    Nature , volume =

    Digital Selection and Analogue Amplification Coexist in a Cortex-Inspired Silicon Circuit , author =. Nature , volume =. doi:10.1038/35016072 , urldate =

    doi:10.1038/35016072

  26. [26]

    Visual Neuroscience , volume =

    Normalization of Cell Responses in Cat Striate Cortex , author =. Visual Neuroscience , volume =. doi:10.1017/S0952523800009640 , urldate =

    doi:10.1017/s0952523800009640

  27. [27]

    and Mackey, Wayne E

    Heeger, David J. and Mackey, Wayne E. , year = 2019, journal =. Oscillatory Recurrent Gated Neural Integrator Circuits (. doi:10.1073/pnas.1911633116 , urldate =

    work page doi:10.1073/pnas.1911633116 2019

  28. [28]

    and Gerstner, Wulfram , year = 2014, journal =

    Hennequin, Guillaume and Vogels, Tim P. and Gerstner, Wulfram , year = 2014, journal =. Optimal. doi:10.1016/j.neuron.2014.04.045 , urldate =

    work page doi:10.1016/j.neuron.2014.04.045 2014

  29. [29]

    and Koch, Christof , year = 1997, journal =

    Holt, Gary R. and Koch, Christof , year = 1997, journal =. Shunting. doi:10.1162/neco.1997.9.5.1001 , urldate =

    doi:10.1162/neco.1997.9.5.1001 1997

  30. [30]

    Jordan, Ian D. and Sok. Gated. Frontiers in Computational Neuroscience , volume =. doi:10.3389/fncom.2021.678158 , urldate =

    arXiv 2021

  31. [31]

    Nature Reviews Neuroscience , volume =

    Attractor and Integrator Networks in the Brain , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/s41583-022-00642-0 , urldate =

    doi:10.1038/s41583-022-00642-0

  32. [32]

    and Ermentrout, Bard , year = 2013, journal =

    Kilpatrick, Zachary P. and Ermentrout, Bard , year = 2013, journal =. Wandering. doi:10.1137/120877106 , urldate =

    work page doi:10.1137/120877106 2013

  33. [33]

    Nature Neuroscience , volume =

    Model for a Robust Neural Integrator , author =. Nature Neuroscience , volume =. doi:10.1038/nn893 , urldate =

    work page doi:10.1038/nn893

  34. [34]

    A Recurrent Neural Network without Chaos , booktitle =

    Laurent, Thomas and. A Recurrent Neural Network without Chaos , booktitle =

  35. [35]

    and Jaitly, Navdeep and Hinton, Geoffrey E

    Le, Quoc V. and Jaitly, Navdeep and Hinton, Geoffrey E. , year = 2015, publisher =. A. doi:10.48550/ARXIV.1504.00941 , urldate =

    Pith/arXiv arXiv doi:10.48550/arxiv.1504.00941 2015

  36. [36]

    Symmetry,

    Li, Xingguo and Lu, Junwei and Arora, Raman and Haupt, Jarvis and Liu, Han and Wang, Zhaoran and Zhao, Tuo , year = 2019, journal =. Symmetry,. doi:10.1109/TIT.2019.2898663 , urldate =

    arXiv 2019

  37. [37]

    Nature Reviews Neuroscience , volume =

    Backpropagation and the Brain , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/s41583-020-0277-3 , urldate =

    doi:10.1038/s41583-020-0277-3

  38. [38]

    Mastrogiuseppe, Francesca and Ostojic, Srdjan , year = 2018, journal =. Linking. doi:10.1016/j.neuron.2018.07.003 , urldate =

    doi:10.1016/j.neuron.2018.07.003 2018

  39. [39]

    PLOS Computational Biology , volume =

    Stochastic Activity in Low-Rank Recurrent Neural Networks , author =. PLOS Computational Biology , volume =. doi:10.1371/journal.pcbi.1013371 , urldate =

    work page doi:10.1371/journal.pcbi.1013371

  40. [40]

    and Miller, Kenneth D

    Murphy, Brendan K. and Miller, Kenneth D. , year = 2009, journal =. Balanced. doi:10.1016/j.neuron.2009.02.005 , urldate =

    doi:10.1016/j.neuron.2009.02.005 2009

  41. [41]

    Understanding and Improving Layer Normalization , booktitle =

    Xu, Jingjing and Sun, Xu and Zhang, Zhiyuan and Zhao, Guangxiang and Lin, Junyang , editor =. Understanding and Improving Layer Normalization , booktitle =

  42. [42]

    Universality and Individuality in Neural Dynamics across Large Populations of Recurrent Networks , booktitle =

    Maheswaranathan, Niru and Williams, Alex and Golub, Matthew and Ganguli, Surya and Sussillo, David , editor =. Universality and Individuality in Neural Dynamics across Large Populations of Recurrent Networks , booktitle =

  43. [43]

    Back to the Continuous Attractor , booktitle =

    S. Back to the Continuous Attractor , booktitle =

  44. [44]

    Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry: Theory and Practice , booktitle =

    Pennington, Jeffrey and Schoenholz, Samuel and Ganguli, Surya , editor =. Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry: Theory and Practice , booktitle =

  45. [45]

    Revisiting the Landscape of Matrix Factorization , booktitle =

    Valavi, Hossein and Liu, Sulin and Ramadge, Peter , editor =. Revisiting the Landscape of Matrix Factorization , booktitle =

  46. [46]

    and Moroshko, Edward and Savarese, Pedro and Golan, Itay and Soudry, Daniel and Srebro, Nathan , editor =

    Woodworth, Blake and Gunasekar, Suriya and Lee, Jason D. and Moroshko, Edward and Savarese, Pedro and Golan, Itay and Soudry, Daniel and Srebro, Nathan , editor =. Kernel and Rich Regimes in Overparametrized Models , booktitle =

  47. [47]

    Deep Sparse Rectifier Neural Networks , booktitle =

    Glorot, Xavier and Bordes, Antoine and Bengio, Yoshua , editor =. Deep Sparse Rectifier Neural Networks , booktitle =

  48. [48]

    Proceedings of the 32nd International Conference on Machine Learning , author =

    Batch Normalization:. Proceedings of the 32nd International Conference on Machine Learning , author =

  49. [49]

    An Empirical Exploration of Recurrent Network Architectures , booktitle =

    Jozefowicz, Rafal and Zaremba, Wojciech and Sutskever, Ilya , editor =. An Empirical Exploration of Recurrent Network Architectures , booktitle =

  50. [50]

    Sharp Minima Can Generalize for Deep Nets , booktitle =

    Dinh, Laurent and Pascanu, Razvan and Bengio, Samy and Bengio, Yoshua , editor =. Sharp Minima Can Generalize for Deep Nets , booktitle =

  51. [51]

    No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis , booktitle =

    Ge, Rong and Jin, Chi and Zheng, Yi , editor =. No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis , booktitle =

  52. [52]

    Algorithmic Regularization in Over-Parameterized Matrix Sensing and Neural Networks with Quadratic Activations , booktitle =

    Li, Yuanzhi and Ma, Tengyu and Zhang, Hongyang , editor =. Algorithmic Regularization in Over-Parameterized Matrix Sensing and Neural Networks with Quadratic Activations , booktitle =

  53. [53]

    and Huerta, Ram

    Rabinovich, Mikhail I. and Huerta, Ram. Transient. PLoS Computational Biology , volume =. doi:10.1371/journal.pcbi.1000072 , urldate =

    work page doi:10.1371/journal.pcbi.1000072

  54. [54]

    Rajan, Kanaka and Abbott, L. F. , year = 2006, journal =. Eigenvalue. doi:10.1103/PhysRevLett.97.188104 , urldate =

    doi:10.1103/physrevlett.97.188104 2006

  55. [55]

    and Martiniani, Stefano , editor =

    Rawat, Shivang and Heeger, David J. and Martiniani, Stefano , editor =. Unconditional Stability of a Recurrent Neural Circuit Implementing Divisive Normalization , booktitle =

  56. [56]

    Renart, Alfonso and Song, Pengcheng and Wang, Xiao-Jing , year = 2003, journal =. Robust. doi:10.1016/S0896-6273(03)00255-1 , urldate =

    work page doi:10.1016/s0896-6273(03)00255-1 2003

  57. [57]

    , year = 2017, journal =

    Sawada, Tadamasa and Petrov, Alexander A. , year = 2017, journal =. The Divisive Normalization Model of. doi:10.1152/jn.00821.2016 , urldate =

    arXiv 2017

  58. [58]

    Proceedings of the National Academy of Sciences , volume =

    How the Brain Keeps the Eyes Still , author =. Proceedings of the National Academy of Sciences , volume =. doi:10.1073/pnas.93.23.13339 , urldate =

    doi:10.1073/pnas.93.23.13339

  59. [59]

    Impact of

    Shao, Yuxiu and Dahmen, David and Recanatesi, Stefano and. Impact of. PRX Life , volume =. doi:10.1103/PRXLife.3.023008 , urldate =

    work page doi:10.1103/prxlife.3.023008

  60. [60]

    Nature Reviews Neuroscience , volume =

    Neuronal Arithmetic , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/nrn2864 , urldate =

    doi:10.1038/nrn2864

  61. [61]

    , year = 2018, edition =

    Strogatz, Steven H. , year = 2018, edition =. Nonlinear. doi:10.1201/9780429492563 , urldate =

    doi:10.1201/9780429492563 2018

  62. [62]

    Opening the

    Sussillo, David and Barak, Omri , year = 2013, journal =. Opening the. doi:10.1162/NECO_a_00409 , urldate =

    doi:10.1162/neco_a_00409 2013

  63. [63]

    Nature Neuroscience , volume =

    Bump Attractor Dynamics in Prefrontal Cortex Explains Behavioral Precision in Spatial Working Memory , author =. Nature Neuroscience , volume =. doi:10.1038/nn.3645 , urldate =

    doi:10.1038/nn.3645

  64. [64]

    Advances in Neural Information Processing Systems , author =

    Autoformer:. Advances in Neural Information Processing Systems , author =

  65. [65]

    The Twelfth International Conference on Learning Representations , author =

    How Over-Parameterization Slows down Gradient Descent in Matrix Sensing:. The Twelfth International Conference on Learning Representations , author =

  66. [66]

    and Graham, Michael D

    Zeng, Kevin and Linot, Alec J. and Graham, Michael D. , year = 2022, journal =. Data-Driven Control of Spatiotemporal Chaos with Reduced-Order Neural. doi:10.1098/rspa.2022.0297 , urldate =

    arXiv 2022

  67. [67]

    The Journal of Neuroscience , volume =

    Representation of Spatial Orientation by the Intrinsic Dynamics of the Head-Direction Cell Ensemble: A Theory , author =. The Journal of Neuroscience , volume =. doi:10.1523/JNEUROSCI.16-06-02112.1996 , urldate =

    arXiv 1996

  68. [68]

    Transactions on Machine Learning Research , issn =

    Symmetry in Neural Network Parameter Spaces , author =. Transactions on Machine Learning Research , issn =

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.