Pith. sign in

REVIEW 4 major objections 6 minor 44 references

RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes ERNNs, recurrent networks whose hidden states are updated to the equilibrium point of an implicit ODE, and proves that this makes the product of Jacobians across time equal to $(-1)^{m-n}I$, so backpropagated gradient…

desk verdict A genuinely new RNN construction with a clean norm-preserving identity at exact equilibrium, but the finite-K unrolled model used in training is not covered by the theorem; worth serious review. read the letter →

arxiv 1908.08574 v2 pith:FNNTEDSD submitted 2019-08-22 cs.LG stat.ML

Anil Kag , Ziming Zhang , Venkatesh Saligrama This is my paper · ORCID
classification cs.LGstat.ML
keywords EquilibriatedRecurrentNeuralNetworksvanishinggradientsexplodingequilibriummanifoldimplicitODEfixed-pointrecursionnorm-preservingJacobianlong-termdependencies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ERNNs (Equilibriated Recurrent Neural Networks) are a family of recurrent networks that, at each time step, solve for the equilibrium of an ordinary differential equation instead of taking one explicit vector-field step. The paper's central claim is that on this equilibrium manifold the Jacobian of each state transition is exactly $-I$, so the product of Jacobians over any time span is $(-1)^{m-n}I$ and backpropagated gradients have norm exactly $1$. This would remove exponential vanishing and exploding gradients at their source, without gates, unitary constraints, or damping factors. The paper reports that on long-term-dependence benchmarks the resulting models match or exceed gated and unitary baselines in accuracy while training 3–10x faster and using 1.5–3x fewer parameters. The argument requires the state to be an exact equilibrium and the activation gradient to be nonzero, while experiments use ReLU and only $K\le 5$ fixed-point iterations, so the identity holds only to the extent that approximation error is negligible.

What carries the argument

The load-bearing object is the equilibrium manifold $\mathcal{M}(h_{k-1}, x_k) = \{h : f(h + h_{k-1}, x_k;\alpha) - \gamma(h + h_{k-1}) = 0\}$, combined with the implicit-function-theorem derivative identity. Differentiating the equilibrium condition with respect to $h_{k-1}$ gives $[\nabla f - \gamma I](\partial h_{\mathrm{eq}}/\partial h_{k-1} + I) = 0$, so, under the nonsingularity assumption, $\partial h_{\mathrm{eq}}/\partial h_{k-1} = -I$. The practical solver is the fixed-point recursion $h_k^{(i+1)} = h_k^{(i)} + \eta_k^{(i)}[f(h_k^{(i)} + h_{k-1}, x_k) - (h_k^{(i)} + h_{k-1})]$, and the paper's Theorem 2 ties its local linear convergence to an inexact-Newton condition. This identity carries the whole argument: it converts a product of $m-n$ Jacobians into a signed identity matrix.

What would settle it

Take a trained ERNN with ReLU and a fixed number of solver steps and use automatic differentiation through the actual unrolled graph to compute the Jacobian product $\partial h_T/\partial h_1$ on sequences that drive many hidden units into saturation; if its singular values deviate substantially from 1 on any such sequence, the identity transition mapping does not hold in the trained model, and the paper's norm-preservation claim rests on an unquantified approximation.

Watch

Extended reading notes

Core claim

The central discovery is that if the hidden state of a recurrent network is defined implicitly as the equilibrium $h_k^*$ of $h' = f(h + h_{k-1}, x_k) - \gamma(h + h_{k-1})$, then the map from the previous state $h_{k-1}$ to the new equilibrium has Jacobian $\partial h_k/\partial h_{k-1} = -I$ on the equilibrium manifold, provided $\nabla f - \gamma I$ is nonsingular and the activation gradient does not vanish. Iterating this identity gives $\partial h_m/\partial h_n = (-1)^{m-n}I$, hence the norm of backpropagated gradients is exactly 1 for any distance in time. The paper argues that this identity is what makes training stable: error signals neither decay nor explode, and long-term dependencies remain reachable. It further proves local linear convergence of the Euler fixed-point recursion used to find the equilibrium, shows equilibria are locally asymptotically stable when eigenvalues of $\nabla\varphi\,U - \gamma I$ are negative, and reports experiments in which ERNNs match or exceed gated and unitary baselines on sequence-classification and language-modeling benchmarks while training several times faster.

Load-bearing premise

The load-bearing premise is that the hidden state at every time step is exactly (or close enough to) an equilibrium point of the implicit ODE, because the identity $\partial h_k/\partial h_{k-1} = -I$ is derived by differentiating the equilibrium condition; if the fixed-point recursion is cut off before convergence, or the activation gradient vanishes, the norm-preservation proof no longer applies.

Editorial extensions

If this is right

  • Backpropagated gradients over any time span have norm exactly 1, so long-term dependencies become reachable without gating, unitary constraints, or specially chosen learning rates.
  • Training converges substantially faster: the paper reports 3–10x speedups and 1.5–3x model-size reductions relative to gated and unitary baselines.
  • Because the identity comes from the equilibrium condition rather than the specific activation, the construction extends to deep transition blocks and arbitrary smooth transition functions.
  • Unrolling the fixed-point recursion for K steps gives a feed-forward training graph; larger K better approximates the equilibrium and empirically improves both convergence rate and accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The norm-preservation theorem applies only at exact equilibrium; for truncated unrolling, the relevant quantity is how fast $\|h_k^{(i)} - h_k^*\|$ shrinks with $i$, so an adaptive step count or residual-based stopping rule could make the guarantee practical and is directly testable.
  • The same implicit-equilibrium construction could be applied to depth in feed-forward or attention networks, not just time, potentially giving identity Jacobians through arbitrarily deep layers.
  • Because Lemma 1 requires $\nabla\varphi$ to be nonzero, ReLU in its saturated region violates the proof's conditions; testing ERNN with activations whose gradients are bounded away from zero (or with a smoothed ReLU) would separate the norm-preservation mechanism from the empirical gains.
  • The input-variation analysis suggests that slowly varying inputs confine equilibrium states to a low-dimensional subspace related to the input projection; if true, this offers a principled explanation for the observed small model sizes and a route to further compression.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Equilibriated Recurrent Neural Networks (ERNNs), which update the hidden state by solving a fixed-point equation so that the state evolves on the equilibrium manifold of an ODE. The central theoretical claim is Theorem 1: at exact equilibrium, ∂h_m/∂h_n = (-1)^{m-n} I, so backpropagated transition Jacobians have unit norm and vanishing/exploding gradients are circumvented. In practice, the equilibrium is computed by a K-step damped Euler recursion (Eq. 22) with a low-rank parameterization (Eq. 10), and experiments on long-term-dependence benchmarks report accuracy gains over RNN, LSTM, GRU, FastRNN, and FastGRNN baselines, with 3-10x training speedups and smaller models. The main unresolved issue is that the theorem's assumptions (exact equilibrium, smooth non-vanishing activation, non-singular Jacobian) are not satisfied by the deployed finite-K ReLU model, and the paper supplies no K-dependent gradient error bound.

Significance. If the central claim were fully supported, the contribution would be significant: a simple, parameter-light recurrent cell with a provable norm-preserving transition Jacobian would address a long-standing training difficulty without unitary constraints or gating. The paper also demonstrates strong empirical results, including the best accuracy among the compared single-cell models on several datasets (e.g., 96.30% vs 95.48% on HAR-2, 94.71% vs 93.18% on Google-12, 54.50% vs 48.63% on Noisy-CIFAR), with consistently smaller models and markedly lower training time. The supplementary contributes useful empirical checks of linear fixed-point convergence and eigenvalue negativity. However, the theoretical guarantee as stated applies to the exact equilibrium map, not to the finite-K unrolled network that is actually trained; absent an error analysis, the paper's headline claim that ERNNs 'overcome' vanishing and exploding gradients is not established. The empirical results are interesting enough to warrant a major revision rather than rejection.

major comments (4)
  1. [Sec. 2.1, Eq. (22); Sec. 3, Table 3] Theorem 1 and Lemma 1 establish ∂heq/∂hk−1 = −I only at an exact equilibrium heq satisfying Eq. (8), but the model actually trained is the K-step unrolled Euler recursion in Eq. (22), with K=1,3,5 in Table 3 (and K=1,2,3 for the large-scale sets). For K=1 the transition Jacobian is ∂hk/∂hk−1 = (1−2η)I + 2η∇φ(·)U under the parametrization used in Eq. (10), which equals −I only for special values of η and ∇φU and is not shown to be close to −I for the trained parameters. For K=3 and K=5 no bound is given on the distance between the finite-K Jacobian and (−1)^K I, and Theorem 3, stated in the supplementary, concerns convergence of the fixed-point iterates in state value, not the Jacobian of the unrolled map. Consequently the paper's central claim that ERNNs 'overcome' vanishing and exploding gradients is not supported by the stated theorems for the architecture used in the experiments; a K-dependent error analysis or a direct Jacobian-norm verification across K and datasets is needed.
  2. [Sec. 2, Lemma 1; Sec. 3, hyper-parameters] Lemma 1 assumes a smooth activation with non-vanishing ∇φ and a non-singular Jacobian ∇f − γI, while the experiments use ReLU throughout (Sec. 3), which is non-smooth at zero and has ∇φ = 0 on the negative orthant. The supplementary's Fig. 5 checks negativity of eigenvalues of ∇φU − I on HAR-2 after training, but this is a post-hoc check on one dataset and does not establish the non-singularity or non-vanishing conditions during training on all datasets. The stability condition of Lemma 2 is also not enforced during optimization, so the training trajectory may leave the regime in which the equilibrium-manifold analysis applies.
  3. [Sec. 2, Eqs. (7)-(9)] The identity ∂heq/∂hk−1 = −I is a direct algebraic consequence of the equilibrium constraint itself: differentiating f(h_eq+h_{k−1}) − γ(h_eq+h_{k−1}) = 0 with respect to h_{k−1} gives [∇f−γI](∂heq/∂hk−1 + I) = 0, so the conclusion follows as soon as the Jacobian is invertible. This means Theorem 1 is a restatement of the architecture's defining constraint rather than an independent mechanism that can be expected to transfer automatically to the finite-K approximation; the substantive claim that needs support is the gradient behavior of the unrolled recursion used in practice, not of the exact equilibrium map.
  4. [Fig. 2; Sec. 3, Evaluation Criteria] The only direct empirical evidence for the norm-preserving gradient claim is Fig. 2, which plots gradient magnitudes for one dataset (HAR-2) and does not report the actual transition-Jacobian norm, the deviation from (−1)^{T−1}I, or confidence intervals; it is a single trajectory-level curve, not a systematic verification across the datasets and K values used in Table 3. A quantitative check of ||∂h_T/∂h_1|| over training, or a histogram of eigenvalues of the finite-K transition Jacobian, would be needed to support the claim that practical ERNNs preserve gradients.
minor comments (6)
  1. [Table 3 caption] Please correct the repeated misspelling of 'equilibrium' (for example, 'equillibrium' in the caption of Table 3).
  2. [Sec. 2, Lemma 1 proof; Sec. 2.1, Theorem 2] Eq. (13) is cited before it is defined in the main text; the ODE in Eq. (6) should be numbered consistently so that the proof of Lemma 1 and Theorem 2 refer to an equation already introduced.
  3. [Sec. 3, K values] The text says results are shown for K=1,3,5 for all datasets except the large-scale ones, but Table 3 uses K=2 for Pixel-MNIST and Yelp-5; please reconcile this inconsistency.
  4. [Sec. 3, experimental setup] The sentence describing the hardware says 'Intel Xeon 2.60 GHz GPU with 20 cores'; this should presumably read 'CPU'.
  5. [Sec. 3, evaluation criteria] The claim of 'similar prediction cost to vanilla RNNs' is not consistently supported by Table 3: for example, ERNN(K=5) reports a test time of 0.03 ms versus 0.01 ms for FastRNN on HAR-2 and 0.05 ms versus 0.01 ms on Google-12; the claim should be qualified.
  6. [Fig. 2] Fig. 2 would be stronger with multiple seeds and with a plot of the actual Jacobian norm ||∂h_T/∂h_1|| rather than a loss-gradient magnitude, which also depends on the loss and the intervening parameter updates.

Circularity Check

1 steps flagged · score 7.0 of 10

The norm-preserving gradient identity is a restatement of the equilibrium constraint, not an independent derivation; the deployed K-step model is not covered by the theorem.

  1. self definitional [Sec. 2, Lemma 1 (Eq. 8) and Theorem 1 (Eq. 9); proof of Lemma 1]
    "heq∈M (hk−1, xk) ={h∈ RD|f (h + hk−1, xk;α)−γ (h + hk−1) = 0}. ... By taking the partial derivatives w.r.t. hk−1 in Eq. 13, at the equilibrium points we have [∇f (heq + hk−1;α)−γI][ ∂heq/∂hk−1 + I] = 0. The proof follows from our assumptions."

    The equilibrium manifold is defined by the equation f(h + h_{k-1}) - γ(h + h_{k-1}) = 0. Differentiating this defining equation with respect to h_{k-1} immediately gives (∇f - γI)(∂h/∂h_{k-1} + I) = 0, and the non-singularity assumption then forces ∂h/∂h_{k-1} = -I. Thus the 'Identity Transition Mapping' is an algebraic rewrite of the model's defining constraint, not a derived property with independent content. Theorem 1 then simply multiplies this forced identity over time steps, so the claimed avoidance of vanishing/exploding gradients is hard-wired into the architecture by construction. The paper presents this as a discovered benefit, but it is equivalent to the ansatz itself.

full rationale

The paper's central theoretical claim, Theorem 1, asserts that backpropagated state-transition Jacobians equal (-1)^{m-n}I and therefore have unit norm. That conclusion is a direct consequence of the defining equilibrium constraint: differentiating f(h+h_{k-1}) - γ(h+h_{k-1}) = 0 with respect to h_{k-1} yields the identity without any additional modeling assumptions beyond non-singularity. Hence the norm-preserving result is self-definitional: the model is constructed so that its state satisfies an equation whose derivative is -I, and the theorem merely restates that construction. This is not a case of fitting parameters to data and then predicting the same data; it is a case of a 'prediction' that is equivalent to the input definition. No load-bearing self-citation chain is present; references to prior work (e.g., FastGRNN, AntisymmetricRNN) are comparisons, not justifications of the core identity. The empirical accuracy and speedup results are independent content and are not circular. However, the core gradient claim is further weakened by a gap between Theorem 1's assumptions and the trained model: experiments use K=1,3,5 unrolled Euler steps (Eq. 22) with ReLU activations, while Lemma 1 requires exact equilibrium, non-vanishing ∇φ, and a non-singular Jacobian. The paper acknowledges that Fig. 2's variance arises from 'approximation of fixed points,' which confirms that the theorem does not strictly apply to the deployed model. That gap is a correctness risk rather than circularity, but it reinforces that the theoretical guarantee is not independently load-bearing. Overall, the derivation chain's key theoretical payoff reduces to the model definition by construction, justifying a circularity score of 7.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on the architecture's implicit-equilibrium definition and on smoothness/stability assumptions that are only partially verified in practice. The main free parameters are the damping γ, the fixed-point step size, the low-rank dimension, and the number of iterations K; all are hand-chosen or learned without guarantees of satisfying the theory's conditions. No physically novel entities are introduced.

free parameters (4)
  • γ (damping multiplier) = set to 1 (not learned)
    In Sec. 2, the authors state they simply set γ to one rather than learning it. The stability condition (Lemma 2) requires ||U|| ≤ γ, so choosing γ=1 is not guaranteed to hold with the learned U = I + VH; it is checked only empirically (Fig. 5).
  • Fixed-point step size η_k^(i) = initialized to 1e-2, learnable
    Eq. 11 uses a learnable step size in the fixed-point recursion; its value affects convergence and the finite-K Jacobian.
  • Low-rank dimension d1 = not specified in the paper
    Eq. 10 uses a low-rank factorization U = I + VH to reduce parameters; the factorization rank is a free design choice not reported.
  • Number of fixed-point iterations K = 1, 2, 3, or 5 depending on dataset
    K is chosen by hand and controls how close the iterate is to the equilibrium; the paper reports results for different K without an automatic selection criterion.
assumptions (5)
  • domain assumption Differentiability and non-vanishing gradient of the activation function
    Lemma 1 requires ∇φ non-vanishing, stated as following for sigmoid/tanh; the experiments use ReLU, where gradients vanish on half the domain. This is acknowledged in Sec. 1.1 but not reconciled with Sec. 3.
  • domain assumption Existence and local asymptotic stability of equilibrium points
    Lemma 2 assumes the Jacobian of F at equilibrium has negative eigenvalues. The paper suggests conditions (||U|| ≤ γ) but does not enforce them during training; Fig. 5 verifies them empirically for one dataset.
  • domain assumption Unique fixed point / well-posedness of the implicit equation
    The equilibrium manifold M(h_{k-1}, x_k) is used to define the state update; if multiple fixed points exist, the map h_k = s* - h_{k-1} may not be well-defined or differentiable as a function of h_{k-1}. The paper assumes non-singularity but does not establish uniqueness.
  • standard math Standard fixed-point convergence theorems (Thm 2.3 in Dembo et al. 1982)
    Theorem 3 uses this external convergence result for inexact Newton methods.
  • ad hoc to paper Inputs are low-dimensional and vary slowly
    The low-rank parameterization P = U = I + VH is justified by the claim that signal vectors are low-dimensional and slowly varying (Sec. 2). This is not validated on all datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?." pith.science (2026). https://pith.science/paper/FNNTEDSD

@misc{pith2026190808574,
  author       = {Pith},
  title        = {Pith review of: RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNNTEDSD}},
  note         = {Machine review of arXiv:1908.08574}
}
read the original abstract

Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recurrent units, well-chosen parametric constraints, and skip-connections, we develop a novel perspective that seeks to evolve the hidden state on the equilibrium manifold of an ordinary differential equation (ODE). We propose a family of novel RNNs, namely {\em Equilibriated Recurrent Neural Networks} (ERNNs) that overcome the gradient decay or explosion effect and lead to recurrent models that evolve on the equilibrium manifold. We show that equilibrium points are stable, leading to fast convergence of the discretized ODE to fixed points. Furthermore, ERNNs account for long-term dependencies, and can efficiently recall informative aspects of data from the distant past. We show that ERNNs achieve state-of-the-art accuracy on many challenging data sets with 3-10x speedups, 1.5-3x model size reduction, and with similar prediction cost relative to vanilla RNNs.

Figures

Figures reproduced from arXiv: 1908.08574 by the authors.

Figure 1
Figure 1. Phase-space trajectory of RNN, FastRNN, ERNN. X-axis denotes 1st dimension, and Y-axis 2nd dimension of 2D hidden state subject to random walk input with variance 10. Phase-Space Plot [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison between RNN and ERNN on the magnitudes of gradients. To verify Theorem. 1 empirically, we train RNN and ERNN on the HAR-2 data set (see more details in Sec. 3), respectively, and plot in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustration of network blocks for computing fixed points in ERNNs. Implementation: Such update rules can be ef￾ficiently implemented using networks, as illus￾trated in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison between baseline RNNs and ERNN on training convergence on HAR-2 dataset. Convergence Comparison wrt Baseline: To further verify the advantage of our approach in terms of rate of convergence, we show the train￾ing behavior of different approaches in [PITH_FU…
Figure 5
Figure 5. Figure 5: Histogram of the eigenvalues of ∇φU − I for ERNN on HAR-2 dataset. 6.1 Non-Singularity of the matrix D For our ERNN parametrization to satisfy the conditions of having equillibrium points to be locally asymptotically stable, the eigen values of the matrix D = (∇φ(·)U −…
Figure 6
Figure 6. Figure 6: Linear convergence in ERNN. Empirically we verify the local convergence to a fixed point with linear rate by comparing the Euclidean distance between the approximate solutions, h (k) t , using Eq. 22 and the fixed points, ht, computed using FSOLVE from SCIPY. The learn…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 33 canonical work pages

  1. [1]

    Altun, B

    K. Altun, B. Barshan, and O. Tunçel. Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recogn., 43(10):3605–3620, Oct. 2010

    work page 2010

  2. [2]

    Anguita, A

    D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In Proceedings of the 4th International Conference on Ambient Assisted Living and Home Care, IWAAL’12, pages 216–223, Berlin, Heidelberg, 2012. Springer-Verlag

    work page 2012

  3. [3]

    Arjovsky, A

    M. Arjovsky, A. Shah, and Y . Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pages 1120–1128, 2016

    work page 2016

  4. [4]

    Bengio, N

    Y . Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8624–8628, 2013

    work page 2013

  5. [5]

    Bengio, P

    Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. Trans. Neur. Netw., 5(2):157–166, Mar. 1994

    work page 1994

  6. [6]

    J. Butcher. Numerical Methods for Ordinary Differential Equations. Wiley, 2003

    work page 2003

  7. [7]

    Campos, B

    V . Campos, B. Jou, X. Giró-i Nieto, J. Torres, and S.-F. Chang. Skip rnn: Learning to skip state updates in recurrent neural networks. arXiv preprint arXiv:1708.06834, 2017

    arXiv 2017

  8. [8]

    Chang, M

    B. Chang, M. Chen, E. Haber, and E. H. Chi. AntisymmetricRNN: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, 2019

    work page 2019

Show all 44 references
  1. [9]

    Chang, Y

    S. Chang, Y . Zhang, W. Han, M. Yu, X. Guo, W. Tan, X. Cui, M. Witbrock, M. A. Hasegawa- Johnson, and T. S. Huang. Dilated recurrent neural networks. InAdvances in Neural Information Processing Systems, pages 77–87, 2017

    2017

  2. [10]

    T. Q. Chen, Y . Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems 31. 2018

    2018

  3. [11]

    K. Cho, B. Van Merriënboer, D. Bahdanau, and Y . Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014

    2014 arXiv

  4. [12]

    Collins, J

    J. Collins, J. Sohl-Dickstein, and D. Sussillo. Capacity and Trainability in Recurrent Neural Networks. arXiv e-prints, page arXiv:1611.09913, Nov. 2016

    2016 arXiv

  5. [13]

    Cooijmans, N

    T. Cooijmans, N. Ballas, C. Laurent, Ç. Gülçehre, and A. Courville. Recurrent batch normaliza- tion. arXiv preprint arXiv:1603.09025, 2016

    2016 arXiv

  6. [14]

    Dauphin, A

    Y . Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In ICML, 2017

    2017

  7. [15]

    R. S. Dembo, S. C. Eisenstat, and T. Steihaug. Inexact newton methods. SIAM Journal on Numerical analysis, 19(2):400–408, 1982

    1982

  8. [16]

    Gehring, M

    J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y . Dauphin. Convolutional sequence to sequence learning. In ICML, 2017

    2017

  9. [17]

    C. Gong, D. He, X. Tan, T. Qin, L. Wang, and T.-Y . Liu. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pages 1334–1345, 2018

    2018

  10. [18]

    Hermans and B

    M. Hermans and B. Schrauwen. Training and analysing deep recurrent neural networks. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors,Advances in Neural Information Processing Systems 26, pages 190–198. Curran Associates, Inc., 2013

    2013

  11. [19]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735– 1780, 1997

    1997

  12. [20]

    Jaeger, M

    H. Jaeger, M. Lukosevicius, D. Popovici, and U. Siewert. Optimization and applications of echo state networks with leaky-integrator neurons. Neural networks : the official journal of the International Neural Network Society, 20:335–52, 05 2007

    2007

  13. [21]

    L. Jing, Y . Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y . LeCun, M. Tegmark, and M. Soljaˇci´c. Tunable efficient unitary neural networks (eunn) and their application to rnns. In International Conference on Machine Learning, pages 1733–1741, 2017

    2017

  14. [22]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICML, 2015

    2015

  15. [23]

    Kusupati, M

    A. Kusupati, M. Singh, K. Bhatia, A. Kumar, P. Jain, and M. Varma. Fastgrnn: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network. InAdvances in Neural Information Processing Systems, 2018

    2018

  16. [24]

    Kusupati, M

    A. Kusupati, M. Singh, K. Bhatia, A. Kumar, P. Jain, and M. Varma. Fastgrnn: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network. In NeurIPS, pages 9031–9042, 2018

    2018

  17. [25]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324, 1998

    1998

  18. [26]

    McAuley and J

    J. McAuley and J. Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, pages 165–172, New York, NY , USA, 2013. ACM. 11

    2013

  19. [27]

    Mhammedi, A

    Z. Mhammedi, A. D. Hellicar, A. Rahman, and J. Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. CoRR, abs/1612.00188, 2016

    2016 arXiv

  20. [28]

    Edge machine learning

    Microsoft. Edge machine learning. 2018

    2018

  21. [29]

    Miller and M

    J. Miller and M. Hardt. When recurrent models don’t need to be recurrent. arXiv preprint arXiv:1805.10369, 2018

    2018 arXiv

  22. [30]

    Mujika, F

    A. Mujika, F. Meier, and A. Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, pages 5915–5924, 2017

    2017

  23. [31]

    M. Y . Niu, L. Horesh, and I. Chuang. Recurrent neural networks in the eye of differential equations. arXiv preprint arXiv:1904.12933, 2019

    1904 arXiv

  24. [33]

    Pascanu, T

    R. Pascanu, T. Mikolov, and Y . Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318, 2013

    2013

  25. [34]

    Pascanu, Çaglar Gülçehre, K

    R. Pascanu, Çaglar Gülçehre, K. Cho, and Y . Bengio. How to construct deep recurrent neural networks. CoRR, abs/1312.6026, 2013

    2013 arXiv

  26. [35]

    M. Spivak. A comprehensive introduction to differential geometry. Number v. 1 in A Compre- hensive Introduction to Differential Geometry. Brandeis University, 1970

    1970

  27. [36]

    S. S. Talathi and A. Vartak. Improving performance of recurrent neural network with relu nonlinearity. arXiv preprint arXiv:1511.03771, 2015

    2015 arXiv

  28. [37]

    Tallec and Y

    C. Tallec and Y . Ollivier. Can recurrent neural networks warp time? InInternational Conference on Learning Representations, 2018

    2018

  29. [38]

    van den Oord, S

    A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. In SSW, 2016

    2016

  30. [39]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017

    2017

  31. [40]

    Vidyasagar

    M. Vidyasagar. Nonlinear Systems Analysis. Prentice-Hall Networks Series. Prentice-Hall, 1978

    1978

  32. [41]

    P. Warden. Speech Commands: A Dataset for Limited-V ocabulary Speech Recognition.arXiv e-prints, page arXiv:1804.03209, Apr. 2018

    2018 arXiv

  33. [42]

    I. Yelp. Yelp dataset challenge. 2017

    2017

  34. [43]

    Zhang, Q

    J. Zhang, Q. Lei, and I. S. Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In ICML, 2018

    2018

  35. [44]

    Zhang, Q

    J. Zhang, Q. Lei, and I. S. Dhillon. Stabilizing gradients for deep neural networks via efficient SVD parameterization. arXiv preprint arXiv:1803.09327, 2018

    2018 arXiv

  36. [45]

    J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. In ICML, pages 4189–4198. JMLR. org, 2017. 12 Supplementary 5 Local Convergence with Linear Rate Recall that we rewrite the fixed-point constraints in our ERNNs as the following ODE: h′ k...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.