{"id":"af7539c8-1225-404d-99ef-4ff5f2fb9b6c","arxiv_id":"1908.08574","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ERNNs set each hidden state to the fixed point of an implicit ODE, making the state-to-state Jacobian exactly -I (norm 1) at equilibrium, eliminating vanishing/exploding gradients in theory and giving strong empirical training speedups.","lead":"This paper introduces ERNNs, recurrent networks that define each new hidden state as the equilibrium solution of an ordinary differential equation, so that the Jacobian of the state transition has norm one and backpropagated gradients neither vanish nor explode.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The norm-preserving Jacobian identity is proven only at exact equilibrium; with K=1,3,5 fixed-point steps and ReLU, the deployed map's Jacobian is not shown to be near -I, so Theorem 1 does not cover the trained ERNN.","rationale":"I agree with the reader's conditional verdict. The strongest claim is Theorem 1, and the load-bearing assumption is exact equilibrium. The paper itself notes in the discussion of Fig. 2 that the variance in the backpropagated gradient 'arises mainly our approximation of fixed points,' which is an admission that the finite-K gap is material rather than cosmetic. The empirical speedups and accuracy are suggestive, but the theoretical justification for the central gradient claim is incomplete: no bound connects the finite-K Euler map to the exact-equilibrium Jacobian -I, and the use of ReLU further removes the smoothness assumptions. This is an addressable gap; a direct measurement of the finite-K Jacobian product norm would settle whether the deployed model actually enjoys the norm-preserving property. Since the reader already assigned CONDITIONAL, my read confirms that verdict, so no change is needed.","tokens_in":16525,"tokens_out":5075,"duration_ms":51911,"concrete_test":"For a trained HAR-2 ERNN, use automatic differentiation on the exact K-step unrolled graph (Eq. 22) to compute J_k = ∂h_k/∂h_{k-1} for K=1,3,5 at every time step, and compute the log spectral norm of ∏_{k=2}^{T} J_k over T=128 on the test set. If this product norm stays within [0.9,1.1] for the K values actually reported, the practical vanishing/exploding concern is resolved; if it decays or grows exponentially with T for K=1 (or K=3,5), then Theorem 1 does not govern the deployed model and the paper's central theoretical claim must be qualified to exact equilibrium plus a quantitative finite-K bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ERNNs overcome vanishing/exploding gradients rests on Theorem 1, which asserts ∂h_m/∂h_n = (-1)^{m-n}I for the constrained model in Eq. 7. That identity follows from Lemma 1, an exact-equilibrium statement: it requires h_k to be the fixed point of F. The practical model, however, is the K-step unrolled Euler recursion in Eq. 22, with K=1,3,5 in the experiments. For K=1, h_k = h_{k-1} + η[f(h_{k-1}+h_{k-1}, x_k) - (h_{k-1}+h_{k-1})], so ∂h_k/∂h_{k-1} = (1-2η)I + 2η∇f(·)U. This equals -I only under special conditions on ∇f and η; for ReLU, on dead units it is (1-2η)I. With η initialized at 0.01, the product over 128 steps would decay like 0.98^128 ≈ 0.075 unless other terms compensate. For K=3 and K=5, the paper offers no bound on how close the K-step Jacobian is to -I, and the convergence result (Theorem 3) is about forward fixed-point iterations to the state value, not about the Jacobian of the unrolled map after a finite number of steps. Figure 2 shows one empirical gradient curve, but that is a single indirect check, not a proof or a systematic bound. Thus the theoretical guarantee does not apply to the model that is actually trained; the claim that ERNNs 'overcome' vanishing/exploding gradients is therefore unsupported at the point where it matters, and the paper offers no K-dependent error analysis to close this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Equilibriated Recurrent Neural Networks (ERNNs), which update the hidden state by solving a fixed-point equation so that the state evolves on the equilibrium manifold of an ODE. The central theoretical claim is Theorem 1: at exact equilibrium, ∂h_m/∂h_n = (-1)^{m-n} I, so backpropagated transition Jacobians have unit norm and vanishing/exploding gradients are circumvented. In practice, the equilibrium is computed by a K-step damped Euler recursion (Eq. 22) with a low-rank parameterization (Eq. 10), and experiments on long-term-dependence benchmarks report accuracy gains over RNN, LSTM, GRU, FastRNN, and FastGRNN baselines, with 3-10x training speedups and smaller models. The main unresolved issue is that the theorem's assumptions (exact equilibrium, smooth non-vanishing activation, non-singular Jacobian) are not satisfied by the deployed finite-K ReLU model, and the paper supplies no K-dependent gradient error bound.","tokens_in":16952,"tokens_out":7842,"duration_ms":72656,"significance":"If the central claim were fully supported, the contribution would be significant: a simple, parameter-light recurrent cell with a provable norm-preserving transition Jacobian would address a long-standing training difficulty without unitary constraints or gating. The paper also demonstrates strong empirical results, including the best accuracy among the compared single-cell models on several datasets (e.g., 96.30% vs 95.48% on HAR-2, 94.71% vs 93.18% on Google-12, 54.50% vs 48.63% on Noisy-CIFAR), with consistently smaller models and markedly lower training time. The supplementary contributes useful empirical checks of linear fixed-point convergence and eigenvalue negativity. However, the theoretical guarantee as stated applies to the exact equilibrium map, not to the finite-K unrolled network that is actually trained; absent an error analysis, the paper's headline claim that ERNNs 'overcome' vanishing and exploding gradients is not established. The empirical results are interesting enough to warrant a major revision rather than rejection.","major_comments":[{"comment":"Theorem 1 and Lemma 1 establish ∂heq/∂hk−1 = −I only at an exact equilibrium heq satisfying Eq. (8), but the model actually trained is the K-step unrolled Euler recursion in Eq. (22), with K=1,3,5 in Table 3 (and K=1,2,3 for the large-scale sets). For K=1 the transition Jacobian is ∂hk/∂hk−1 = (1−2η)I + 2η∇φ(·)U under the parametrization used in Eq. (10), which equals −I only for special values of η and ∇φU and is not shown to be close to −I for the trained parameters. For K=3 and K=5 no bound is given on the distance between the finite-K Jacobian and (−1)^K I, and Theorem 3, stated in the supplementary, concerns convergence of the fixed-point iterates in state value, not the Jacobian of the unrolled map. Consequently the paper's central claim that ERNNs 'overcome' vanishing and exploding gradients is not supported by the stated theorems for the architecture used in the experiments; a K-dependent error analysis or a direct Jacobian-norm verification across K and datasets is needed.","section":"Sec. 2.1, Eq. (22); Sec. 3, Table 3"},{"comment":"Lemma 1 assumes a smooth activation with non-vanishing ∇φ and a non-singular Jacobian ∇f − γI, while the experiments use ReLU throughout (Sec. 3), which is non-smooth at zero and has ∇φ = 0 on the negative orthant. The supplementary's Fig. 5 checks negativity of eigenvalues of ∇φU − I on HAR-2 after training, but this is a post-hoc check on one dataset and does not establish the non-singularity or non-vanishing conditions during training on all datasets. The stability condition of Lemma 2 is also not enforced during optimization, so the training trajectory may leave the regime in which the equilibrium-manifold analysis applies.","section":"Sec. 2, Lemma 1; Sec. 3, hyper-parameters"},{"comment":"The identity ∂heq/∂hk−1 = −I is a direct algebraic consequence of the equilibrium constraint itself: differentiating f(h_eq+h_{k−1}) − γ(h_eq+h_{k−1}) = 0 with respect to h_{k−1} gives [∇f−γI](∂heq/∂hk−1 + I) = 0, so the conclusion follows as soon as the Jacobian is invertible. This means Theorem 1 is a restatement of the architecture's defining constraint rather than an independent mechanism that can be expected to transfer automatically to the finite-K approximation; the substantive claim that needs support is the gradient behavior of the unrolled recursion used in practice, not of the exact equilibrium map.","section":"Sec. 2, Eqs. (7)-(9)"},{"comment":"The only direct empirical evidence for the norm-preserving gradient claim is Fig. 2, which plots gradient magnitudes for one dataset (HAR-2) and does not report the actual transition-Jacobian norm, the deviation from (−1)^{T−1}I, or confidence intervals; it is a single trajectory-level curve, not a systematic verification across the datasets and K values used in Table 3. A quantitative check of ||∂h_T/∂h_1|| over training, or a histogram of eigenvalues of the finite-K transition Jacobian, would be needed to support the claim that practical ERNNs preserve gradients.","section":"Fig. 2; Sec. 3, Evaluation Criteria"}],"minor_comments":[{"comment":"Please correct the repeated misspelling of 'equilibrium' (for example, 'equillibrium' in the caption of Table 3).","section":"Table 3 caption"},{"comment":"Eq. (13) is cited before it is defined in the main text; the ODE in Eq. (6) should be numbered consistently so that the proof of Lemma 1 and Theorem 2 refer to an equation already introduced.","section":"Sec. 2, Lemma 1 proof; Sec. 2.1, Theorem 2"},{"comment":"The text says results are shown for K=1,3,5 for all datasets except the large-scale ones, but Table 3 uses K=2 for Pixel-MNIST and Yelp-5; please reconcile this inconsistency.","section":"Sec. 3, K values"},{"comment":"The sentence describing the hardware says 'Intel Xeon 2.60 GHz GPU with 20 cores'; this should presumably read 'CPU'.","section":"Sec. 3, experimental setup"},{"comment":"The claim of 'similar prediction cost to vanilla RNNs' is not consistently supported by Table 3: for example, ERNN(K=5) reports a test time of 0.03 ms versus 0.01 ms for FastRNN on HAR-2 and 0.05 ms versus 0.01 ms on Google-12; the claim should be qualified.","section":"Sec. 3, evaluation criteria"},{"comment":"Fig. 2 would be stronger with multiple seeds and with a plot of the actual Jacobian norm ||∂h_T/∂h_1|| rather than a loss-gradient magnitude, which also depends on the loss and the intervening parameter updates.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest contribution is empirical: ERNN appears to train fast and accurately across a range of long-term-dependence benchmarks. The theoretical contribution, however, is much weaker than the title and abstract suggest, because the norm-preserving identity is essentially a consequence of the equilibrium constraint rather than a property that is shown to hold for the finite-K unrolled model. A revised version that either proves a finite-K Jacobian perturbation bound or substantially softens the theoretical claims while repositioning Theorem 1 as motivation would be a much stronger paper. I would not reject on these grounds, given the potential practical significance of the empirical results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real idea, not a repackaging. The ERNN cell defines the hidden state as the fixed point of an implicit ODE, h_k = h_{k-1} + eta[f(h_k + h_{k-1}, x_k) - (h_k + h_{k-1})], and solves it with a few Euler steps. The new observation is that at an exact fixed point, the transition Jacobian is forced to be -I by the constraint itself, so the BPTT product has norm one. Lemma 1 and Theorem 1 are correct under their stated assumptions. That alone is worth a serious look.\n\nWhat the paper does well: the empirical work is broad (10 datasets), the model is genuinely smaller and often trains 3-6x faster than FastRNN or GRU with competitive accuracy, and the authors include useful ablations (K, sigmoid vs ReLU, eigenvalue histogram). They also make the equilibrium idea concrete with a low-rank parameterization that keeps the model small. I believe the architecture is new relative to AntisymmetricRNN and FastRNN, and the citation pattern looks fair.\n\nThe soft spot is the one the stress test found: Theorem 1 is stated for the exact equilibrium model, but training uses K=1,3,5 unrolled fixed-point steps with ReLU. For K=1, the transition Jacobian is (1-2eta)I + 2eta nabla-phi U, which is not -I in general; for ReLU dead units it is (1-2eta)I, and with eta initialized at 0.01 the product over 128 steps decays like 0.98^128 ~ 0.075. The paper gives no K-dependent error bound and no argument that the finite-K Jacobian stays near -I. Theorem 3 is about forward convergence of the state, not the backward Jacobian of the unrolled graph. Figure 2 is a single empirical gradient curve, not a systematic check. So the central claim that ERNNs \"overcome\" vanishing/exploding gradients is not actually proven for the model that is trained. This is fixable: use implicit differentiation (which would make the training gradient equal to the IFT gradient, recovering the -I identity) or analyze the finite-K Jacobian. The authors should also correct the \"similar prediction cost\" claim (the tables show test time is the same order but not identical to vanilla RNNs) and report variance across seeds.\n\nThe \"circularity\" of Lemma 1 is not a vice; it is a clean observation about what the constraint forces.\n\nBottom line: this paper deserves a serious referee. I would accept it conditional on closing the theory-experiment gap, either by switching to implicit differentiation or by providing finite-K Jacobian bounds, and on tightening the overclaims. The core idea is sound and likely to be cited.","headline":"A genuinely new RNN construction with a clean norm-preserving identity at exact equilibrium, but the finite-K unrolled model used in training is not covered by the theorem; worth serious review.","tokens_in":17508,"tokens_out":3486,"would_cite":true,"duration_ms":34378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes ERNNs, recurrent networks whose hidden states are updated to the equilibrium point of an implicit ODE, and proves that this makes the product of Jacobians across time equal to $(-1)^{m-n}I$, so backpropagated gradient…","keywords":["Equilibriated Recurrent Neural Networks","vanishing gradients","exploding gradients","equilibrium manifold","implicit ODE","fixed-point recursion","norm-preserving Jacobian","long-term dependencies"],"falsifier":"Take a trained ERNN with ReLU and a fixed number of solver steps and use automatic differentiation through the actual unrolled graph to compute the Jacobian product $\\partial h_T/\\partial h_1$ on sequences that drive many hidden units into saturation; if its singular values deviate substantially from 1 on any such sequence, the identity transition mapping does not hold in the trained model, and the paper's norm-preservation claim rests on an unquantified approximation.","tokens_in":16313,"feed_emoji":"🧠","tokens_out":10861,"duration_ms":89468,"temperature":0.7,"pith_summary":"ERNNs (Equilibriated Recurrent Neural Networks) are a family of recurrent networks that, at each time step, solve for the equilibrium of an ordinary differential equation instead of taking one explicit vector-field step. The paper's central claim is that on this equilibrium manifold the Jacobian of each state transition is exactly $-I$, so the product of Jacobians over any time span is $(-1)^{m-n}I$ and backpropagated gradients have norm exactly $1$. This would remove exponential vanishing and exploding gradients at their source, without gates, unitary constraints, or damping factors. The paper reports that on long-term-dependence benchmarks the resulting models match or exceed gated and unitary baselines in accuracy while training 3–10x faster and using 1.5–3x fewer parameters. The argument requires the state to be an exact equilibrium and the activation gradient to be nonzero, while experiments use ReLU and only $K\\le 5$ fixed-point iterations, so the identity holds only to the extent that approximation error is negligible.","feed_headline":"Pinning RNN states to equilibrium manifold keeps gradients at norm 1","feed_subtitle":"Solving an implicit ODE at each step makes every time-step Jacobian a signed identity, giving 3-10x faster training.","key_machinery":"The load-bearing object is the equilibrium manifold $\\mathcal{M}(h_{k-1}, x_k) = \\{h : f(h + h_{k-1}, x_k;\\alpha) - \\gamma(h + h_{k-1}) = 0\\}$, combined with the implicit-function-theorem derivative identity. Differentiating the equilibrium condition with respect to $h_{k-1}$ gives $[\\nabla f - \\gamma I](\\partial h_{\\mathrm{eq}}/\\partial h_{k-1} + I) = 0$, so, under the nonsingularity assumption, $\\partial h_{\\mathrm{eq}}/\\partial h_{k-1} = -I$. The practical solver is the fixed-point recursion $h_k^{(i+1)} = h_k^{(i)} + \\eta_k^{(i)}[f(h_k^{(i)} + h_{k-1}, x_k) - (h_k^{(i)} + h_{k-1})]$, and the paper's Theorem 2 ties its local linear convergence to an inexact-Newton condition. This identity carries the whole argument: it converts a product of $m-n$ Jacobians into a signed identity matrix.","core_discovery":"The central discovery is that if the hidden state of a recurrent network is defined implicitly as the equilibrium $h_k^*$ of $h' = f(h + h_{k-1}, x_k) - \\gamma(h + h_{k-1})$, then the map from the previous state $h_{k-1}$ to the new equilibrium has Jacobian $\\partial h_k/\\partial h_{k-1} = -I$ on the equilibrium manifold, provided $\\nabla f - \\gamma I$ is nonsingular and the activation gradient does not vanish. Iterating this identity gives $\\partial h_m/\\partial h_n = (-1)^{m-n}I$, hence the norm of backpropagated gradients is exactly 1 for any distance in time. The paper argues that this identity is what makes training stable: error signals neither decay nor explode, and long-term dependencies remain reachable. It further proves local linear convergence of the Euler fixed-point recursion used to find the equilibrium, shows equilibria are locally asymptotically stable when eigenvalues of $\\nabla\\varphi\\,U - \\gamma I$ are negative, and reports experiments in which ERNNs match or exceed gated and unitary baselines on sequence-classification and language-modeling benchmarks while training several times faster.","pith_inferences":["The norm-preservation theorem applies only at exact equilibrium; for truncated unrolling, the relevant quantity is how fast $\\|h_k^{(i)} - h_k^*\\|$ shrinks with $i$, so an adaptive step count or residual-based stopping rule could make the guarantee practical and is directly testable.","The same implicit-equilibrium construction could be applied to depth in feed-forward or attention networks, not just time, potentially giving identity Jacobians through arbitrarily deep layers.","Because Lemma 1 requires $\\nabla\\varphi$ to be nonzero, ReLU in its saturated region violates the proof's conditions; testing ERNN with activations whose gradients are bounded away from zero (or with a smoothed ReLU) would separate the norm-preservation mechanism from the empirical gains.","The input-variation analysis suggests that slowly varying inputs confine equilibrium states to a low-dimensional subspace related to the input projection; if true, this offers a principled explanation for the observed small model sizes and a route to further compression."],"forward_implications":["Backpropagated gradients over any time span have norm exactly 1, so long-term dependencies become reachable without gating, unitary constraints, or specially chosen learning rates.","Training converges substantially faster: the paper reports 3–10x speedups and 1.5–3x model-size reductions relative to gated and unitary baselines.","Because the identity comes from the equilibrium condition rather than the specific activation, the construction extends to deep transition blocks and arbitrary smooth transition functions.","Unrolling the fixed-point recursion for K steps gives a feed-forward training graph; larger K better approximates the equilibrium and empirically improves both convergence rate and accuracy."],"supporting_citations":[{"why":"Establishes the vanishing-gradient failure for long-term dependencies that ERNN targets.","marker":"[5]"},{"why":"Provides the implicit Euler and fixed-point recursion framework used to reach equilibrium.","marker":"[6]"},{"why":"Defines the AntisymmetricRNN damping analysis and the stability proposition that Corollary 1 extends.","marker":"[8]"},{"why":"Supplies the inexact Newton convergence theorem used to prove local linear convergence.","marker":"[15]"},{"why":"Provides the experimental baselines, dataset preprocessing, and reported comparison numbers.","marker":"[23]"},{"why":"Formalizes the Jacobian-product structure behind vanishing and exploding gradients.","marker":"[33]"},{"why":"Underlies Lemma 1 through the implicit function theorem on the equilibrium manifold.","marker":"[35]"},{"why":"Supplies the asymptotic stability criterion used in Lemma 2.","marker":"[40]"}],"fun_headline_variants":["RNNs on equilibrium manifold fix vanishing gradients","Equilibrium RNNs keep backprop gradients at norm 1","ERNNs: stable gradients via implicit ODE equilibria","Gradient norm 1: RNNs evolved on equilibrium manifold","Solving implicit ODE keeps RNN gradients from exploding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hidden state at every time step is exactly (or close enough to) an equilibrium point of the implicit ODE, because the identity $\\partial h_k/\\partial h_{k-1} = -I$ is derived by differentiating the equilibrium condition; if the fixed-point recursion is cut off before convergence, or the activation gradient vanishes, the norm-preservation proof no longer applies.","fun_headline_variants_meta":{"raw":{"variants":["RNNs on equilibrium manifold fix vanishing gradients","Equilibrium RNNs keep backprop gradients at norm 1","ERNNs: stable gradients via implicit ODE equilibria","Gradient norm 1: RNNs evolved on equilibrium manifold","Solving implicit ODE keeps RNN gradients from exploding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000463,"raw_usage":{"total_tokens":2352,"prompt_tokens":1024,"completion_tokens":1328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1246}},"tokens_in":640,"tokens_out":1328,"duration_ms":8084,"temperature":1.0,"reasoning_tokens":1246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:37:17.033145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained ERNN with ReLU and a fixed number of solver steps and use automatic differentiation through the actual unrolled graph to compute the Jacobian product $\\partial h_T/\\partial h_1$ on sequences that drive many hidden units into saturation; if its singular values deviate substantially from 1 on any such sequence, the identity transition mapping does not hold in the trained model, and the paper's norm-preservation claim rests on an unquantified approximation.","supporting_citations":[{"cited_title":"Bengio, P","cited_arxiv_id":null,"evidence_quote":"Establishes the vanishing-gradient failure for long-term dependencies that ERNN targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the implicit Euler and fixed-point recursion framework used to reach equilibrium."},{"cited_title":"Chang, M","cited_arxiv_id":null,"evidence_quote":"Defines the AntisymmetricRNN damping analysis and the stability proposition that Corollary 1 extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inexact Newton convergence theorem used to prove local linear convergence."},{"cited_title":"Kusupati, M","cited_arxiv_id":null,"evidence_quote":"Provides the experimental baselines, dataset preprocessing, and reported comparison numbers."},{"cited_title":"Pascanu, T","cited_arxiv_id":null,"evidence_quote":"Formalizes the Jacobian-product structure behind vanishing and exploding gradients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underlies Lemma 1 through the implicit function theorem on the equilibrium manifold."},{"cited_title":"Vidyasagar","cited_arxiv_id":null,"evidence_quote":"Supplies the asymptotic stability criterion used in Lemma 2."}],"review_version":1}