REVIEW 4 cited by
AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.
Forward citations
Cited by 4 Pith papers
-
Universal Approximation Theorems for Dynamical Systems with Infinite-Time Horizon Guarantees
Neural ODEs can approximate Morse-Smale and continuous-attractor dynamical systems over infinite time in an ε-δ sense, provided limit-cycle periods are matched exactly.
-
Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum
A Lyapunov-spectrum-based distance to the dense network lets hyperparameter search for pruned RNNs stop early and select models that beat both loss-based baselines and the dense originals.
-
Temporal horizons in forecasting: a performance-learnability trade-off
Longer training horizons improve forecast quality but worsen learnability, with loss-landscape roughness growing exponentially for chaotic dynamics and linearly for limit cycles.
-
Temporal Chunking Enhances Recognition of Implicit Sequential Patterns
A context-tagged chunked RNN reaches optimal accuracy on a synthetic community-sequence task with a BPTT window of 1, where a naive RNN needs a window of 7.
Discussion (0). Sign in to comment.