Pith. sign in

REVIEW 3 major objections 6 minor 35 references

Approximating Stacked and Bidirectional Recurrent Architectures with the Delayed Recurrent Neural Network

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A delayed single-layer RNN can exactly reproduce a stacked RNN by converting layers into time.

desk verdict The block-bidiagonal equivalence is genuinely useful, but Lemma 1's pseudoinverse skips a rank condition, so the 'any stacked RNN' claim is overbroad. read the letter →

arxiv 1909.00021 v2 pith:LXPIAFHJ submitted 2019-08-30 cs.LG cs.CLcs.NEstat.ML

Javier S. Turek , Shailee Jain , Vy Vo , Mihai Capota , Alexander G. Huth , Theodore L. Willke This is my paper · ORCID
classification cs.LGcs.CLcs.NEstat.ML
keywords delayedrecurrentneuralnetworkstackednetworksbidirectionalacausalprocessingdepthintimeLSTMsequencemodelingarchitectureequivalence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that one design choice—inserting a delay of $d$ timesteps between an input and its output in a single-layer recurrent network—can absorb two things that usually require architectural changes: depth and acausality. The exact result is that a $k$-layer stacked RNN is equivalent to a single-layer delayed RNN with delay $d=k-1$ and a block-bidiagonal weight matrix: the same outputs appear, shifted by $k-1$ timesteps. The approximate result is that the same delayed network, trained without those weight constraints, can match or beat a bidirectional LSTM on tasks requiring future context, with a single forward pass. If correct, this unifies layer depth and bidirectional access as one tunable parameter and gives a cheaper way to get both.

What carries the argument

The central object is the delayed RNN (d-RNN), an ordinary single-layer recurrent network whose output is withheld by $d$ timesteps. The load-bearing construction is the block-bidiagonal recurrent weight matrix $\hat{W}_h$: each layer of a stacked RNN becomes a block of units whose recurrent weights sit on the diagonal, and the connection from layer $i-1$ to layer $i$ becomes the subdiagonal block. This makes the hidden state act as a buffer, moving information one block forward per timestep, so the identity $\hat{h}^{(i)}_{t+i-1}=h^{(i)}_t$ carries the induction proof. For bidirectional approximation, the mechanism is simply that by the time the output for time $t$ is read out at $\hat{y}_{t+d}$, the network has consumed $d$ future inputs, giving it partial acausality with no second backward pass.

What would settle it

Construct a two-layer stacked RNN with a rank-deficient $W_h^{(2)}$ and an initial state $h_0^{(2)}$ such that $r(h_0^{(2)}) - W_x^{(2)} h_0^{(1)} - b^{(2)}$ lies outside the column space of $W_h^{(2)}$. If no $\hat{h}_0$ satisfies the recursive initialization from Lemma 1, that instance is a counterexample to the claim that every stacked RNN can be represented, even though the induction in Theorem 1 remains valid for networks whose initialization is constructible.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 1: given a stacked RNN with $k$ layers and a single-layer delayed RNN (d-RNN) with delay $d=k-1$ whose recurrent weight matrix is block-bidiagonal, holding the stacked layer weights on the diagonal and subdiagonal, the d-RNN produces exactly the stacked network's output sequence delayed by $k-1$ timesteps, $\hat{y}_{t+k-1}=y_t$ for every $t$. The proof by induction also shows $\hat{h}^{(i)}_{t+i-1}=h^{(i)}_t$, meaning block $i$ of the delayed hidden state at time $t+i-1$ equals layer $i$'s hidden state at time $t$. The same construction extends to LSTMs via matching hidden and cell states. The paper then shows empirically that an unconstrained delayed LSTM has partial acausality—it sees inputs $x_{t+1},\ldots,x_{t+d}$ before producing $\hat{y}_{t+d}$—and that this lets it approximate bidirectional networks on sequence reversal, nonlinear filtering, masked character language modeling, and part-of-speech tagging.

Load-bearing premise

The proof that a delayed network can be initialized to match a stacked network assumes a certain linear equation can always be solved for the initial hidden state, but such equations do not have a solution for every choice of recurrent weights and initial states.

Editorial extensions

If this is right

  • A $k$-layer stacked RNN can be flattened into a single-layer d-RNN with delay $k-1$ with exactly the same outputs and hidden states, and the weight definitions make the two interchangeable.
  • Relaxing the sparse constraints yields a d-RNN that sees $d$ future inputs, giving partial acausality in a single forward pass without the synchronization of a bidirectional network.
  • In the synthetic filtering experiment, delayed LSTMs solve tasks whose acausal span is at most the delay, and outperform bidirectional LSTMs on highly nonlinear functions.
  • On masked character-level language modeling, delayed LSTMs with intermediate delays come within 5% of the bits-per-character of stacked bidirectional LSTMs while running about four times faster.
  • Adding weights to the empty regions of the constrained matrix corresponds to known stacked architectures such as skip connections and gated feedback networks, making the d-RNN a common superclass of these topologies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the exact initialization in Lemma 1 can fail, the practical route is to train the delayed network directly with the delay built in, rather than translating a pretrained stack.
  • The delay parameter is a continuous knob between causal processing and full acausal access, suggesting a design rule: choose the delay to cover the longest future dependency the task needs, and let the single layer provide the depth.
  • A testable extension is to read the learned off-diagonal blocks of a d-RNN's weight matrix as a diagnostic of which cross-time, cross-layer connections a task actually needs, since each block has a known topological meaning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the delayed recurrent neural network (d-RNN), a single-layer RNN whose output is delayed by d timesteps relative to its input. The central theoretical claim is that a k-layer stacked RNN (Eqs. (3)-(5)) can be exactly simulated by a d-RNN with delay d=k-1 when the d-RNN's weights are constrained to the block-bidiagonal form of Eqs. (6)-(8) and its initial hidden state satisfies h_hat_{i-1}^(i) = h_0^(i) for every block i. Theorem 1 states this equivalence and is proved by induction in Supplement A. Lemma 1 in the main text asserts that such an initialization can always be computed using a pseudoinverse, but the proof in Supplement B does not address consistency of the linear solves. The paper also argues that the delay provides the d-RNN with partial acausal access similar to a Bi-RNN, and reports experiments on sequence reversal, filtered sine regression, masked character-level language modeling, and multilingual part-of-speech tagging, comparing d-LSTMs with unidirectional and bidirectional LSTMs.

Significance. If Theorem 1 held as stated, the result would be a clean architectural equivalence: depth in layers is exchanged for depth in time in a single recurrent layer with the same number of units, and the constrained d-RNN is exactly interchangeable with the stacked RNN. The empirical contribution is solid in its own right: all tables report means and standard deviations over five repetitions, baselines are parameter-matched, and the synthetic tasks provide falsifiable predictions (e.g., the d-LSTM's performance cliff at acausality a=d in Figure 5) that the experiments confirm. The GPU runtime measurements give a practical reason to consider d-LSTMs. However, the universality of the stacking equivalence is not fully proven, because Lemma 1's pseudoinverse construction requires an unstated rank/range condition; the main theoretical claim therefore needs repair before the paper can be accepted.

major comments (3)
  1. [Section 3.2, Lemma 1 and Supplement B (Eqs. B.12-B.13)] Lemma 1 asserts that the initialization h_hat_0 with h_hat_{i-1}^(i) = h_0^(i) always exists, but the proof reduces each step to a linear solve W_h^(i) x = r(h_0^(i)) - W_x^(i)(.) - b_h^(i) and applies the Moore-Penrose pseudoinverse. A pseudoinverse solve is consistent only when the right-hand side lies in the column space of W_h^(i); no rank or range condition is stated, and the stacked RNN's weights are arbitrary. The issue is concrete: for k=2, W_h^(2)=[[1,0],[0,0]], W_x^(2)=0, b_h^(2)=0, f=tanh, and h_0^(2)=(0.5,0.2), the second coordinate of h_hat_1^(2)=tanh(W_h^(2) h_hat_0^(2)) is 0 for every h_hat_0^(2), so no initialization can satisfy h_hat_1^(2)=h_0^(2). Consequently Theorem 1's 'any stacked RNN' claim is not established. Please add the needed rank/consistency assumption to Lemma 1 and Theorem 1, or replace the universality claim with the correct conditional statement, and adjust the abstract and conclusions accordingly. The same caveat applies to Theorem 2, which assumes the analogous initialization for LSTM hidden and cell states without an existence proof.
  2. [Supplement B, Eq. (B.13)] The recursive construction in the proof of Lemma 1, Supplement B, Eq. (B.13), does not match the block dynamics of Eqs. (6)-(8). The dynamics give h_hat_t^(i) = f(W_x^(i) h_hat_{t-1}^(i-1) + W_h^(i) h_hat_{t-1}^(i) + b^(i)), so when solving for h_hat_{i-j}^(i) the lower-block state must be evaluated at time i-j-1, not i-j+1, and the final step requires h_hat_0^(i-1), not h_hat_1^(i-1). As printed, the backward recursion is not derivable from the recurrence and the lemma proof is incomplete; please correct the indices and specify that the needed lower-block states at intermediate times are obtained by forward simulation of blocks 1 through i-1 once their initial states are fixed.
  3. [Supplement C, Theorem 2] Theorem 2 in Supplement C is presented as the LSTM extension of the equivalence, but it simply assumes initial states with h_hat_{i-1}^(i) = h_0^(i) and c_hat_{i-1}^(i) = c_0^(i); no analogue of Lemma 1 is proved for the cell-state initialization. Since the main text advertises that the proof can be extended to more complex recurrent cells, the authors should either supply an existence lemma for the LSTM initialization (with its consistency conditions) or explicitly state the initialization existence as an assumption of Theorem 2.
minor comments (6)
  1. [Title and Abstract] The title and abstract contain a spacing error: 'Archit ectures' should read 'Architectures'.
  2. [Section 4.3] The sentence 'The results are summarized in Table 3' should refer to Table 1 in the main text; Table 3 is an appendix table.
  3. [Section 3.2] The sentence after Eq. (8) claiming that 'any stacked RNN can be flattened' overstates the result until Lemma 1 is repaired; please qualify it with the initialization condition.
  4. [Lemma 1] The statement that all commonly used activation functions are surjective should specify the codomain (e.g., ReLU maps onto [0, inf), tanh onto (-1,1)) and note that the right-inverse r(d)=d for ReLU is defined on that domain.
  5. [Section 4] The code is available at an anonymous URL; a persistent repository identifier should be provided for a journal submission.
  6. [Figure 3] Figure 3 is dense and would benefit from a legend or expanded caption explaining the piecewise-linear curves for the number of non-linearities.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the stacked-to-delayed equivalence is a direct weight-construction proof and the empirical comparisons use held-out evaluation.

full rationale

Theorem 1 establishes the equivalence by explicitly constructing the d-RNN weight matrices from the stacked RNN parameters (Equations 6-8) and then proving by induction that the block-hidden states coincide with a shift of one timestep per layer. The delay d = k-1 is chosen from the number of stacked layers, not fitted from the outputs; the initialization condition h_hat_{i-1}^{(i)} = h_0^{(i)} is a stated premise of the theorem, not a prediction extracted from data. Lemma 1's pseudoinverse construction is an auxiliary existence argument for that initialization; its unstated rank/range condition is a correctness gap, not a circular reuse of the conclusion. The empirical sections compare independently trained d-LSTM, LSTM, and Bi-LSTM models on held-out validation and test sets, with the sequence-reversal performance bound derived analytically rather than fitted. The paper contains no load-bearing self-citation chain and no fitted parameter is renamed as a prediction. The central claim therefore has independent mathematical content and is not circular.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The mathematical equivalence relies on the standard RNN definition, a surjective activation assumption, and the unstated solvability condition for initial-state construction. The empirical Bi-RNN approximation introduces the hand-chosen delay d as a free parameter. No new physical entities are introduced.

free parameters (1)
  • delay d = 1, 5, 8, 10, or 19 depending on experiment
    The delay is chosen by hand for each task and controls how much future context the network can access. It is not derived from data, but the approximation claim for bidirectional networks depends on selecting d large enough for the task's acausality.
assumptions (3)
  • domain assumption Activation functions are surjective and have a right-inverse (e.g., tanh onto (-1,1), ReLU onto [0,∞)).
    Used in Lemma 1 to invert the activation when constructing the d-RNN initial state; standard for common activations but not universal.
  • ad hoc to paper The stacked RNN's recurrent weight matrices W_h^{(i)} satisfy a rank and solvability condition so the pseudoinverse construction in Lemma 1 yields valid initial states.
    The paper does not state or prove this condition; it is implicitly required by Eqs. B.12 and B.13 in the supplementary material.
  • domain assumption Padding input sequences with null (zero) vectors does not change the target task.
    The d-RNN input needs T+d elements; the paper pads with zero vectors, assuming this is information-neutral for the tasks considered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Approximating Stacked and Bidirectional Recurrent Architectures with the Delayed Recurrent Neural Network." pith.science (2026). https://pith.science/paper/LXPIAFHJ

@misc{pith2026190900021,
  author       = {Pith},
  title        = {Pith review of: Approximating Stacked and Bidirectional Recurrent Architectures with the Delayed Recurrent Neural Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LXPIAFHJ}},
  note         = {Machine review of arXiv:1909.00021}
}
read the original abstract

Recent work has shown that topological enhancements to recurrent neural networks (RNNs) can increase their expressiveness and representational capacity. Two popular enhancements are stacked RNNs, which increases the capacity for learning non-linear functions, and bidirectional processing, which exploits acausal information in a sequence. In this work, we explore the delayed-RNN, which is a single-layer RNN that has a delay between the input and output. We prove that a weight-constrained version of the delayed-RNN is equivalent to a stacked-RNN. We also show that the delay gives rise to partial acausality, much like bidirectional networks. Synthetic experiments confirm that the delayed-RNN can mimic bidirectional networks, solving some acausal tasks similarly, and outperforming them in others. Moreover, we show similar performance to bidirectional networks in a real-world natural language processing task. These results suggest that delayed-RNNs can approximate topologies including stacked RNNs, bidirectional RNNs, and stacked bidirectional RNNs - but with equivalent or faster runtimes for the delayed-RNNs.

Figures

Figures reproduced from arXiv: 1909.00021 by the authors.

Figure 1
Figure 1. A delayed-recurrent neural network (d-RNN) process￾ing a sequence of T elements. The output is delayed by d = 2 timesteps. The first output element is in ˆy3 and the last in ˆyT+d. The input sequence has d additional elements, such as ‘[NULL]’ symbols. During training, the outputs are compared with the T elements of the labeled sequence {zj}j . as a means to increase depth within a single-layer RNN. However, single-… view at source ↗
Figure 2
Figure 2. A stacked RNN is equivalent to a single-layer d-RNN under the given sparse weight constraints. The d-RNN produces the same representations as the stacked network. (a) Stacked RNN with k = 4 layers where connections show the different weight parameters. (b) Weights of the d-RNN that are equivalent to connections in the stacked RNN. 1 d+1 Number of non-linearities Timestep relative to current input (Δt) max(1-Δt, 1+Δt… view at source ↗
Figure 3
Figure 3. Number of non-linearities that can be applied to past and future sequence elements with respect to current input (∆t=0). The d-RNN only sees d steps into the future. the first d acausal inputs with a decreasing number of non￾linearities until the number reaches zero at t = t0+d+1. In order for a d-RNN to have at least as many non-linearities as a Bi-RNN for every element in a sequence, it would need a delay that is … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: shows accuracy on this task as a function of the applied delay. The LSTM does not use acausal informa￾tion and is unable to reverse more than half of the input sequence. Conversely, the Bi-LSTM has full access to ev￾ery element in the sequence, and can perfectly solve …
Figure 5
Figure 5. Figure 5: Error maps for the sine function experiment with different degrees of non-linearity (horizontal axis) and amounts of acausality of the filter (vertical axis). Tested architectures: (a) LSTM, (b) Bi-LSTM, (c) d-LSTM with delay=5, and (d) d-LSTM with delay=10. Dark blue …
Figure 6
Figure 6. Figure 6: (a) Weights of the single-layer and weight constrained d-RNN that are equivalent to connections in the stacked RNN from [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Error maps presented in [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

  1. [1]

    P olyglot: Distributed word representations for multilingual NLP

    Al-Rfou ' , R., Perozzi, B., and Skiena, S. P olyglot: Distributed word representations for multilingual NLP . In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp.\ 183--192, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W13-3520

    work page 2013

  2. [2]

    Deep speech 2: End-to-end speech recognition in english and mandarin

    Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning, pp.\ 173--182, 2016

    work page 2016

  3. [3]

    O., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., Sengupta, S., and Shoeybi, M

    Arik, S. O., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., Sengupta, S., and Shoeybi, M. Deep voice: Real-time neural text-to-speech. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp.\ 195--204. JMLR.org, 2017. URL http://dl.acm.org/citation.cfm?id=...

    arXiv 2017

  4. [4]

    Ballesteros, M., Dyer, C., and Smith, N. A. Improved transition-based parsing by modeling characters instead of words with LSTM s. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.\ 349--359, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi:10.18653/v1/D15-1041. URL https://www.a...

    work page doi:10.18653/v1/d15-1041 2015

  5. [5]

    Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis

    Baziotis, C., Pelekis, N., and Doulkeridis, C. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pp.\ 747--754, 2017

    work page 2017

  6. [6]

    Learning deep architectures for ai

    Bengio, Y. Learning deep architectures for ai. Foundations and Trends in Machine Learning, 2 0 (1): 0 1--127, 2009. ISSN 1935-8237. doi:10.1561/2200000006. URL http://dx.doi.org/10.1561/2200000006

    doi:10.1561/2200000006 2009

  7. [7]

    Learning long-term dependencies with gradient descent is difficult

    Bengio , Y., Simard , P., and Frasconi , P. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 0 (2): 0 157--166, March 1994. ISSN 1045-9227. doi:10.1109/72.279181

    doi:10.1109/72.279181 1994

  8. [8]

    Scaling learning algorithms towards ai

    Bengio, Y., LeCun, Y., et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34 0 (5): 0 1--41, 2007

    work page 2007

Show all 35 references
  1. [9]

    and Scarselli , F

    Bianchini , M. and Scarselli , F. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks and Learning Systems, 25 0 (8): 0 1553--1565, Aug 2014. ISSN 2162-237X. doi:10.1109/TNNLS.2013.2293637

    2014

  2. [10]

    Gated feedback recurrent neural networks

    Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Gated feedback recurrent neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pp.\ 2067--2075. JMLR.org, 2015. URL http://dl.acm.org/citation....

    2015

  3. [11]

    BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . arXiv:1810.04805 [cs], May 2019. URL http://arxiv.org/abs/1810.04805

    2019 arXiv

  4. [12]

    Generating sequences with recurrent neural networks

    Graves, A. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850, 2013

    2013 arXiv

  5. [13]

    Adaptive computation time for recurrent neural networks

    Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016

    2016 arXiv

  6. [14]

    and Schmidhuber, J

    Graves, A. and Schmidhuber, J. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18 0 (5): 0 602 -- 610, 2005. ISSN 0893-6080. doi:https://doi.org/10.1016/j.neunet.2005.06.042. URL http://www.sciencedirect.com/sci...

    2005 doi

  7. [15]

    Robust online time series prediction with recurrent neural networks

    Guo , T., Xu , Z., Yao , X., Chen , H., Aberer , K., and Funaya , K. Robust online time series prediction with recurrent neural networks. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp.\ 816--825, Oct 2016. doi:10.1109/DSAA.2016.92

    2016 doi

  8. [16]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997. doi:10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  9. [17]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  10. [18]

    Independently recurrent neural network (indrnn): Building a longer and deeper rnn

    Li, S., Li, W., Cook, C., Zhu, C., and Gao, Y. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    2018

  11. [19]

    W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., and Luis, T

    Ling, W., Dyer, C., Black, A. W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., and Luis, T. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pr...

    2015 doi

  12. [20]

    Relationship of Wikipedia Text to Clean Text , June 2006

    Mahoney, M. Relationship of Wikipedia Text to Clean Text , June 2006. URL http://mattmahoney.net/dc/textdata.html

    2006

  13. [21]

    Subword language modeling with neural networks

    Mikolov, T., Sutskever, I., Deoras, A., Le, H.-S., and Kombrink, S. Subword language modeling with neural networks. Preprint, 2012. URL http://www.fit.vutbr.cz/ imikolov/rnnlm/char.pdf

    2012

  14. [22]

    D., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., Tsarfaty, R., and Zeman, D

    Nivre, J., de Marneffe, M.-C., Ginter, F., Goldberg, Y., Haji c , J., Manning, C. D., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., Tsarfaty, R., and Zeman, D. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Confe...

    2016

  15. [23]

    On the difficulty of training recurrent neural networks

    Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML'13, pp.\ III--1310--III--1318. JMLR.org, 2013. URL http://dl....

    2013

  16. [24]

    How to construct deep recurrent neural networks

    Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), 2014

    2014

  17. [25]

    Automatic differentiation in PyTorch

    Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch . In NIPS Workshop on the future of gradient-based machine learning software & techniques, 2017

    2017

  18. [26]

    Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss

    Plank, B., S gaard, A., and Goldberg, Y. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 412--418,...

    2016 doi

  19. [27]

    Deep learning made easier by linear transformations in perceptrons

    Raiko, T., Valpola, H., and Lecun, Y. Deep learning made easier by linear transformations in perceptrons. In Lawrence, N. D. and Girolami, M. (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Ma...

    2012

  20. [28]

    Learning complex, extended sequences using the principle of history compression

    Schmidhuber, J. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4 0 (2): 0 234--242, 1992

    1992

  21. [29]

    and Paliwal , K

    Schuster , M. and Paliwal , K. K. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45 0 (11): 0 2673--2681, Nov 1997. ISSN 1053-587X. doi:10.1109/78.650093

    1997 doi

  22. [30]

    A gold standard dependency corpus for E nglish

    Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., and Manning, C. A gold standard dependency corpus for E nglish. In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC '14) , pp.\ 2897--2904, Reykjavik, I...

    2014

  23. [31]

    Sutskever, I., Martens, J., and Hinton, G. E. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp.\ 1017--1024, 2011

    2011

  24. [32]

    Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp.\ 3104--3112, 2014

    2014

  25. [33]

    R., and Bengio, Y

    Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R. R., and Bengio, Y. Architectural complexity measures of recurrent neural networks. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Sy...

    2016

  26. [34]

    G., Srivastava, R

    Zilly, J. G., Srivastava, R. K., Koutn\' k, J., and Schmidhuber, J. Recurrent highway networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 4189--4198, In...

    2017

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.