Pith. sign in

REVIEW 3 major objections 7 minor 36 references

Compact Recurrent Transformer with Persistent Memory

T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a compact Transformer with a single RNN memory vector can match or beat full-context Transformers on language modeling and video classification while using much shorter segments and substantially fewer FLOPs.

desk verdict A sensible but incremental recurrent-memory Transformer whose main claims are plausible yet under-reported, with an unaddressed capacity question for long sequences. read the letter →

arxiv 2505.00929 v1 pith:FOHBX5M2 submitted 2025-05-02 cs.LG stat.ML

Edison Mucllari , Zachary Daniels , David Zhang , Qiang Ye This is my paper · ORCID
classification cs.LGstat.ML
keywords compactrecurrenttransformerpersistentmemorysingle-vectorsegment-levelrecurrencelanguagemodelingvideoclassificationedgecomputingNCGRU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that long-range context in a Transformer does not have to be stored as many key/value vectors: one RNN hidden vector, updated once per segment and attending as a single memory token, can carry the global information. If true, the result matters because it promises Transformer-level language modeling and video classification at a fraction of the compute and memory, within reach of low-power edge devices. On WordPTB and WikiText-103 the reported perplexities match or beat Transformer-XL at half or quarter segment lengths, and on Toyota Smarthome the reported accuracy tops prior state of the art without using pose data.

What carries the argument

The central mechanism is a single persistent memory vector $m_{tn}$: after a Transformer processes a segment, the segment's output embeddings pass through a GRU/NCGRU (a gated recurrent unit with an orthogonal Neumann–Cayley transition) whose final hidden state becomes the memory for the next segment and is concatenated to that next segment as one extra token in self-attention. A second GRU/NCGRU serves as recurrent positional encoding, deriving token positions from the previous memory state so the memory token and input embeddings are compatible. The argument carried by the machinery is the gradient bound in Eq. (2), which factors the influence of an earlier token on a later one through the RNN memory and, for NCGRU with saturated gates, removes the exponential decay factor $\alpha+\beta\|U_c\|_2$, so memory can flow across segments without vanishing.

What would settle it

Run CRT against Transformer-XL on a long-context corpus with segment lengths of 512 or 1024 tokens (or on a benchmark like PG-19) and track perplexity as segment length grows. If CRT's perplexity gap to Transformer-XL widens steadily while the memory vector's norm or update magnitude saturates, then a one-vector memory is the bottleneck; if the gap stays constant, the capacity concern is resolved for practical lengths.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a persistent memory consisting of one vector—obtained by feeding a segment's Transformer output through a GRU or NCGRU and keeping the final hidden state—preserves enough long-range information that a shallow Transformer over short segments performs comparably to or better than Transformers that attend over much longer contexts. The strongest concrete evidence is a 16-layer CRT reaching 55.7 perplexity on WordPTB with 70-token segments (Transformer-XL: 54.5) and 63.0 with only 17-token segments, and a 3-layer CRT reaching 31.8 on WikiText-103 at segment 150 (Transformer-XL: 32.6). In video, CR-ViT reports 73.4 mean class accuracy on Toyota Smarthome, above PI-ViT's 72.9, without skeleton inputs. These results are presented as showing that explicit RNN memory and RNN positional encoding together close most of the gap to full-context attention.

Load-bearing premise

The load-bearing premise is that a single RNN hidden vector of fixed dimension can store enough of a long history to substitute for attention over all previous tokens; the paper itself notes RNN states can saturate and provides no experiments beyond 150-token segments.

Editorial extensions

If this is right

  • Because memory is one vector instead of $n$ cached key/value vectors, self-attention cost stays dominated by the current segment; the paper's FLOP table shows CRT near the plain Transformer and well below Transformer-XL.
  • A 16-layer CRT with 17-token segments reaches 63.0 perplexity on WordPTB, where a plain Transformer at the same segment length scores 79.6 and Transformer-XL 68.2—showing the hidden state substitutes for most of the extended context.
  • On WikiText-103, the 3-layer CRT at 150-token segments scores 31.8 perplexity, below Transformer-XL's 32.6, and the 16-layer model comes within 1.8 points of Transformer-XL while using fewer FLOPs.
  • On Toyota Smarthome, CR-ViT reports 73.4 mCA, beating PI-ViT's 72.9 without the 2D/3D skeleton modules PI-ViT relies on.
  • Inference time is roughly equal to the plain Transformer (997.5 ms/batch) versus 1421.3 for Transformer-XL on 16-layer WikiText-103.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests segments up to 150 tokens; our inference is that on much longer documents the single-vector memory will eventually saturate, and the orthogonal NCGRU variant is the part most likely to delay that saturation.
  • Because the memory RNN is decoupled from the Transformer body, the persistent-memory idea could be grafted onto cheaper attention approximations (linear attention, sliding-window attention) with the same one-token overhead, which the paper does not explore.
  • The recurrent positional encoding's strong ablation effect (61.0 vs 58.3 perplexity contributions) suggests that a segment-relative position signal, not just memory content, is doing a large share of the work; an explicit comparison of CRT against a model with only learned segment embeddings would isolate that contribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes the Compact Recurrent Transformer (CRT), a segment-based Transformer with a single persistent memory vector maintained by a GRU or NCGRU, plus a recurrent positional encoding mechanism. On Word PTB and WikiText-103 the authors report next-token prediction perplexities for 3-layer and 16-layer models, comparing CRT with a standard Transformer and Transformer-XL across segment lengths of 17, 35, 70, and 150 tokens. On Toyota Smarthome they report mean class accuracy for a video variant, CR-ViT, and claim state-of-the-art performance. An appendix provides additional GRU-versus-NCGRU comparisons and a proof of a gradient bound for the recurrent memory path.

Significance. If the empirical claims hold, the architecture is a meaningful step toward deploying Transformer-style models on resource-constrained devices: it uses one memory vector instead of a block of memory tokens, and the ablation shows that both the RNN memory and the recurrent positional encoding contribute additively. The paper also includes a complexity table and a gradient-stability argument, which are useful for understanding the proposed design. The main limitations are that all results are single-run point estimates, the long-context extrapolation is not stress-tested, and the FLOP formulas are not derived in the text. These issues do not invalidate the core idea, but they need to be addressed before the central claims are fully supported.

major comments (3)
  1. [Section 5, Tables 2–5 and Table 7] All experimental results are reported as single point estimates, with no number of seeds, standard deviations, or significance tests. This is particularly problematic for the headline SOTA claim in Section 5.3, which rests on a 0.5-point mCA difference (73.4 vs. 72.9 in Table 7); this margin is well within typical run-to-run variability for video classification. The language-modeling comparisons in Tables 2–5 also need error bars or repeated-seed confidence intervals, especially in the 16-layer cases where CRT is within 1–2 perplexity points of Transformer-XL.
  2. [Section 1 and Section 6] The paper itself concedes in Section 1 that RNN hidden states "have information capacity limits which can become saturated when summarizing long sequences," yet no experiment stresses the memory beyond 150-token segments in language modeling or 16 frames in video. The gradient bound in Eq. (3) controls the Jacobian norm of the recurrent path; it does not address the information capacity of a single d_m-dimensional memory vector. The concluding extrapolation to "context lengths of hundreds of thousands of tokens" (Section 6) is therefore not supported by the present evidence. The authors should either add long-sequence stress tests (e.g., segment lengths of 256 or 512 on WikiText-103) or substantially moderate the conclusion.
  3. [Table 1 and Section 4] The FLOP formulas in Table 1 are presented without derivation and without a precise counting convention (FLOPs versus multiply-adds; whether softmax, layer norm, and the output linear layer are included). Since the paper's central advantage is "substantially reduced FLOPs," the formulas must be derived in the text or an appendix and, ideally, validated with a profiler or an independent reference count. In addition, the sentence "Ours is comparable to Transformer that has additional cost related to the relative positional encoding" is unclear because the Transformer row in Table 1 does not include any relative-position-encoding term; the comparison baseline needs to be specified precisely.
minor comments (7)
  1. [Abstract, first paragraph] The phrase "these approached" should be "these approaches."
  2. [Figures 5 and 6 captions] The captions refer to the "Word PDB" dataset; this should be "Word PTB."
  3. [Section 5.3, text near Table 7] The text reports an mCA of 73.5 for CRViT, while Table 7 lists 73.4; these numbers should be reconciled.
  4. [Section 2.2, Eq. (1) introduction] The sentence "queries Q, and values Q" should read "queries Q, keys K, and values V".
  5. [Section 4, Eqs. (2)–(3), and Appendix B] The notation in the gradient analysis is inconsistent: the text uses both "m2n" and "mtn", "yj" and "yk", and "hj" and "hi" in the same chain-rule expressions. The indices should be corrected so that the derivation in Theorem 3 can be verified.
  6. [Section 5.2, Table 6] The inference-time comparison reports one number per model without specifying hardware, batch size, or measurement procedure; without these details the claim that Transformer-XL's inference time is "nearly 50% higher" is not reproducible.
  7. [Abstract and Section 5.1] The phrase "full-length Transformers" is ambiguous: the experiments compare against Transformer and Transformer-XL with the stated segment lengths, not against a Transformer that attends to all previous tokens in the corpus. Please rephrase to describe the baselines accurately.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: empirical benchmarks are self-contained; the NCGRU self-citation supports only an independent gradient bound.

full rationale

The paper's central claims are empirical, not derived from fitted constants: the reported perplexities in Tables 2-5 and the mCA in Table 7 are measured against Transformer-XL, standard Transformer, and published video baselines, and they are not obtained from the model equations by construction. The only notable self-citation is to the authors' NCGRU work [27], used in Section 4 and Appendix B: Lemma 1 and Lemma 2 are quoted from [27] to prove the gradient bound in Eq. (2)/(3). That bound is an independent mathematical estimate of GRU derivatives and is used only to argue that memory information can propagate across segments without exponential gradient decay; it does not compute or fit any of the reported perplexities or accuracies. The architectural choices (single memory token, RNN memory, RNN positional encoding) are design choices validated by the ablation in Table 8, not definitions that force the benchmark outcomes. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no empirical result reduces by construction to a self-citation. The concern that a single RNN memory vector may saturate on longer sequences is a correctness or robustness risk, not a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's central empirical claims depend on architecture choices (single-vector memory, RNN position encoding, BPTT) rather than on a small set of fitted constants. The main borrowed mathematical content is the GRU derivative bound from the authors' earlier NCGRU paper; it is self-cited but not used to fabricate the headline results. Hyperparameters such as segment length and embedding dimension are hand-chosen and only partially swept.

free parameters (3)
  • Segment lengths = 17, 35, 70 tokens (language); 16 frames (video)
    Chosen by hand; the central efficiency claim depends on the segment-length sweep, but the paper reports no principled selection and no sensitivity analysis outside these values.
  • Memory vector dimension = equal to embedding dimension d_m
    The memory capacity is tied to d_m, but no experiments vary d_m for the memory alone; the single-vector capacity assumption is untested.
  • Number of Transformer layers = 3 or 16 for language; unspecified depth for CR-ViT
    Two model sizes are used; the 'compact' claim is only shown at these depths.
assumptions (4)
  • standard math GRU derivative bounds in Lemma 1 and Lemma 2 of Appendix B are taken without proof from the authors' prior NCGRU paper [27].
    Theorem 3 and Eq. 2 rely directly on these bounds; the paper does not reprove them.
  • domain assumption All cross-segment information reaches the current segment only through the single memory vector; no other tokens from previous segments are attended to.
    This is the architectural definition in Section 3 and Figure 2, and it is what makes the FLOP savings possible.
  • domain assumption The RNN positional encoding supplies sufficient positional information so the simplified attention score A_CRT with no relative position terms is adequate.
    Stated in Section 3 'Recurrent Position Encoding'; supported only by the ablation in Table 8, not by an analysis of representational capacity.
  • domain assumption Backpropagation through time over the memory vector trains the recurrent path stably enough for the claimed results.
    Section 3 and Section 4 assert BPTT through the memory vector; the gradient bound is loose and does not prove convergence.
invented entities (1)
  • Persistent memory vector (single memory token)
    purpose: Compresses all context before the current segment into one RNN hidden state that is concatenated to the segment as a token and updated after each segment.
    The only evidence is the paper's own experiments; there is no theoretical capacity bound, no external benchmark, and no released implementation to independently confirm that one vector suffices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compact Recurrent Transformer with Persistent Memory." pith.science (2026). https://pith.science/paper/FOHBX5M2

@misc{pith2026250500929,
  author       = {Pith},
  title        = {Pith review of: Compact Recurrent Transformer with Persistent Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FOHBX5M2}},
  note         = {Machine review of arXiv:2505.00929}
}
read the original abstract

The Transformer architecture has shown significant success in many language processing and visual tasks. However, the method faces challenges in efficiently scaling to long sequences because the self-attention computation is quadratic with respect to the input length. To overcome this limitation, several approaches scale to longer sequences by breaking long sequences into a series of segments, restricting self-attention to local dependencies between tokens within each segment and using a memory mechanism to manage information flow between segments. However, these approached generally introduce additional compute overhead that restricts them from being used for applications where limited compute memory and power are of great concern (such as edge computing). We propose a novel and efficient Compact Recurrent Transformer (CRT), which combines shallow Transformer models that process short local segments with recurrent neural networks to compress and manage a single persistent memory vector that summarizes long-range global information between segments. We evaluate CRT on WordPTB and WikiText-103 for next-token-prediction tasks, as well as on the Toyota Smarthome video dataset for classification. CRT achieves comparable or superior prediction results to full-length Transformers in the language datasets while using significantly shorter segments (half or quarter size) and substantially reduced FLOPs. Our approach also demonstrates state-of-the-art performance on the Toyota Smarthome video dataset.

Figures

Figures reproduced from arXiv: 2505.00929 by the authors.

Figure 1
Figure 1. Compact Recurrent Transformer Architecture [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Transformer with Memory Token ory to be used in the next segment. Simultaneously, mn acts as the initial hidden state for the next iteration of the memory RNN. Extending this concept, the last hidden state for the current iteration, h2n = m2n, becomes the mem￾ory for the next iteration and serves as the initial hidden state for the subsequent memory RNN, creating a continu￾ous chain. This demonstrates that the memor… view at source ↗
Figure 4
Figure 4. Compact Recurrent ViT Architecture Compact Recurrent ViT Architecture: The Compact Recurrent ViT model employs a unique two-iteration train￾ing approach for each video. In the first iteration, the mem￾ory vector is initialized to zeros. After processing through the network, the output, except the class token, is extracted and passed through a Recurrent Neural Network (RNN) to generate the memory vector for the subse… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Recurrent Memory Architecture Recurrent Position Encoding: A crucial aspect of our architecture is the Recurrent Position Encoding, designed to handle the positional encoding when processing infor￾mation from the new input sequence. We introduce a new GRU/NCGRU model f…
Figure 5
Figure 5. Figure 5: Comparing the CRT model to the baseline Transformer [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparing the CRT model to the baseline Transformer [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    A neural probabilistic language model

    Yoshua Bengio, R ´ejean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural in- formation processing systems, 13, 2000. 3

    work page 2000

  3. [3]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 1, 3

    work page 2021

  4. [4]

    Recur- rent memory transformer

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recur- rent memory transformer. Advances in Neural Information Processing Systems, 35:11079–11091, 2022. 1, 3

    work page 2022

  5. [5]

    Memory transformer

    Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grig- ory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527, 2020. 1

    arXiv 2006

  6. [6]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. 2, 3, 4

    arXiv 2014

  7. [7]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 1, 3

    work page 2023

  8. [8]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. 1, 3, 6, 7, 8

    arXiv 1901

Show all 36 references
  1. [9]

    Toyota smarthome: Real-world activities of daily living

    Srijan Das, Rui Dai, Michal Koperski, Luca Minci- ullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 833–842, 2019. 1, 3, 8

    2019

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  3. [11]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  4. [12]

    A theoretically grounded application of dropout in recurrent neural networks

    Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Ad- vances in neural information processing systems , 29, 2016. 3

    2016

  5. [13]

    Ef- ficient softmax approximation for gpus

    E Grave, A Joulin, M Ciss ´e, D Grangier, and H J ´egou. Ef- ficient softmax approximation for gpus. arxiv 2016. arXiv preprint arXiv:1609.04309. 3

    2016 arXiv

  6. [14]

    Eigenvalue normalized recur- rent neural networks for short term memory

    Kyle Helfrich and Qiang Ye. Eigenvalue normalized recur- rent neural networks for short term memory. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 4115–4122, 2020. 3

    2020

  7. [15]

    Orthogo- nal recurrent neural networks with scaled cayley transform

    Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogo- nal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning , pages 1969–1978. PMLR, 2018. 3

    1969

  8. [16]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 2, 3

    1997

  9. [17]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982. 2

    1982

  10. [18]

    Block-recurrent transformers

    DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers. Advances in Neural Information Processing Systems , 35: 33248–33261, 2022. 1, 3

    2022

  11. [19]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. 1

    2021 arXiv

  12. [20]

    Gated orthog- onal recurrent units: On learning to forget

    Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated orthog- onal recurrent units: On learning to forget. Neural computa- tion, 31(4):765–783, 2019. 3

    2019

  13. [21]

    Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group

    Mario Lezcano-Casado and David Mart ´ınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In Pro- ceedings of the 36th International Conference on Machine Learning, pages 3794–3803. PMLR, 2019. 3

    2019

  14. [22]

    Complex unitary recurrent neural networks using scaled cayley transform

    Kehelwala Dewage Maduranga, Kyle Helfrich, and Qiang Ye. Complex unitary recurrent neural networks using scaled cayley transform. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4528–4535, 2019. 3

    2019

  15. [23]

    Building a large annotated corpus of english: The penn treebank

    Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. 1993. 6

    1993

  16. [24]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. 3, 6

    2016 arXiv

  17. [25]

    Hellicar, Ashfaqur Rahman, and James Bailey

    Zakaria Mhammedi, Andrew D. Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Proceedings of ICML 2017, pages 2401–2409. PMLR, 2017. 3

    2017

  18. [26]

    Recurrent neural network based language model

    Tomas Mikolov, Martin Karafi ´at, Lukas Burget, Jan Cer- nock`y, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, pages 1045–1048. Makuhari, 2010. 3

    2010

  19. [27]

    Orthogonal gated recurrent unit with neumann-cayley transformation

    Edison Mucllari, Vasily Zadorozhnyy, Cole Pospisil, Duc Nguyen, and Qiang Ye. Orthogonal gated recurrent unit with neumann-cayley transformation. arXiv preprint arXiv:2208.06496, 2022. 2, 3, 4, 13, 14

    2022 arXiv

  20. [28]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 1, 3

    2018

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. 1

    2020

  22. [30]

    Just add?! pose induced video transformers for understanding activities of daily liv- ing

    Dominick Reilly and Srijan Das. Just add?! pose induced video transformers for understanding activities of daily liv- ing. In Proceedings of the IEEE/CVF Conference on Com- 10 puter Vision and Pattern Recognition , pages 18340–18350,

  23. [31]

    Learning internal representations by error propagation,

    David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal representations by error propagation,

  24. [32]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. 1, 3

    2022 arXiv

  25. [33]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1, 3, 4

    2017

  26. [34]

    R- transformer: Recurrent neural network enhanced trans- former

    Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. R- transformer: Recurrent neural network enhanced trans- former. arXiv preprint arXiv:1907.05572, 2019. 5

    1907 arXiv

  27. [35]

    Full-capacity unitary recurrent neu- ral networks

    Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neu- ral networks. In Advances in Neural Information Processing Systems 29, pages 4880–4888, 2016. 3

    2016

  28. [36]

    Memformer: The memory-augmented transformer

    Qingyang Wu, Zhenzhong Lan, Jing Gu, and Zhou Yu. Memformer: The memory-augmented transformer. 2020. 1 11 Compact Recurrent Transformer with Persistent Memory Appendix A. Additional Recurrence Analysis from Experiments In this section, we will examine an analysis of the differ...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.