pith. sign in

arxiv: 1907.05572 · v1 · pith:MO3HD27Jnew · submitted 2019-07-12 · 💻 cs.LG · cs.CL· cs.CV· eess.AS

R-Transformer: Recurrent Neural Network Enhanced Transformer

Zhiwei Wang , Yao Ma , Zitao Liu , Jiliang Tang This is my paper

Pith reviewed 2026-05-24 22:53 UTC · model grok-4.3

classification 💻 cs.LG cs.CLcs.CVeess.AS
keywords sequence modelingrecurrent neural networkstransformermulti-head attentionlocal structureslong-term dependenciesposition embeddings
0
0 comments X

The pith

R-Transformer combines recurrent layers with multi-head attention to model both local structures and long-term dependencies in sequences without position embeddings.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces R-Transformer as a sequence model that merges recurrent neural networks with the multi-head attention of Transformers. It aims to fix the inability of RNNs to handle very long dependencies and to parallelize, while also fixing the lack of local structure modeling and the reliance on position embeddings in attention-only models. A reader would care because many practical tasks involve sequences where both nearby details and distant context matter, and removing the need to engineer position embeddings could simplify design. The authors report that the resulting model beats prior methods by a large margin across tasks from multiple domains.

Core claim

The R-Transformer enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks.

What carries the argument

The R-Transformer architecture that places recurrent components ahead of multi-head attention blocks to process input sequences.

If this is right

  • The model outperforms state-of-the-art methods by a large margin in most tasks across a wide range of domains.
  • It captures both local structures and global long-term dependencies without using position embeddings.
  • It retains the parallelization benefits of attention while gaining local modeling from recurrence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Removing position embeddings could lower the design effort required when adapting the model to new sequence domains.
  • The hybrid pattern might extend naturally to tasks that mix short-range patterns with planning over long horizons.
  • Further scaling could test whether the same local-global split remains effective at much larger sequence lengths.

Load-bearing premise

The recurrent components can capture local structures so effectively that position embeddings become unnecessary while attention still handles the long-range dependencies.

What would settle it

An experiment on a long-sequence benchmark in which the R-Transformer shows no improvement over a standard Transformer that also omits position embeddings would falsify the claimed benefit of the hybrid design.

Figures

Figures reproduced from arXiv: 1907.05572 by Jiliang Tang, Yao Ma, Zhiwei Wang, Zitao Liu.

Figure 1
Figure 1. Figure 1: The illustration of one layer of R-Transformer. There are three different networks that are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of the original and local RNN. In contrast to orignal RNN which maintains [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
read the original abstract

Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code publicly available at \url{https://github.com/DSE-MSU/R-transformer}.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript proposes the R-Transformer, a hybrid architecture that augments multi-head attention with recurrent components to capture both local structures and global long-term dependencies in sequences. It asserts that the model achieves this without any position embeddings and reports empirical outperformance over state-of-the-art methods by a large margin across tasks from diverse domains, with code released publicly.

Significance. If the empirical claims hold under rigorous verification, the work would be moderately significant: it offers a concrete hybrid that leverages RNN locality and attention globality while sidestepping position-embedding design, and the public code link directly supports reproducibility of the reported results.

minor comments (2)
  1. Abstract: the phrase 'outperforms the state-of-the-art methods by a large margin in most of the tasks' should be accompanied by quantitative margins, number of tasks/domains, and at least one table reference in the main text for immediate clarity.
  2. The manuscript would benefit from an explicit statement (perhaps in §3 or §4) confirming that no positional information of any form is injected, together with a short ablation removing the recurrent component to isolate its contribution.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our work on the R-Transformer and for recommending minor revision. The report correctly summarizes the model's design for capturing local structures via recurrent components and global dependencies via multi-head attention without position embeddings, along with the public code release. No specific major comments were enumerated in the report.

Circularity Check

0 steps flagged

No significant circularity in architecture proposal or empirical claims

full rationale

The paper introduces the R-Transformer as a hybrid architecture and supports its claims solely through new empirical evaluations on diverse sequence tasks, with public code provided for reproducibility. No derivation chain, first-principles prediction, or fitted parameter is presented that reduces by construction to the model's own inputs or self-citations; the absence of position embeddings and the local/global dependency capture are design choices validated externally via experiments rather than tautological definitions or load-bearing self-citations.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review is based on the abstract only; no specific free parameters, axioms, or invented entities are detailed. The claim rests on the domain assumption that a hybrid RNN-attention block will jointly capture local and global structure.

pith-pipeline@v0.9.0 · 5756 in / 1056 out tokens · 51083 ms · 2026-05-24T22:53:54.806470+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Titans: Learning to Memorize at Test Time

    cs.LG 2024-12 unverdicted novelty 6.0

    Titans combine attention for current context with a learnable neural memory for long-term history, achieving better performance and scaling to over 2M-token contexts on language, reasoning, genomics, and time-series tasks.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 1 Pith paper · 14 internal anchors

  1. [1]

    Character-Level Language Modeling with Deeper Self-Attention

    Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level lan- guage modeling with deeper self-attention. arXiv preprint arXiv:1808.04444,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Layer Normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

    Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. arXiv preprint arXiv:1206.6392,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

    Kyunghyun Cho, Bart Van Merri¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

    Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860,

    work page internal anchor Pith review Pith/arXiv arXiv 1901

  8. [8]

    Universal Transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    A Convolutional Encoder Model for Neural Machine Translation

    Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. A convolutional encoder model for neural machine translation. arXiv preprint arXiv:1611.02344,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Session-based Recommendations with Recurrent Neural Networks

    Bal´azs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based rec- ommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

    David Krueger, Tegan Maharaj, J´anos Kram´ar, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

    Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Recurrent Memory Networks for Language Modeling

    Ke Tran, Arianna Bisazza, and Christof Monz. Recurrent memory networks for language modeling. arXiv preprint arXiv:1601.01272,

    work page internal anchor Pith review Pith/arXiv arXiv