Pith. sign in

REVIEW 10 cited by

Recurrent Memory Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2207.06881 v2 pith:4PR2P4ZE submitted 2022-07-14 cs.CL cs.LG

Aydar Bulatov , Yuri Kuratov , Mikhail S. Burtsev This is my paper · ORCID
classification cs.CLcs.LG
keywords memorysequenceinformationtransformerprocessingrecurrentrepresentationstasks
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Transformer-based models show their effectiveness across multiple domains and tasks. The self-attention allows to combine information from all sequence elements into context-aware representations. However, global and local information has to be stored mostly in the same element-wise representations. Moreover, the length of an input sequence is limited by quadratic computational complexity of self-attention. In this work, we propose and study a memory-augmented segment-level recurrent Transformer (RMT). Memory allows to store and process local and global information as well as to pass information between segments of the long sequence with the help of recurrence. We implement a memory mechanism with no changes to Transformer model by adding special memory tokens to the input or output sequence. Then the model is trained to control both memory operations and sequence representations processing. Results of experiments show that RMT performs on par with the Transformer-XL on language modeling for smaller memory sizes and outperforms it for tasks that require longer sequence processing. We show that adding memory tokens to Tr-XL is able to improve its performance. This makes Recurrent Memory Transformer a promising architecture for applications that require learning of long-term dependencies and general purpose in memory processing, such as algorithmic tasks and reasoning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 26 citations worldwide. Full citation record

  1. Consolidator: Learning Persistent Routed Memory Across Context Boundaries

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A learned slot-local transform lets a frozen backbone update and retrieve a persistent memory after context reset, and using that memory to guide routing nearly doubles updated-mapping recall.

  2. HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A memory adapter with separate retrieval and workspace registers routes historical visual evidence into a frozen video predictor and improves physical prediction accuracy on IntPhys2.

  3. LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

    cs.CL 2026-08 reject novelty 6.0 of 10

    LiveMem augments a pretrained LLM with a GDN2 recurrent state that is trained under context turnover, and claims the state remains useful after the supporting tokens leave attention.

  4. Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SPIRAL aligns a model's rendered-image input behavior to its native-text input behavior using self-generated text-path supervision, nearly closing the vision-text compression gap on VTCBench.

  5. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  6. A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Gist-token compression is near-lossless for fuzzy long-context tasks but fails at exact recall; fine-grained autoencoding and token importance weighting recover some of the gap.

  7. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

  8. Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes

    cs.LG 2025-05 conditional novelty 4.0 of 10

    DBGFQN swaps transformer feed-forward networks for a single BiGRU layer and reports improved average success rate on 23 POMDP environments, but the headline gains and parameter reductions are not reproducible from the...

  9. LM2: Large Memory Models

    cs.CL 2025-02 conditional novelty 4.0 of 10

    LM2 adds a cross-attention memory bank with input, forget, and output gates to every decoder block, reporting large BABILong gains and no MMLU drop.

  10. Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

    cs.CL 2025-01 conditional novelty 3.0 of 10

    A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.

Pith tools