Pith. sign in

REVIEW 5 cited by

Associative Recurrent Memory Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.04841 v2 pith:C4W7KMFQ submitted 2024-07-05 cs.CL cs.AIcs.LG

Ivan Rodkin , Yuri Kuratov , Aydar Bulatov , Mikhail Burtsev This is my paper
classification cs.CLcs.AIcs.LG
keywords associativetransformerarmtcontextinformationlongmemoryrecurrent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  2. SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation

    cs.RO 2026-03 conditional novelty 6.0 of 10

    SeedPolicy introduces self-evolving gated attention to extend the temporal horizon of diffusion policies, yielding 36.8% and 169% relative gains over standard DP on clean and randomized RoboTwin 2.0 tasks.

  3. Extending LLM Context via Associative Recurrent Memory

    cs.CL 2026-07 conditional novelty 5.0 of 10

    ARMT-augmented 1B-class LLMs, trained with continued pretraining, synthetic long data, curriculum, and selective memory layers, keep in-window quality while generalizing past 32k–65k tokens at constant memory and ~30%...

  4. Re:Frame -- Retrieving Experience From Associative Memory

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A plug-in that retrieves expert actions from a small associative memory buffer improves offline Decision Transformer performance on three of four D4RL MuJoCo tasks, with gains up to 10.7 points.

  5. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Pith tools