Pith. sign in

REVIEW 12 cited by

Memory Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2006.11527 v2 pith:4FWAI53M submitted 2020-06-20 cs.CL cs.LGcs.NE

Mikhail S. Burtsev , Yuri Kuratov , Anton Peganov , Grigory V. Sapunov This is my paper · ORCID
classification cs.CLcs.LGcs.NE
keywords memoryrepresentationslanguagemodeltaskstransformerglobalinformation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Transformer-based models have achieved state-of-the-art results in many natural language processing tasks. The self-attention architecture allows transformer to combine information from all elements of a sequence into context-aware representations. However, information about the context is stored mostly in the same element-wise representations. This might limit the processing of properties related to the sequence as a whole more difficult. Adding trainable memory to selectively store local as well as global representations of a sequence is a promising direction to improve the Transformer model. Memory-augmented neural networks (MANNs) extend traditional neural architectures with general-purpose memory for representations. MANNs have demonstrated the capability to learn simple algorithms like Copy or Reverse and can be successfully trained via backpropagation on diverse tasks from question answering to language modeling outperforming RNNs and LSTMs of comparable complexity. In this work, we propose and study few extensions of the Transformer baseline (1) by adding memory tokens to store non-local representations, (2) creating memory bottleneck for the global information, (3) controlling memory update with dedicated layer. We evaluate these memory augmented Transformers and demonstrate that presence of memory positively correlates with the model performance for machine translation and language modelling tasks. Augmentation of pre-trained masked language model with memory tokens shows mixed results for tasks from GLUE benchmark. Visualization of attention patterns over the memory suggest that it improves the model's ability to process a global context.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Appending a few trainable tokens to each encoder layer of a frozen monocular depth estimator aligns fisheye image embeddings with perspective embeddings, enabling zero-shot fisheye depth estimation.

  2. A standard transformer and attention with linear biases for molecular conformer generation

    q-bio.BM 2025-06 conditional novelty 6.0 of 10

    S23D, a standard LLaMA-style transformer with fixed linear attention biases based on shortest-path graph distances, sets state-of-the-art recall on GEOM-DRUGS at 24.8 million parameters.

  3. Learning to Insert [PAUSE] Tokens for Better Reasoning

    cs.CL 2025-06 reject novelty 6.0 of 10

    A likelihood-based [PAUSE] token insertion method for fine-tuning shows small gains on GSM8K and MBPP, but the AQUA-RAT result is unreliable because the test set contains training samples.

  4. Multi-Token Prediction Needs Registers

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Interleaved register tokens with offset-encoded position ids add a training-only multi-token prediction objective that improves fine-tuning, PEFT, and image-generation pretraining over next-token baselines.

  5. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Adaptive Parallel Encoding restores sequential-style attention in parallel KV-cache RAG/ICL with three training-free tweaks, enabling fast pre-cached long-context generation.

  6. SRMT: Shared Memory for Multi-agent Lifelong Pathfinding

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Shared recurrent memory with global broadcast improves coordination in decentralized multi-agent pathfinding and generalizes to longer corridors better than private-memory baselines.

  7. Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Padding tokens in text-to-image models can carry semantic information or act as diffusion-time registers, depending on training and attention architecture.

  8. Hymba: A Hybrid-head Architecture for Small Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Hymba-1.5B, a hybrid-head model with parallel attention and Mamba heads plus learnable meta tokens, achieves state-of-the-art accuracy among sub-2B models and outperforms Llama-3.2-3B with an 11.67x smaller cache and ...

  9. Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A clip-level HD map construction method that updates bird's-eye-view features with compact clip tokens and propagates tokens across clips, reporting state-of-the-art mAP on nuScenes and Argoverse2.

  10. KiVi: Kinesthetic-Visuospatial Integration for Dynamic and Safe Egocentric Legged Locomotion

    cs.RO 2025-09 conditional novelty 5.0 of 10

    A quadruped locomotion controller that explicitly separates proprioceptive and visual pathways stays stable under camera occlusion and visual corruption that destabilizes fused-vision policies.

  11. Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers

    cs.LG 2025-05 reject novelty 5.0 of 10

    The paper claims pause tokens strictly increase constant-precision, constant-depth Transformer expressivity from a subset of AC0 to AC0 (and log-precision to TC0), but the constant-precision proof is not sound as written.

  12. Compact Recurrent Transformer with Persistent Memory

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A recurrent-Transformer hybrid that compresses long-range context into a single RNN state vector matches segment-based Transformer baselines with fewer FLOPs on two language benchmarks and one video benchmark.

Pith tools