REVIEW 13 cited by
Memory Layers at Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Memory Layers at Scale
read the original abstract
Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
Forward citations
Cited by 13 Pith papers
-
Can a Language Model Learn Facts Continually in Its Weights?
Facts written into LLM weights are stored but question-keyed; broad data creates usable knowledge while later writes destroy access rather than storage, leaving context more reliable than weights for composition and r...
-
A Parametric Memory Head for Continual Generative Retrieval
A product-key parametric memory head with selective sparse updates mitigates catastrophic forgetting in generative retrieval models during sequential addition of new documents.
-
MemSFT: Mitigating Alignment Tax with an External Parametric Memory
MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...
-
Can a Language Model Learn Facts Continually in Its Weights?
Weight writes create question-keyed knowledge that later sequential writes redirect, so context—not weights—is the reliable channel for facts that must survive or be composed.
-
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.
-
Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback
A hybrid attention mechanism with editable request-local memory slots and sparse fallback achieves high accuracy on synthetic overwrite, version, and anti-pollution tasks where pure fixed-state or sparse methods fail,...
-
Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memory
A 2x2 ablation shows repeated shared access enables grokking while addressable memory (not recurrence) enables edit propagation in transformer variants on synthetic KG QA.
-
Titans: Learning to Memorize at Test Time
Titans combine attention for current context with a learnable neural memory for long-term history, achieving better performance and scaling to over 2M-token contexts on language, reasoning, genomics, and time-series tasks.
-
Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback
Editable bounded memory slots with lifecycle control plus sparse fallback cover overwrite and no-signal long-context cases that pure fixed-state or pure sparse methods fail under controlled conditions.
-
More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval
Scaling a model's own per-layer feed-forward writes shows signed, task-dependent retrieval roles, and the sign of the write–gradient dot product identifies suppressors from amplifiers.
-
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
SMF adds KV memory layers and sparsely updates only heavily-read rows, yielding +2.5pp on MedMCQA with near-zero drift on WikiText and TriviaQA probes versus larger gains but clear forgetting from LoRA and full finetuning.
-
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
SMF improves MedMCQA accuracy by 2.5 points while keeping WikiText perplexity and TriviaQA accuracy within 1 point of the base model, outperforming LoRA and full finetuning on forgetting metrics.
-
Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation
A minimalist retrieval-and-generation framework using turn isolation and query-driven pruning outperforms complex memory systems by directly addressing signal sparsity and dual-level redundancy in dialogues.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.