Pith. sign in

REVIEW 10 cited by

$\text{Memory}^3$: Language Modeling with Explicit Memory

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.01178 v1 pith:TEJA2DYE submitted 2024-07-01 cs.CL cs.AIcs.LG

Hongkang Yang , Zehao Lin , Wenjin Wang , Hao Wu , Zhiyu Li , Bo Tang , Wenqiang Wei , Jinbo Wang
show 8 more authors
Zeyun Tang Shichao Song Chenyang Xi Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E
This is my paper
classification cs.CLcs.AIcs.LG
keywords memoryexplicitknowledgellmscostmodeltextinference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named $\text{Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning

    cs.CL 2025-11 unverdicted novelty 7.0 of 10

    MemSearcher trains LLMs to manage compact memory in multi-turn searches via multi-context GRPO for end-to-end RL, outperforming ReAct-style baselines with stable token counts.

  2. MemSFT: Mitigating Alignment Tax with an External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...

  3. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  4. How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Introduces Parametric Memory Law as power law for LoRA memory capacity and MemFT threshold-guided optimization for better memory fidelity.

  5. MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    MemReranker applies multi-stage distillation to Qwen3-Reranker to produce reasoning-aware rerankers that outperform baselines on memory tasks with temporal and causal constraints.

  6. Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A learnable memory cycle with adaptive retrieval, merging, and storage, trained online, improves LLM agent accuracy on HotpotQA and MemDaily for most backbones.

  7. From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs

    cs.IR 2025-04 unverdicted novelty 5.0 of 10

    The paper surveys human memory categories, maps them to LLM memory, and proposes a new three-dimension (object, form, time) categorization into eight quadrants to organize existing work and highlight open problems.

  8. MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    MemReranker applies multi-teacher pairwise distillation, BCE pointwise training, and InfoNCE contrastive learning on mixed general and memory-specific dialogue data to produce efficient rerankers that improve calibrat...

  9. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

  10. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Pith tools