REVIEW 6 cited by
Memformer: A Memory-Augmented Transformer for Sequence Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformers have reached remarkable success in sequence modeling. However, these models have efficiency issues as they need to store all the history token-level representations as memory. We present Memformer, an efficient neural network for sequence modeling, that utilizes an external dynamic memory to encode and retrieve past information. Our model achieves linear time complexity and constant memory space complexity when processing long sequences. We also propose a new optimization scheme, memory replay back-propagation (MRBP), which promotes long-range back-propagation through time with a significantly reduced memory requirement. Experimental results show that Memformer has achieved comparable performance compared to the baselines by using 8.1x less memory space and 3.2x faster on inference. Analysis of the attention pattern shows that our external memory slots can encode and retain important information through timesteps.
Forward citations
Cited by 6 Pith papers
-
Vision encoders should be image size agnostic and task driven
A position paper proposing task-driven, image-size-agnostic vision encoders, demonstrated by an iterative transformer with top-down multi-zoom patches and a GRPO-trained gaze policy on ImageNet.
-
MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.
-
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
A frozen LLM can process very long contexts by recurrently compressing them with a Perceiver and injecting the compressed memory through gated cross-attention, with query-dependent compression boosting QA performance.
-
Episodic memory in AI agents poses risks that should be studied and mitigated
Episodic memory in AI agents could enable both safety benefits and significant new risks, and developers should adopt principles that keep memories interpretable, user-controllable, detachable, and not editable by the...
-
Quantum-Enhanced Optimization by Warm Starts
Quantum-generated QAOA samples, used as warm starts for classical heuristics, yield faster Max-Cut and MIS solving, including on quantum hardware.
-
Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.
Discussion (0). Continue with ORCID to comment.