REVIEW 3 cited by
The Exponential Capacity of Dense Associative Memories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The Exponential Capacity of Dense Associative Memories
read the original abstract
Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(\alpha N)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $\alpha_1$, and lower bounds for the maximum of the load $\alpha$ for which all patterns can be retrieved, $\alpha_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.
Forward citations
Cited by 3 Pith papers
-
Polyamorphism in Glassy Network Materials
A tunable microscopic model of network liquids with a liquid-liquid phase transition, analyzed via RFOT theory, predicts nanonucleation near the glass transition and links thermodynamic and kinetic anomalies when matc...
-
Extending LLM Context via Associative Recurrent Memory
ARMT-augmented 1B-class LLMs, trained with continued pretraining, synthetic long data, curriculum, and selective memory layers, keep in-window quality while generalizing past 32k–65k tokens at constant memory and ~30%...
-
Yet another exponential Hopfield model
An exponential Hopfield model with binary neurons, defined by exponentials of quadratic losses, stores exponentially many patterns and retains exponential capacity under noise, with a quantified basin threshold.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.