REVIEW 3 cited by
The Exponential Capacity of Dense Associative Memories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(\alpha N)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $\alpha_1$, and lower bounds for the maximum of the load $\alpha$ for which all patterns can be retrieved, $\alpha_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.
Forward citations
Cited by 3 Pith papers
-
Extending LLM Context via Associative Recurrent Memory
ARMT-augmented 1B-class LLMs, trained with continued pretraining, synthetic long data, curriculum, and selective memory layers, keep in-window quality while generalizing past 32k–65k tokens at constant memory and ~30%...
-
Yet another exponential Hopfield model
An exponential Hopfield model with binary neurons, defined by exponentials of quadratic losses, stores exponentially many patterns and retains exponential capacity under noise, with a quantified basin threshold.
-
Hopfield Networks as Models of Emergent Function in Biology
A review article that explains Hopfield network mathematics and interprets the dynamics as signal retrieval, subspace projection, and energy landscape descent, then surveys applications to cell fate, self-assembly, an...
Discussion (0). Continue with ORCID to comment.