Pith. sign in

REVIEW 1 major objections 5 minor 1 cited by

In-context denoising with one-layer transformers: connections between attention and associative memory retrieval

T0 review · 1 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For spherical token distributions, a single softmax attention layer is a Bayes-optimal denoiser in the infinite-context limit.

desk verdict Clean theoretical result on attention as a Bayes-optimal denoiser, with honest empirics; the soft spots are real but minor and mostly acknowledged. read the letter →

arxiv 2502.05164 v2 pith:LHIFZPQB submitted 2025-02-07 cs.LG cond-mat.dis-nn

Matthew Smart , Alberto Bietti , Anirvan M. Sengupta This is my paper · ORCID
classification cs.LGcond-mat.dis-nn MSC 68T0762F1560F15
keywords in-contextdenoisinglearningsoftmaxattentionBayesoptimaldenoiserdenseassociativememorymodernHopfieldnetworksone-layertransformerGaussian
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In-context denoising is an unsupervised version of in-context learning: a prompt supplies many clean samples from an unknown distribution and one corrupted query, and the model must reconstruct the clean query. The paper's central claim is that a single softmax attention layer can perform this denoising optimally when the context length is large. For any task distribution whose samples lie on a sphere centered at the origin, the attention output with identity value weights and key scale $1/\sigma_Z^2$ converges almost surely to the Bayes-optimal denoiser—the best possible reconstruction given full knowledge of the data distribution and the Gaussian noise. The same layer, trained from random weights on linear subspaces, spheres, and Gaussian mixtures, settles on nearly identical scaled-identity weights, and its update is exactly one gradient-descent step on a context-dependent associative-memory energy.

What carries the argument

The load-bearing object is the softmax attention map with weights $W_{PV}=I_n$ and $W_{KQ}=\sigma_Z^{-2}I_n$, i.e. $F=\sum_t X_t e^{\langle X_t,\tilde x\rangle/\sigma_Z^2}/\sum_t e^{\langle X_t,\tilde x\rangle/\sigma_Z^2}$. This formula is simultaneously the Bayes posterior mean (because sphere support makes norm-dependent Gaussian factors constant), a softmax-weighted average of context tokens, and one step of gradient descent on the energy $E(X_{1:L},s)=\frac{1}{2\alpha}\|s\|^2-\frac{1}{\beta}\log\sum_{t=1}^L e^{\beta X_t^\top s}$ initialized at $s=\tilde x$. The energy is the dense-associative-memory (modern Hopfield) energy adapted with a Lagrange multiplier term, and the identity of these three descriptions is what carries the argument.

What would settle it

Take a denoising task whose context tokens are drawn from the union of two concentric spheres of different radii, run the scaled-identity softmax attention with $W_{PV}=I$ and $W_{KQ}=\sigma_Z^{-2}I$ at large $L$, and compare its mean squared error to the exact posterior mean computed by numerical integration; if the attention output does not approach the posterior mean as $L$ grows, the sphere-support assumption is necessary, not cosmetic.

Watch

Extended reading notes

Core claim

The paper establishes Theorem 3.1: for any task distribution $\mathcal{D}$ such that the support of each $p_X$ lies on a sphere centered at the origin with radius $R$, the function $$F((\{X_t\}_{t=1}^L,\tilde x),\$\theta$^*) = \frac{\sum_{t=1}^L X_t $e^{{\langle X_t,\tilde x\rangle/\sigma_Z^2}}$}{\sum_{t=1}^L $e^{{\langle X_t,\tilde x\rangle/\sigma_Z^2}}$}$$ with $\theta^*=(W_{PV}=I_n,\,W_{KQ}=\sigma_Z^{-2}I_n)$ converges almost surely to the Bayes-optimal denoiser $f_{\mathrm{opt}}(\tilde x)=\mathbb{E}[X\mid\tilde x]$ as $L\to\infty$. The reason this identity works is that on a sphere the quadratic term in the Gaussian posterior is constant, so the posterior mean collapses to a softmax-weighted average of the clean context tokens. Empirically, one-layer transformers trained on the three task families learn weights whose diagonal scales match this prediction. The paper then shows that this single attention step equals one gradient-descent step on the dense-associative-memory energy $E(X_{1:L},s)=\frac{1}{2\alpha}\|s\|^2-\frac{1}{\beta}\log\sum_t e^{\beta X_t^\top s}$, with the query as the initial state—so the trained layer is a deliberate first step, not an iteration to a stored memory or a spurious fixed point.

Load-bearing premise

The entire optimality result rests on each task's tokens all lying on one sphere centered at the origin, with the sphere's radius allowed to vary from task to task; if token norms vary within a task, the quadratic terms in the Gaussian noise kernel do not cancel and the softmax attention output is no longer the exact Bayes-optimal denoiser.

Editorial extensions

If this is right

  • As $L\to\infty$, any spherical-manifold denoising task is solved exactly by a single attention head, independent of the sphere's radius or embedding dimension.
  • Randomly initialized one-layer transformers reach essentially these optimal weights after standard training, so unsupervised in-context denoising requires no bespoke architecture or pretraining objective.
  • The trained update is one gradient step on a context-dependent associative-memory energy; further iteration can pull the estimate toward query-independent fixed points and hurt denoising.
  • Softmax attention with small key scale reduces to linear attention, so the linear-manifold task can be solved by either attention form in the appropriate limit.
  • Because the denoiser averages many context tokens rather than retrieving one, the result gives a concrete case where associative memory networks solve a task beyond exact pattern retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its sphere-support condition is naturally supplied by pre-norm layers such as RMSNorm, which project activations onto a sphere up to learned diagonal multipliers; testing the theorem on internal transformer blocks with that normalization would be a direct transfer experiment.
  • A simple extension would add per-token norm terms $e^{(\langle X_t,\tilde x\rangle-\|X_t\|^2/2)/\sigma_Z^2}$ to the kernel, which should restore Bayes optimality for tasks with variable token norms, such as Gaussian mixtures with unequal component variances.
  • The one-step versus iterated distinction suggests that deeper transformers with tied weights could implement several gradient steps on the same energy landscape, and that the utility of iteration depends on whether the energy has query-dependent minima to stop at.
  • The paper's convergence bounds depend on the intrinsic dimension of the tokens rather than the ambient dimension, so the number of context tokens needed for near-optimal denoising may stay manageable even in high-dimensional embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper introduces in-context denoising, an unsupervised in-context learning task in which a prompt consists of L clean tokens sampled from a task-specific distribution plus a Gaussian-corrupted query, and the model must reconstruct the clean query. The authors study three token families (linear subspaces, spheres, and Gaussian mixtures with fixed-norm centers), derive Bayes-optimal denoisers for each, and show that a one-layer softmax attention transformer with scaled identity weights can realize these denoisers in the infinite-context limit. Theorem 3.1 states that for any token distribution supported on a sphere centered at the origin, the softmax attention output converges almost surely to the Bayes-optimal posterior mean as L tends to infinity; Propositions 5 and 6 provide finite-sample concentration bounds. Experiments show that training a one-layer attention model from random initialization on finite data learns approximately diagonal weights and approaches the Bayes-optimal MSE, and the learned update is represented as one gradient step on an associative-memory energy function.

Significance. If Theorem 3.1 holds, the paper provides a clean and explicit connection between attention and optimal Bayesian denoising, going beyond the existing 'attention as Hopfield retrieval' analogy by showing that the same one-layer mechanism can perform denoising rather than exact retrieval. The construction of the optimal weights is explicit and parameter-free, and the finite-sample concentration bounds (Propositions 5-6) plus the publicly available code strengthen the empirical component. The paper is honest about scope: the optimality guarantee requires constant-norm (sphere-supported) token distributions, and the training-convergence claims are empirical rather than proven. The energy-function interpretation in Section 4 is constructed so that one gradient step equals the attention update; this is a legitimate mechanistic interpretation rather than a predictive theory, and the paper does not overclaim it.

major comments (1)
  1. [Section 3.1 / Appendix D.3 (Theorem 3.1)] The theorem states that the softmax attention ratio converges almost surely to the Bayes-optimal denoiser for all x̃ ∈ R^n, but the proof in Appendix D.3 only establishes pointwise almost-sure convergence for each fixed x̃ via the strong law of large numbers. The null set can depend on x̃, so the 'for all' claim is stronger than the proof as written. The gap is repairable: the map in Eq. (13) has a derivative with operator norm bounded by R^2/σ_Z^2, uniformly in L (it is the weighted covariance of the bounded context tokens), so pointwise almost-sure convergence on a countable dense set together with equicontinuity gives uniform convergence on compact sets, hence convergence for all x̃. The authors should either add this standard uniformity argument or weaken the theorem statement to pointwise convergence.
minor comments (5)
  1. [Abstract / Section 4 (Fig. 5)] The statement that the one-step update 'yields better solutions than exact retrieval of either a context token or a spurious local minimum' is illustrated with a single 2D trajectory but not quantified; please either add a quantitative comparison or explicitly label this as a qualitative illustration.
  2. [Section 3.4 / Proposition 4] For finite σ0 in the Gaussian mixture case, the Bayes predictor in Proposition 4 contains the linear term σ0^2/(σ0^2+σ_Z^2) x̃, which the zero-variance softmax attention (α=1, β=1/σ_Z^2) cannot represent. The text notes the scaling discrepancy (β≈5.127 < 10) and discusses it via the loss landscape, but it would be clearer to state explicitly that the trained network is being compared to the σ0→0 limit rather than to the exact finite-σ0 predictor.
  3. [Section 3.2, Case 1] The statement that training 'indeed achieves this bound' is empirical; no theorem guarantees convergence of the training dynamics to the constructed weights. The paper already uses 'empirically demonstrate' in the introduction, but a sentence in the experimental sections clarifying that this is not a proven guarantee would avoid over-reading.
  4. [Appendix E, Proposition 6] In the second probabilistic inequality, the expression 'Re^{R||x̃||2/σ_Z^2}' is ambiguous: it should be typeset as R e^{R||x̃||2/σ_Z^2} to avoid confusion with the real part operator.
  5. [Section 3.1, Proposition 3.2] The informal joint limit 'L large and ϵ small with ϵ√L large' would benefit from a precise statement, e.g., L→∞ and ϵ→0 with ϵ√L→∞, to make clear why the 1/ϵ ar{X} term vanishes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 3.1 is a constructive SLLN argument under explicit sphere-support assumptions; training results are empirical and honestly scoped; Section 4's energy function is a post-hoc algebraic equivalence, not a fitted prediction.

full rationale

The paper's central claim is not circular. Theorem 3.1 (Section 3.1, proof in Appendix D.3) assumes each task distribution is supported on a sphere centered at the origin with task-dependent radius R; this makes ||x||^2 constant, so the quadratic term in exp(-||x - x_tilde||^2 / 2 sigma_Z^2) cancels and the Bayes posterior mean equals E[X e^{<X, x_tilde>/sigma_Z^2}] / E[e^{<X, x_tilde>/sigma_Z^2}]. The theorem then invokes the strong law of large numbers on the i.i.d. bounded numerator and denominator terms to show that the softmax attention ratio converges to this exact integral. This is a constructive expressivity and convergence proof, not an identity imposed by definition: the Bayes baselines in Propositions 1-4 are computed from p_X and the noise model, and the trained weights in Section 3 are measured empirically and compared to those baselines, not fed back as predictions. The energy function in Section 4 (Eq. 16) is defined post hoc so that one gradient-descent step with step size gamma = alpha algebraically reproduces the trained attention update; the paper presents this as a mechanistic equivalence rather than as a predicted consequence of the energy, so it does not constitute circular reasoning. The only self-citation (Smart & Zilman 2021) appears in the Discussion as related work and is not load-bearing. Limitations--the sphere-support assumption, the pointwise-versus-uniform a.s.-convergence wording in Theorem 3.1, and the unquantified finite-training weight error noted in Appendix E--are explicitly stated in the paper and do not undermine the self-containedness of the derivation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are introduced in the theory; the optimal attention weights are predicted from the known noise variance. The paper's guarantees rest on the restricted data geometries (linear subspaces, spheres, zero-variance Gaussian mixtures) and on i.i.d. context sampling, which are stated assumptions rather than fitted quantities.

assumptions (5)
  • domain assumption Support of each p_X is contained in a sphere centered at the origin (Theorem 3.1).
    Used in Appendix D.3 to cancel the ||x||^2 terms in the Gaussian posterior and reduce the posterior mean to a softmax over context tokens.
  • domain assumption Noise is additive isotropic Gaussian Z ~ N(0, σ_Z^2 I_n) with known variance.
    Defined in Section 2.1; the optimal softmax temperature is 1/σ_Z^2.
  • domain assumption Context tokens X_1,...,X_L are i.i.d. draws from the same distribution p_X as the target token.
    Needed for the strong law of large numbers convergence in Theorem 3.1.
  • standard math Posterior mean minimizes squared error loss.
    Proposition 1 in Section 2.3, proof in Appendix B.1.
  • standard math Strong law of large numbers and Hoeffding's inequality.
    Used in Appendix D.3 and E for convergence statements and finite-sample bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-context denoising with one-layer transformers: connections between attention and associative memory retrieval." pith.science (2026). https://pith.science/paper/LHIFZPQB

@misc{pith2026250205164,
  author       = {Pith},
  title        = {Pith review of: In-context denoising with one-layer transformers: connections between attention and associative memory retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHIFZPQB}},
  note         = {Machine review of arXiv:2502.05164}
}
read the original abstract

We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show theoretically and empirically that certain restricted denoising problems can be solved optimally even by a single-layer transformer. We demonstrate that a trained attention layer processes each denoising prompt by performing a single gradient descent update on a context-aware DAM energy landscape, where context tokens serve as associative memories and the query token acts as an initial state. This one-step update yields better solutions than exact retrieval of either a context token or a spurious local minimum, providing a concrete example of DAM networks extending beyond the standard retrieval paradigm. Overall, this work solidifies the link between associative memory and attention mechanisms first identified by Ramsauer et al., and demonstrates the relevance of associative memory models in the study of in-context learning.

Figures

Figures reproduced from arXiv: 2502.05164 by the authors.

Figure 1
Figure 1. (a) Problem formulation for a general in-context denoising task. (b) The three denoising tasks considered here include instances of linear and non-linear manifolds as well as Gaussian mixtures. In each case, the task embedding E (i) consists of a sequence of pure tokens from the data distribution p (i) X ∼ D where D denotes the task distribution, along with a single query token that has been corrupted by Gaussian no… view at source ↗
Figure 2
Figure 2. Baseline estimators for the case of random linear mani￾folds with projection operator P (i) . Manifold case. In the nonlinear manifold denoising prob￾lem, we focus on the case of lower dimensional spheres S (e.g. the circle S 1 ⊂ R 2 ). For such manifolds, the Bayes optimal answer is given by the following proposition. Proposition 3. For pX defined as in Subsection 2.2.2, with P being the orthogonal projection opera… view at source ↗
Figure 3
Figure 3. (a) Training dynamics for the studied cases using one-layer softmax attention (circles) as well as linear attention (triangles). Solid lines represent the average loss over six seeds, with the shaded area indicating the range for cases 2 and 3. For each case, the grey dashed baseline indicates the 0-predictor, and the pink line indicates the Bayes optimal predictor. All cases use a context length of L = 500, ambient… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Trained linear attention network converges to Bayes optimal estimator as context length increases (n = 16, d = 8, σ 2 0 = 2, σ2 z = 1). (b) A network trained to denoise subspaces of dimension d = 8 can accurately denoise subspaces of different dimensions presented …
Figure 5
Figure 5. Figure 5: Gradient descent denoising for the nonlinear manifold case (spheres) in n = 2 with d = 1. A context-aware dense asso￾ciative memory network E(X1:L, s) is constructed whose gradient corresponds to the Bayes optimal update (trained attention layer). Note that the density…
Figure 6
Figure 6. Figure 6: Loss landscape corresponding to Case 2 and Case 3 of [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Linear attention performance for Cases 2 and 3. Additional empirical results for the nonlinear manifolds case (left) and the Gaussian mixtures case (right). (a) Loss dynamics for randomly initialized softmax and linear attention layers. Solid lines represent the averag…
Figure 8
Figure 8. Figure 8: (a) Example transformation A used to globally alter the in-context denoising prompts. (b) Structure of the optimal attention weights for this transformed subspace-denoising task. (c,d) Empirically, we find that both linear attention and softmax attention layers are abl…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Muon in Associative Memory Learning: Training Dynamics and Scaling Laws

    cs.LG 2026-02 conditional novelty 6.0 of 10

    In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.

Reference graph

Works this paper leans on

51 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36: 0 45614--45650, 2023

    2023

  2. [2]

    What learning algorithm is in-context learning? investigations with linear models

    Aky \"u rek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    work page 2023

  3. [3]

    Albergo, M. S. and Vanden-Eijnden, E. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2209.15571

    arXiv 2023

  4. [4]

    Learning patterns and pattern sequences by self-organizing nets of threshold elements

    Amari, S.-I. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on computers, 100 0 (11): 0 1197--1206, 1972

    work page 1972

  5. [5]

    In search of dispersed memories: Generative diffusion models are associative memory networks

    Ambrogioni, L. In search of dispersed memories: Generative diffusion models are associative memory networks. Entropy, 26 0 (5), 2024. ISSN 1099-4300. doi:10.3390/e26050381. URL https://www.mdpi.com/1099-4300/26/5/381

    doi:10.3390/e26050381 2024

  6. [6]

    J., Gutfreund, H., and Sompolinsky, H

    Amit, D. J., Gutfreund, H., and Sompolinsky, H. Spin-glass models of neural networks . Physical Review A, 32 0 (2): 0 1007--1018, 1985. ISSN 10502947. doi:10.1103/PhysRevA.32.1007

    doi:10.1103/physreva.32.1007 1985

  7. [7]

    M., Castillo, I

    Boll \'e , D., Nieuwenhuizen, T. M., Castillo, I. P., and Verbeiren, T. A spherical hopfield model. Journal of Physics A: Mathematical and General, 36 0 (41): 0 10269, 2003

    work page 2003

  8. [8]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

Show all 51 references
  1. [9]

    Skyformer: Remodel self-attention with gaussian kernel and nyström method

    Chen, Y., Zeng, Q., Ji, H., and Yang, Y. Skyformer: Remodel self-attention with gaussian kernel and nyström method. Advances in Neural Information Processing Systems, 34: 0 2122--2135, 2021

    2021

  2. [10]

    M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J

    Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations...

    2021

  3. [11]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023

    Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023. URL https://arxiv.org/abs/2212.10559

    2023 arXiv

  4. [12]

    On a model of associative memory with huge storage capacity

    Demircigil, M., Heusel, J., L \"o we, M., Upgang, S., and Vermet, F. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168: 0 288--299, 2017

    2017

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2019

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2021

  7. [15]

    Fischer, K. H. and Hertz, J. A. Spin Glasses. Cambridge University Press, 1993

    1993

  8. [16]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pp.\ 30583--30598, 2022. URL https://arxiv.org/abs/2208.01066

    2022 arXiv

  9. [17]

    Sampling with flows, diffusion, and autoregressive neural networks from a spin-glass perspective

    Ghio, D., Dandi, Y., Krzakala, F., and Zdeborov \'a , L. Sampling with flows, diffusion, and autoregressive neural networks from a spin-glass perspective. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311810121, 2024

    2024

  10. [18]

    Gradshteyn, I. S. and Ryzhik, I. M. Table of Integrals, Series, and Products. Elsevier/Academic Press, Amsterdam, seventh edition, 2007

    2007

  11. [19]

    Probability inequalities for sums of bounded random variables

    Hoeffding, W. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp.\ 409--426, 1994

    1994

  12. [20]

    H., Zaki, M., and Krotov, D

    Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D. H., Zaki, M., and Krotov, D. Energy transformer. In Advances in Neural Information Processing Systems, volume 36, pp.\ 27532--27559, 2023 a

    2023

  13. [21]

    H., Zaki, M

    Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D. H., Zaki, M. J., and Krotov, D. Energy transformer. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 b . URL https://openreview.net/forum?id=MbwVNEx9KS

    2023

  14. [22]

    H., Strobelt, H., Ram, P., and Krotov, D

    Hoover, B., Chau, D. H., Strobelt, H., Ram, P., and Krotov, D. Dense associative memory through the lens of random features. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a

    2024

  15. [23]

    Hoover, B., Strobelt, H., Krotov, D., Hoffman, J., Kira, Z., and Chau, D. H. Memory in plain sight: Surveying the uncanny resemblances of associative memories and diffusion models, 2024 b . URL https://arxiv.org/abs/2309.16750

    2024 arXiv

  16. [24]

    Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79 0 (8): 0 2554--2558, 1982. ISSN 00278424. doi:10.1073/pnas.79.8.2554

    1982 doi

  17. [25]

    Y.-C., Yang, D., Wu, D., Xu, C., Chen, B.-Y., and Liu, H

    Hu, J. Y.-C., Yang, D., Wu, D., Xu, C., Chen, B.-Y., and Liu, H. On sparse modern hopfield model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, 2023

    2023

  18. [26]

    Transformers are rnns: fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  19. [27]

    A new frontier for hopfield networks

    Krotov, D. A new frontier for hopfield networks. Nature Reviews Physics, 5 0 (7): 0 366--367, 2023

    2023

  20. [28]

    and Hopfield, J

    Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. In Advances in Neural Information Processing Systems, volume 29, 2016

    2016

  21. [29]

    and Hopfield, J

    Krotov, D. and Hopfield, J. J. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=X4y_10OX-hX

    2021

  22. [30]

    Little, W. A. The existence of persistent states in the brain. Mathematical biosciences, 19 0 (1-2): 0 101--120, 1974

    1974

  23. [31]

    Probability theory i

    Lo \`e ve, M. Probability theory i. Graduate Texts in Mathematics, 1977

    1977

  24. [32]

    and M\'ezard, M

    Lucibello, C. and M\'ezard, M. Exponential capacity of dense associative memories. Phys. Rev. Lett., 132: 0 077301, Feb 2024. doi:10.1103/PhysRevLett.132.077301. URL https://link.aps.org/doi/10.1103/PhysRevLett.132.077301

    2024 doi

  25. [33]

    Universal hopfield networks: A general framework for single-shot associative memory models

    Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., and Bogacz, R. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pp.\ 15561--15583. PMLR, 2022

    2022

  26. [34]

    Associatron-a model of associative memory

    Nakano, K. Associatron-a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, 2: 0 380--388, 1972

    1972

  27. [35]

    J., Ambrogioni, L., and Krotov, D

    Pham, B., Raya, G., Negri, M., Zaki, M. J., Ambrogioni, L., and Krotov, D. Memorization to generalization: The emergence of diffusion models from associative memory. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024

    2024

  28. [36]

    P., Kopp, M

    Ramsauer, H., Sch \" a fl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D. P., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In 9th International Conference on Learning Representati...

    2021

  29. [37]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task

    Reddy, G. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aN4Jf6Cx69

    2024

  30. [38]

    and H \"u tter, J.-C

    Rigollet, P. and H \"u tter, J.-C. High-dimensional statistics. arXiv preprint arXiv:2310.19244, 2023

    2023 arXiv

  31. [39]

    Santos, S. J. R. D., Niculae, V., Mcnamee, D. C., and Martins, A. Sparse and structured hopfield networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 43368--43388. PMLR, 21--27 Jul 2024....

    2024

  32. [40]

    Shen, L., Mishra, A., and Khashabi, D. Position: Do pretrained transformers learn in-context by gradient descent? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 44712--44740. PMLR, 21--27 Ju...

    2024

  33. [41]

    and Zilman, A

    Smart, M. and Zilman, A. On the mapping between hopfield networks and restricted boltzmann machines. International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=RGJbergVIoO

    2021

  34. [42]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  35. [43]

    and Kolter, J

    Trockman, A. and Kolter, J. Z. Mimetic initialization of self-attention layers. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  36. [44]

    N., Łukasz Kaiser, and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Łukasz Kaiser, and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 2017-December, 2017

    2017

  37. [45]

    Transformers learn in-context by gradient descent

    Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp.\ 35151--35174. PMLR, 2023

    2023

  38. [46]

    Y.-C., Hsiao, T.-Y., and Liu, H

    Wu, D., Hu, J. Y.-C., Hsiao, T.-Y., and Liu, H. Uniform memory retrieval with larger capacity for modern hopfield models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 a

    2024

  39. [47]

    Y.-C., Li, W., Chen, B.-Y., and Liu, H

    Wu, D., Hu, J. Y.-C., Li, W., Chen, B.-Y., and Liu, H. Stanhop: Sparse tandem hopfield model for memory-enhanced time series prediction. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://arxiv.org/abs/2312.17346

    2024 arXiv

  40. [48]

    Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25 0 (49): 0 1--55, 2024. URL http://jmlr.org/papers/v25/23-1042.html

    2024

  41. [49]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  42. [50]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  43. [51]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.