Pith. sign in

REVIEW 2 major objections 5 minor 54 references

Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

T0 review · 2 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A classical Hopfield network can memorize an entire graph isomorphism class from a small random sample, because gradient descent on energy flow favors smallest-norm parameters, which are the invariant ones.

desk verdict A genuinely new invariant-subspace construction for Hopfield memorization of graph orbits, with a sample-complexity theorem whose MEF half has a fixable proof gap. read the letter →

arxiv 2512.14338 v3 pith:3OC5LOXA submitted 2025-12-16 cs.LG

Michael Murray , Tenzin Chan , Kedar Karhadker , Christopher J. Hillar This is my paper
classification cs.LG MSC 68T0768Q3205C60
keywords Hopfieldnetworksimplicitbiasgraphisomorphismclassesinvariantsubspacesamplecomplexityminimum-normsolutionenergyfloworbits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Hopfield networks—the simplest associative memories—can learn the full isomorphism class of a graph (all graphs related by relabeling vertices) from a tiny random subset of that class. The mechanism is implicit bias: minimizing the energy-flow loss by gradient descent drives the parameters toward the minimum-norm solution, and minimum-norm solutions that memorize a group orbit must lie in a three-dimensional invariant subspace. The authors prove a polynomial sample-complexity bound: roughly O~(n||θ*||²m/ε²) random samples suffice for both the hard-margin SVM solution and gradient descent to memorize new samples with probability at least 1−ε; for k-cliques this becomes O~(v³k²). The same construction shows every graph isomorphism class is strictly memorizable by a 3-parameter network, and experiments show learned weights concentrating toward that invariant subspace as sample size grows. If correct, this explains few-shot-to-orbit generalization as a consequence of norm-efficiency, not architecture.

What carries the argument

The energy-flow loss L(ω;S) = Σ exp(−⟨uⱼ(x),ω⟩), reparameterized so each strict-memorization constraint becomes a linear inequality ⟨uⱼ(x),ω⟩ ≥ 1. Minimizing L with gradient descent is directionally biased toward the hard-margin support-vector-machine solution HSVM(S) = argmin ||ω|| subject to these constraints. The invariant subspace Ψ(Qₙ) of parameters preserving edge adjacency is the image of a 3-parameter linear map F(β₁,β₂,β₃) assigning one weight to adjacent edges, one to non-adjacent edges, and one constant threshold; Lemma 4.3's construction F(2,2,1−2m) memorizes all m-sparse vectors, extending to orbits via Lemma 4.1. The HSVM solution on a full orbit is provably invariant (Lemma 4.

What would settle it

Construct a strictly memorizable set of sparse vectors for which the empirical min-norm solution HSVM(S_N) is known to have norm much larger than the population min-norm solution HSVM(D), then train MEF-GD at the sample size predicted by the theorem. If the empirical HSVM solution itself fails to memorize new samples at that size, the bound is false. Alternatively, track the normalized gradient-descent iterates on a separable dataset: if they do not converge directionally to HSVM(S_N), the implicit-bias transfer fails at the root.

Watch

Extended reading notes

Core claim

The central claim is that invariance is not built in but emerges from norm minimization. Memorization of a set of binary vectors is equivalent to feasibility of a system of linear inequalities; the minimum-norm feasible point is an HSVM solution. Gradient descent on the exponential energy-flow loss converges in direction to this HSVM solution (Theorem 3.1), and the paper proves that if a dataset D is strictly memorizable with min-norm parameter θ*, then N ~ O~(n||θ*||²m/ε²) i.i.d. samples suffice for the HSVM solution and for MEF-GD to memorize a new sample with error at most ε (Theorem 3.2). For graph data, the invariant subspace for edge-adjacency-preserving permutations is exactly the thr

Load-bearing premise

The MEF half of the sample-complexity theorem assumes that gradient descent on the exponential energy-flow loss inherits a known implicit-bias convergence result—proved for logistic loss with a step-size schedule—to a fixed step size and to the population optimum, whereas the quoted theorem only yields convergence to the empirical optimum; if this transfer fails, the MEF generalization bound collapses (the HSVM half would survive).

Editorial extensions

If this is right

  • For k-cliques on v vertices, the sample-complexity bound N ~ O~(v³k²) follows from Theorem 3.2 with Lemmas 4.3–4.4; for k = cv the critical ratio to class size decays exponentially in v.
  • Any graph isomorphism class is strictly memorizable by a 3-parameter Hopfield network using F(2,2,1−2m), so no architectural invariance is needed for exact orbit storage.
  • The HSVM solution on a full isomorphism class lies exactly in the invariant subspace Ψ(Qₙ) (Lemma 4.5), so in the infinite-sample limit the learned network is exactly invariant, not merely approximately.
  • For the simplified average-HSVM surrogate on k-cliques, the sample solution approaches the invariant subspace at rate O~(v^{3/2}/√N) (Corollary 4.0.1), the paper's first quantitative emergence-of-invariance statement.
  • Because Theorem 3.2 applies to any strictly memorizable sparse set, the polynomial generalization guarantee is not specific to graphs; it covers any distribution over m-sparse binary vectors with a finite min-norm memorizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Direct repair, not paper claim: the MEF half of Theorem 3.2 can likely be fixed by applying the quoted implicit-bias theorem with the empirical optimum ω̂ = HSVM(S_N), using ||ω̂|| ≤ ||ω*|| to carry the Rademacher bound; as written the proof skips this step.
  • Consequence the paper leaves implicit: if min-norm bias is the mechanism, then the double-descent curves in the appendix should align with the distance from the learned weights to F(R³) — a measurable proxy for how invariant the current model is, and a predictor of when test accuracy next rises.
  • Testable extension: train the same MEF rule on other group-structured sets such as hypergraph orbits or all permutations of a fixed code; the unified claim predicts the same polynomial few-shot-to-orbit behavior whenever the min-norm memorizer lies in a low-dimensional invariant subspace.
  • Caveat forward: the paper's Section 5 explicitly leaves full HSVM/MEF convergence to the invariant subspace open; if that convergence is slow or absent for some classes, orbit generalization would fail despite the sample bound, since the bound covers memorizing random samples, not the whole orbit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies classical Hopfield networks trained to memorize binary vectors representing graphs, focusing on graph isomorphism classes. It identifies a three-dimensional subspace of parameters invariant under edge-adjacency-preserving permutations, gives explicit invariant parameter constructions that memorize all fixed-sparsity vectors (Lemma 4.3) and all k-cliques with small norm (Lemma 4.4), and proves that the full-orbit HSVM solution is invariant (Lemma 4.5). The main theoretical result is Theorem 3.2, a Rademacher-based polynomial sample-complexity bound for both the HSVM solution and MEF gradient descent to memorize new samples from a strictly memorizable set D; specialized to k-cliques this gives N ≈ O~(v^3 k^2) samples. The paper also proves an AHSVM surrogate result showing convergence of the sample solution to the invariant subspace, and reports extensive experiments on few-shot orbit generalization.

Significance. If the main result holds, this is a valuable contribution: it gives a rare quantitative explanation of emergent invariance in a classical associative memory, and it connects the implicit bias of MEF-GD to max-margin solutions in a Hopfield setting. Notable strengths are the explicit parameter-free constructions in Lemmas 4.3 and 4.4, the symmetrization proof of orbit invariance in Lemma 4.5, reproducible code, and a concrete falsifiable sample-complexity prediction for clique classes. The paper is also candid in Section 5 that convergence of the true HSVM/MEF solutions to the invariant subspace is not proved. However, as printed the MEF half of Theorem 3.2 relies on a convergence theorem applied to the wrong target, and Lemma 4.4 contains algebraic errors; both are localized and appear repairable, but they are load-bearing for the paper's central quantitative claims.

major comments (2)
  1. [Appendix B.1, proof of Theorem 3.2 (MEF half)] The GD iterates ω(t) in (8) minimize the empirical loss L(·;S_N), so [31, Thm. 5] yields directional convergence to the empirical optimum ω̂ = HSVM(S_N), not to the population optimum ω* = HSVM(D). The proof instead asserts convergence to ω* and then constructs ω̄(t) with norm ||ω*|| that is claimed to have margin at least 1−o(1) on S_N. This step is not valid as written. A repair appears possible: since S_N ⊆ D, ||ω̂|| ≤ ||ω*||, so the scaled empirical optimum (||ω*||/||ω̂||)ω̂ is feasible and has margin ≥1 on S_N; replacing ω* by ω̂ in the directional convergence argument should recover the stated O(√m||ω*||/log t)+ε bound. The correction is necessary because the MEF-GD guarantee is a headline consequence of Theorem 3.2.
  2. [Lemmas B.1 and 4.4] Lemma B.1, r=2 case, contains an algebraic typo: the printed term '2β1 − 3/2β1' is not the coefficient that follows from the preceding counts. Substituting β = [−5/k, 14/k², 0] into the printed formula gives a negative margin for j ∈ Φ2, contradicting Lemma 4.4. The correct coefficient is 2β1 − (5/2)β2, which yields margin 3 + 15/k − 42/k² ≥ 1 for k ≥ 5. Separately, the norm estimate in Lemma 4.4 has wrong constants: the first term should be 196/k^4 v^4, not 14/k^4 v^4, and the second should be 50/k² v^3, not 225/k² v^3; the subsequent inequality '≤ 2(14c^{-2})^2 v' is also not justified as written. Since Lemma 4.4 underpins the k-clique sample-complexity corollary, these errors need to be corrected even though the qualitative O(√v) norm bound is recoverable.
minor comments (5)
  1. [Abstract and Section 3] The abstract states sample complexity as N = Ω~(n||θ*||²mε^{-2}); for an upper bound this should be O~(·). The same convention appears in Section 2's experimental scaling discussion. Please use the correct asymptotic direction.
  2. [Theorem 3.1] The statement says 'let ω* = HSVM(S_N)' but S_N is not defined in Theorem 3.1; it should likely be HSVM(S) or 'ω̂ = HSVM(S)'. The proof also refers to both [31, Thm. 3] and [31, Thm. 5]; the references should be reconciled.
  3. [Theorem 3.2] The theorem does not quantify the step-size condition needed for the quoted convergence result from [31, Thm. 5]. Theorem 3.1 only asserts existence of such an η; Theorem 3.2 should state the relevant condition or at least link to Theorem 3.1.
  4. [Corollary 4.0.1] The sample is written as '(x_i)_{i∈[n]}' but should be '(x_i)_{i∈[N]}' with the sample size N appearing in the bound. Also, the corollary number '4.0.1' is nonstandard.
  5. [Section 5 and abstract] The abstract says 'across multiple learning rules, parameters converge toward the invariant subspace as sample sizes grow,' which reads as a proven statement. Section 5 correctly states that convergence of the full HSVM/MEF solutions to the invariant subspace is not proved, and the theoretical result (Lemma 4.6, Corollary 4.0.1) covers only the AHSVM surrogate. The abstract and introduction should be calibrated to this acknowledged limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central constructions and sample-complexity bounds are self-contained; the MEF proof error is a correctness gap, not an input-output circularity.

full rationale

The derivation chain is self-contained and does not reduce to its inputs by construction. Theorem 3.2 bounds generalization using Rademacher/vector-contraction arguments and the external Soudry et al. convergence theorem [31]; the norm ||ω*|| enters as a hypothesis-class radius, not as a fitted parameter, and the HSVM half follows directly from feasibility. Lemmas 4.2—4.4 give explicit analytic parameters (β=[2,2,1−2m] and β=[−5/k,14/k^2,0]) and verify the required energy-gap inequalities directly, so the claim that graph isomorphism classes are memorizable is proved rather than assumed. Lemma 4.5 is a symmetrization argument with no hidden dependence on the target claim, and Lemma 4.6/Corollary 4.0.1 use standard concentration of the AHSVM surrogate; Section 5 explicitly concedes that full HSVM/MEF convergence to the invariant subspace is not proved, so that part is honestly empirical. The self-citations [8,9] motivate the problem and provide the MEF objective and clique baselines, but no lemma or theorem's conclusion is imported from them; the paper's own proofs carry the weight. The main issue in the MEF half of Theorem 3.2 is a localized proof error—GD on the empirical loss converges to HSVM(S_N), not HSVM(D)—which is repairable and concerns correctness, not a circular identification of a fitted input with a predicted output.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its hand-chosen 3-parameter beta constructions and the AHSVM surrogate are the main extra assumptions; the beta values are analytic rather than fitted, while the AHSVM-to-HSVM transfer is an acknowledged gap.

free parameters (3)
  • beta in Lemma 4.3 = [2, 2, 1 - 2m]
    Hand-chosen invariant parameters that give margin 1 for every m-sparse binary vector; used to prove every isomorphism class is memorizable. Not fitted to data but chosen by construction.
  • beta in Lemma 4.4 = [-5/k, 14/k^2, 0]
    Hand-chosen coefficients tailored to k-clique geometry to give a low-norm invariant memorizer with ||theta||^2 = O(v). Analytic construction, but ad hoc to make the clique margin constraints work.
  • empirical sample-complexity exponent p = p ~ 1 for cliques, p ~ 2 for Paley graphs
    Obtained by log-log linear regression of s50 vs v in Fig. 2. Illustrative and not used in any theorem, but it is a number fit to experimental data.
assumptions (4)
  • domain assumption Asynchronous Hopfield dynamics converges to a fixed point, and an energy gap to all Hamming neighbors is sufficient for strict memorization.
    Used throughout Sections 2 and 4 to define memorization and to prove Lemma 4.1; standard for this model.
  • standard math Soudry et al.'s implicit-bias theorem [31] transfers to the exponential energy-flow loss with a fixed step size and yields O(log log t / log t) directional convergence.
    Central to Theorem 3.1 and the MEF half of Theorem 3.2. The paper cites [31] but does not prove the transfer to the exponential loss; the proof currently applies it to the wrong target (population vs empirical HSVM solution).
  • standard math Rademacher complexity bounds ([39,40]) and the Hilbert-space Hoeffding inequality ([51]) hold as stated.
    Used in the proof of Theorem 3.2 and Lemma 4.6; accepted external results.
  • domain assumption The AHSVM single-averaged-constraint problem is representative of the HSVM/MEF behavior for k-cliques.
    Section 4.3 says AHSVM is a 'related but simpler problem' and leaves the HSVM analog to future work; Corollary 4.0.1 only concerns AHSVM, so any inference about actual learned solutions relies on this unproved transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits." pith.science (2026). https://pith.science/paper/3OC5LOXA

@misc{pith2026251214338,
  author       = {Pith},
  title        = {Pith review of: Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OC5LOXA}},
  note         = {Machine review of arXiv:2512.14338}
}
read the original abstract

Many learning problems involve symmetries, and while invariance can be built into neural architectures, it can also emerge implicitly when training on group-structured data. We study this phenomenon in classical Hopfield networks and show they can infer the full isomorphism class of a graph from a small random sample. Our results reveal that: (i) graph isomorphism classes can be represented within a three-dimensional invariant subspace, (ii) using gradient descent to minimize energy flow (MEF) has an implicit bias toward norm-efficient solutions, which underpins a polynomial sample complexity bound for learning isomorphism classes, and (iii) across multiple learning rules, parameters converge toward the invariant subspace as sample sizes grow. Together, these findings highlight a unifying mechanism for generalization in Hopfield networks: a bias toward norm efficiency in learning drives the emergence of approximate invariance under group-structured data.

Figures

Figures reproduced from arXiv: 2512.14338 by the authors.

Figure 1
Figure 1. Test accuracy vs. training sample size for isomorphism classes at two scales. Top row: v = 8 (isomorphism class sizes: bipartite 35, Paley 2520). Bottom row: v = 20 (for reference class size for bipartite is 92,378). Curves show mean and min-max over 10 trials. Networks are trained with Perceptron, Delta (MSE), and MEF learning rules [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. estimates and compares the specific polynomial sample complexity of learning k-cliques versus Paley graphs. We do this in order to highlight that different isomorphism classes may be harder or easier to learn depending on their connectivity structure. For each graph size v we record s50, which we define as the smallest training sample size for which MEF attains ≥ 50% average test accuracy on test samples of size 100… view at source ↗
Figure 3
Figure 3. shows histograms of weights concentrating about two values for several classes (chain, bipartite, Johnson) learned with different numbers of samples. In [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: shows the weight matrices for MEF and Delta on clique and Paley graph data. We observe that the Delta rule also returns solutions which approach the invariant space as the sample size N increases [PITH_FULL_IMAGE:figures/full_fig_p027_4.png]
Figure 5
Figure 5. Figure 5: Generalization solves the Hidden Clique Problem. a) Generalization and denoising accuracy (exact / average bits) are plotted as a function of number of 50-clique samples (in 100-vertex graphs; n = 4950 bit networks) for MEF-trained HNs. Accuracy for generalization was …
Figure 6
Figure 6. Figure 6: DAM models trained on cliques. We compare the generalization performance between DAMs and MEF￾trained HNs for a) the 6-clique problem on graphs with v = 12 vertices and b) the 16-clique problem on graphs with v = 32 vertices (496-bit networks). The Train accuracy is th…
Figure 7
Figure 7. Figure 7: Generalization for other graph classes. Accuracy of MEF-trained HN networks generalizing as a function of number of training samples for: a) Chain graph with v = 32, k = 16 (496-bit), b) Cycle graph with v = 32, k = 16 (496-bit), c) Bipartite graph with v = 32, k = 16 …
Figure 8
Figure 8. Figure 8: Generalization for Circulant graphs. Accuracy of MEF-trained HN networks generalizing as a function of number of training samples for: a) Circulant graph with v = 32, jump sequence [2, 4], b) For each type of learned parameter (weights indexed by adjacent/non-adjacent …
Figure 9
Figure 9. Figure 9: Double Descent Phenomenon. a) We trained Hopfield networks using MEF with increasing numbers of training samples for the case of learning cliques with v = 32, k = 8 and find that average test bit error (on 10000 random cliques) decreases with number of training samples…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 4 linked inside Pith

  1. [1]

    Hopfield

    John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982

    1982

  2. [2]

    The perceptron: a probabilistic model for information storage and organization in the brain

    Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386–408, 1958

    1958

  3. [3]

    Non-holographic associative memory.Nature, 222(5197):960–962, 1969

    David J Willshaw, Oliver P Buneman, and HC Longuet-Higgins. Non-holographic associative memory.Nature, 222(5197):960–962, 1969

    1969

  4. [4]

    Learning patterns and pattern sequences by self-organizing nets of threshold elements.IEEE Transactions on computers, 100(11):1197–1206, 1972

    Shun-ichi Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements.IEEE Transactions on computers, 100(11):1197–1206, 1972

    1972

  5. [5]

    The existence of persistent states in the brain.Mathematical biosciences, 19(1-2):101–120, 1974

    William A Little. The existence of persistent states in the brain.Mathematical biosciences, 19(1-2):101–120, 1974

    1974

  6. [6]

    Exactly soluble model of a spin glass.Soviet Journal of Low Temperature Physics, 3(6):378–383, 1977

    Leonid A Pastur and Alexander L Figotin. Exactly soluble model of a spin glass.Soviet Journal of Low Temperature Physics, 3(6):378–383, 1977

    1977

  7. [7]

    A logical calculus of the ideas immanent in nervous activity.The bulletin of mathematical biophysics, 5:115–133, 1943

    Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity.The bulletin of mathematical biophysics, 5:115–133, 1943

    1943

  8. [8]

    Robust exponential memory in Hopfield networks.The Journal of Mathematical Neuroscience, 8:1–20, 2018

    Christopher J Hillar and Ngoc M Tran. Robust exponential memory in Hopfield networks.The Journal of Mathematical Neuroscience, 8:1–20, 2018

    2018

Show all 54 references
  1. [9]

    Hidden hypergraphs, error-correcting codes, and critical learning in Hopfield networks.Entropy, 23(11), 2021

    Christopher Hillar, Tenzin Chan, Rachel Taubman, and David Rolnick. Hidden hypergraphs, error-correcting codes, and critical learning in Hopfield networks.Entropy, 23(11), 2021

    2021

  2. [10]

    Efficient and optimal binary Hopfield associative memory storage using minimum probability flow

    Christopher Hillar, Jascha Sohl-Dickstein, and Kilian Koepsell. Efficient and optimal binary Hopfield associative memory storage using minimum probability flow. In4th Neural Information Processing Systems (NeurIPS) Workshop on Discrete Optimization in Machine Learning (DISCML)...

    2012

  3. [11]

    Finding hidden cliques in linear time with high probability

    Yael Dekel, Ori Gurel-Gurevich, and Yuval Peres. Finding hidden cliques in linear time with high probability. Combinatorics, Probability and Computing, 23(1):29–49, 2014

    2014

  4. [12]

    Storing infinite numbers of patterns in a spin-glass model of neural networks.Physical Review Letters, 55(14):1530, 1985

    Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Storing infinite numbers of patterns in a spin-glass model of neural networks.Physical Review Letters, 55(14):1530, 1985

    1985

  5. [13]

    The space of interactions in neural network models.Journal of Physics A: Mathematical and General, 21(1):257, 1988

    Elizabeth Gardner. The space of interactions in neural network models.Journal of Physics A: Mathematical and General, 21(1):257, 1988

    1988

  6. [14]

    Storage capacity of memory networks with binary couplings.Journal de Physique, 50(20):3057–3066, 1989

    Werner Krauth and Marc Mézard. Storage capacity of memory networks with binary couplings.Journal de Physique, 50(20):3057–3066, 1989

    1989

  7. [15]

    McEliece, Edward C

    Robert J. McEliece, Edward C. Posner, Eugene R. Rodemich, and Santosh S. Venkatesh. The capacity of the Hopfield associative memory.IEEE Trans. Inform. Theory, 33(4):461–482, 1987

    1987

  8. [16]

    Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition.IEEE Transactions on Electronic Computers, 14(14):326–334, 1965

    TM Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition.IEEE Transactions on Electronic Computers, 14(14):326–334, 1965

    1965

  9. [17]

    M. V . Tsodyks and M. V . Feigel’man. The enhanced storage capacity in neural networks with low activity level. Europhysics Letters, 6(2):101–105, May 1988

    1988

  10. [18]

    Characteristics of sparsely encoded associative memory.Neural networks, 2(6):451–457, 1989

    Shun-Ichi Amari. Characteristics of sparsely encoded associative memory.Neural networks, 2(6):451–457, 1989

    1989

  11. [19]

    Dense associative memory for pattern recognition.Advances in Neural Information Processing Systems (NeurIPS), 29, 2016

    Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition.Advances in Neural Information Processing Systems (NeurIPS), 29, 2016

    2016

  12. [20]

    Capacities of multiconnected memory models.Journal de Physique, 49(3):389–395, 1988

    David Horn and Marius Usher. Capacities of multiconnected memory models.Journal de Physique, 49(3):389–395, 1988

    1988

  13. [21]

    Hopfield networks is all you need

    Hubert Ramsauer, Bernhard Schäfl, David Hopkins, Michael Widrich, Thomas Adler, Lisa Gruber, Markus Holzleitner, Milena Pavlovi´c, Geir Kjetil Sandve, Vibeke Greiff, David P Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is...

    2020

  14. [22]

    On a model of associative memory with huge storage capacity.Journal of Statistical Physics, 168(2):288–299, 2017

    Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity.Journal of Statistical Physics, 168(2):288–299, 2017

    2017

  15. [23]

    Generalization in a Hopfield network.Journal de Physique, 51(21):2421–2430, 1990

    JF Fontanari. Generalization in a Hopfield network.Journal de Physique, 51(21):2421–2430, 1990

    1990

  16. [24]

    Storage and learning phase transitions in the random-features Hopfield model.arXiv preprint arXiv:2303.16880, 2023

    Matteo Negri, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, and Enrico Malatesta. Storage and learning phase transitions in the random-features Hopfield model.arXiv preprint arXiv:2303.16880, 2023. 10 Implicit Bias and Invariance: How Hopfield Networks Efficiently Lear...

    2023 arXiv

  17. [25]

    Malatesta, and Matteo Negri

    Silvio Kalaj, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, Enrico M. Malatesta, and Matteo Negri. Random features Hopfield networks generalize retrieval to previously unseen examples.arXiv preprint arXiv:2407.05658, 2024

    2024 arXiv

  18. [26]

    Shuxiao Chen, Edgar Dobriban, and Jane H. Lee. A group-theoretic framework for data augmentation.Journal of Machine Learning Research, 21(245):1–71, 2020

    2020

  19. [27]

    A kernel theory of modern data augmentation

    Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Christopher De Sa, and Christopher Ré. A kernel theory of modern data augmentation. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97 ofPMLR, pages 1528–1537, 2019

    2019

  20. [28]

    Provably strict generalisation benefit for invariance in kernel ridge regression

    Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for invariance in kernel ridge regression. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  21. [29]

    Gerken and Pan Kessel

    Jan E. Gerken and Pan Kessel. Emergent equivariance in deep ensembles. InProceedings of the 41st International Conference on Machine Learning (ICML), volume 235 ofPMLR, pages 15438–15465, 2024

    2024

  22. [30]

    Klindt, Anne E

    Jakob Marthaler, Oleg Makarov, Tobias Rupprecht, David A. Klindt, Anne E. Urai, Ladislau Bölöni, Matthias Bethge, and Alexander S. Ecker. Equivariant neural tangent kernels.arXiv preprint arXiv:2406.06504, 2024

    2024 arXiv

  23. [31]

    The implicit bias of gradient descent on separable data

    Daniel Soudry, Elad Hoffer, and Nathan Srebro. The implicit bias of gradient descent on separable data. In International Conference on Learning Representations, 2018

    2018

  24. [32]

    Risk and parameter convergence of logistic regression.arXiv preprint arXiv:1803.07300, 2018

    Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression.arXiv preprint arXiv:1803.07300, 2018

    2018 arXiv

  25. [33]

    Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry

    Mor Shpigel Nacson, Jason D. Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), vo...

    2019

  26. [34]

    J. Bruck. On the convergence properties of the Hopfield model.Proceedings of the IEEE, 78(10):1579–1585, 1990

    1990

  27. [35]

    Addison-Wesley, 1991

    JA Hertz, A Krogh, and RG Palmer.Introduction to the theory of neural computation. Addison-Wesley, 1991

    1991

  28. [36]

    Pavel Tolmachev and Jonathan H. Manton. New insights on learning rules for Hopfield networks: Memory and objective function minimisation. In2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020

    2020

  29. [37]

    On the limited memory bfgs method for large scale optimization.Mathematical programming, 45(1):503–528, 1989

    Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization.Mathematical programming, 45(1):503–528, 1989

    1989

  30. [38]

    Battaglino, and Michael R

    Jascha Sohl-Dickstein, Peter B. Battaglino, and Michael R. DeWeese. New method for parameter estimation in probabilistic models: Minimum probability flow.Phys. Rev. Lett., 107:220601, Nov 2011

    2011

  31. [39]

    A vector-contraction inequality for rademacher complexities

    Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles, editors,Algorithmic Learning Theory, pages 3–17, Cham, 2016. Springer International Publishing

    2016

  32. [40]

    The MIT Press, 2nd edition, 2018

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar.Foundations of Machine Learning. The MIT Press, 2nd edition, 2018

    2018

  33. [41]

    Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks

    NetworkX Developers. Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks

  34. [42]

    Cambridge Studies in Advanced Mathematics

    Béla Bollobás.Explicit Constructions, page 348–382. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2001

    2001

  35. [43]

    Hebb.The Organization of Behavior: A Neuropsychological Theory

    Donald O. Hebb.The Organization of Behavior: A Neuropsychological Theory. Wiley, New York, 1949

    1949

  36. [44]

    Associatron-a model of associative memory.IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3):380–388, 1972

    Kaoru Nakano. Associatron-a model of associative memory.IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3):380–388, 1972

    1972

  37. [45]

    Adaptive switching circuits

    ME Hoff and B Widrow. Adaptive switching circuits. In1960 IRE WESCON Convention Record, Part 4, pages 96–104. IRE New York, NY , USA, 1960

    1960

  38. [46]

    A theory of Pavlovian conditioning.Classical conditioning, Current research and theory, 2:64–69, 1972

    Robert A Rescorla. A theory of Pavlovian conditioning.Classical conditioning, Current research and theory, 2:64–69, 1972

    1972

  39. [47]

    Storage and retrieval capacities of associative memories

    Lionel Personnaz, Isabelle Guyon, and Gérard Dreyfus. Storage and retrieval capacities of associative memories. Physical Review A, 32(6):4292, 1985. 11 Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph OrbitsA PREPRINT

    1985

  40. [48]

    Collective computational properties of neural networks: New learning mechanisms.Physical Review A, 34(5):4217–4228, 1986

    Laurent Personnaz, Isabelle Guyon, and Georges Dreyfus. Collective computational properties of neural networks: New learning mechanisms.Physical Review A, 34(5):4217–4228, 1986

    1986

  41. [49]

    Increasing the capacity of a hopfield network without sacrificing functionality

    Amos Storkey. Increasing the capacity of a hopfield network without sacrificing functionality. In Wulfram Gerstner, Alain Germond, Martin Hasler, and Jean-Daniel Nicoud, editors,Artificial Neural Networks — ICANN’97, pages 451–456, Berlin, Heidelberg, 1997. Springer Berlin Heidelberg

    1997

  42. [50]

    Storkey and R

    A.J. Storkey and R. Valabregue. The basins of attraction of a new hopfield learning rule.Neural Networks, 12(6):869–876, 1999

    1999

  43. [51]

    Optimum bounds for the distributions of martingales in banach spaces.The Annals of Probability, 22(4):1679–1706, 1994

    Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces.The Annals of Probability, 22(4):1679–1706, 1994. Appendix A Background A.1 Energy gap for binary vectors a hamming distance one apart As discussed in Section 2, memorization is equivalent of a...

    1994

  44. [52]

    Note, as θ∈Ψ(Q n) this implies bi =b π(i) =b j

    For any i, j∈[n] there exists a π∈Π Q n such that π(i) =j . Note, as θ∈Ψ(Q n) this implies bi =b π(i) =b j

  45. [53]

    Note, and again asθ∈Ψ(Q n), this impliesw ij =w π(i)π(j) =w ab

    For any i, j, a, b∈[n]satisfying i∼j and a∼b , or i≁j and a≁b , there exists a π∈Π Q n such that π(i) =aandπ(j) =b. Note, and again asθ∈Ψ(Q n), this impliesw ij =w π(i)π(j) =w ab. In all that follows, for l∈[8] let νl ∈[v] . To prove the first statement, let i, j∈[n] and suppo...

  46. [54]

    double ascent/descent

    Ifk=cvfor some constantc∈(0,1], then there exists a constantC >0such that∥θ∥ 2 ≤Cv. Proof. For the proof of the first statement, from Lemma B.1 there are three cases we need to check. First, if j∈Φ 0(x) then yj(x)z(x)T θj = β2 2 k2 + β2 2 k+β 3 = 7 1 + 1 k ≥1. Second, ifj∈Φ 1(...

    2000

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.