arxiv: 2604.17936 · v2 · submitted 2026-04-20 · ❄️ cond-mat.dis-nn · cond-mat.stat-mech

Recognition: unknown

Replica Theory of Spherical Boltzmann Machine Ensembles

Thomas Tulinski (LPENS) , Jorge Fernandez-De-Cossio-Diaz (IPHT , LPENS) , Simona Cocco (LPENS) , R\'emi Monasson

Authors on Pith no claims yet

Pith reviewed 2026-05-10 03:45 UTC · model grok-4.3

classification ❄️ cond-mat.dis-nn cond-mat.stat-mech

keywords Boltzmann machinesensemble learningreplica methodspin glasseslarge deviationsmachine learning theoryspherical models

0 comments

The pith

Replica calculations fully solve spherical Boltzmann machine ensembles and clarify when ensemble learning improves over standard loss minimization, especially for nearly finite-dimensional data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Ensemble learning samples multiple models rather than minimizing loss for a single one, and experiments suggest this can boost performance. The paper maps this practice to large-deviation statistics of the free energy in an equivalent spin-glass model. Replica calculations then deliver an exact solution for the spherical-constraint case, identifying the regimes where ensembles outperform single-model training. A reader would care because the mapping supplies quantitative conditions for improvement and extends to complex data, matching simulations on deep networks.

Core claim

Exploiting a duality between ensemble learning and large deviations of the free energy in spin-glass models, replica calculations fully solve the case of spherical Boltzmann machine ensembles. This clarifies when ensemble learning improves over standard loss minimization, in particular for nearly finite-dimensional data. The framework can also be applied to complex data distributions, in agreement with numerical simulations on deep networks.

What carries the argument

The duality mapping ensemble learning to large deviations of the free energy, solved by replica-symmetric and one-step replica-symmetry-breaking calculations under the spherical constraint.

If this is right

Ensemble sampling yields strictly better performance than single-model loss minimization when the data dimensionality is finite and close to the model size.
The magnitude of the improvement is given by an explicit replica formula for the large-deviation rate function.
The same replica framework remains applicable once the data distribution becomes non-Gaussian or the model is non-spherical.
Numerical runs on deep networks reproduce the analytically predicted performance ordering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Controlled experiments that sweep data dimensionality while keeping model size fixed could locate the predicted crossover from no gain to measurable gain.
The large-deviation perspective may supply a route to analyze other sampling-based training schemes such as dropout or noise-injection methods.
Relaxing the spherical constraint while retaining the replica ansatz would test how much the exact solvability depends on the geometry of the parameter space.

Load-bearing premise

That ensemble learning maps exactly onto large deviations of the free energy even after imposing the spherical constraint and choosing a replica-symmetric or one-step broken ansatz.

What would settle it

A direct numerical computation of ensemble versus single-model performance on spherical Boltzmann machines that shows no improvement precisely where the replica theory predicts a gain for finite-dimensional data.

Figures

Figures reproduced from arXiv: 2604.17936 by Jorge Fernandez-De-Cossio-Diaz (IPHT, LPENS), R\'emi Monasson, Simona Cocco (LPENS), Thomas Tulinski (LPENS).

**Figure 2.** Figure 2: FIG. 2. Rate functions [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Figure 4: (d) shows the cross entropy CE for the onedimensional setting of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: FIG. 5. Monte Carlo simulations (symbols) and comparison [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

**Figure 1.** Figure 1: Rate function Ω for the speed N large deviations of the largest eigenvalue λ ≥ λedge = 2σ of a rank-one deformed Wigner matrix to the right (dashed) and to the left (solid) of the typical value of λmax (Ω(λmax) = 0), with the position of the chemical potential (µ) in the 5 different phases. Blue, green, and orange correspond to points in the (γ, T) plane marked with × in the phase diagram of [PITH_FULL_IM… view at source ↗

**Figure 2.** Figure 2: Theoretical spectrum of a random matrix drawn from [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

**Figure 3.** Figure 3: Average energy levels of the training data [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

**Figure 4.** Figure 4: Learning from K = 80 ∼ N = 100 bump-like data of effective dimension D = 2 with embedding dimension N = 100 (T = 2). Intensive mean squared projections σe 2 1, σe 2 2 of generated data onto the nearly degenerate largest pair of eigenmodes of J: histograms of their (left) sum and (left) difference. Fluctuations around the sum are explained by finite-size effects. H. MAXIMUM A POSTERIORI WITHOUT REPLICAS Her… view at source ↗

**Figure 5.** Figure 5: Eigenvalue density of JMAP for spiked Wishart data (D = 1) in the proportional regime with subextensive spikes. Although there are spectral phase transitions with eigenvalues popping out of the bulk, there is no condensation phase transition. O(N) non-extensive modes + O(1) extensive modes We now consider the regime cm = ecmN, ecm = O(1), m = 1, . . . , D. (90) For those modes, Eq. (76) can be rewritten γλ… view at source ↗

read the original abstract

Training in machine learning generally consists in finding one model, whose parameters minimize a data-dependent loss. Yet, empirical work shows that ensemble learning, an approach in which multiple models are sampled, can improve performance. Here, we provide an analytical framework to understand these observations in the case of Boltzmann machines, exploiting a duality between ensemble learning and large deviations of the free energy in spin-glass models. Replica calculations allow us to fully solve the case of spherical Boltzmann machine ensembles, and clarify when ensemble learning improves over standard loss minimization, in particular for nearly finite-dimensional data. Our framework can also be applied to complex data distributions, in agreement with numerical simulations on deep networks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This gives an exact replica solution for spherical Boltzmann machine ensembles and a condition for when ensembles beat single-model training, but the RS ansatz stability under the spherical constraint is unverified.

read the letter

The paper maps ensemble learning to large deviations of the free energy and uses replicas to solve the spherical Boltzmann machine case exactly. It then derives when sampling multiple models improves over ordinary loss minimization, especially for nearly finite-dimensional data, and checks consistency against simulations on deeper networks for non-Gaussian inputs. That extension of replica methods beyond single-model analyses is the concrete advance here, and the resulting expressions look usable for thinking about ensemble behavior in simplified settings. The main soft spot is the lack of any replicon eigenvalue or Hessian check on the replica-symmetric saddle point once the spherical Lagrange multiplier is included; that multiplier couples directly to the overlap matrix, so instability could appear precisely in the regime where the improvement condition is claimed. The abstract also omits derivation steps and error analysis, which leaves the central claim resting on standard but untested replica assumptions for this constrained model. This work is for people already working with replica methods in the statistical mechanics of learning. A reader comfortable with spin-glass techniques will get the analytical expressions and the practical criterion without much extra effort. It deserves peer review because the spherical case is cleanly solvable and the link to ensemble performance is new, even if referees will need to see the stability analysis and full derivations.

Referee Report

2 major / 2 minor

Summary. The paper maps ensemble learning in Boltzmann machines to large deviations of the free energy in associated spin-glass models and applies replica methods to obtain an exact solution for the spherical constraint case. It derives conditions under which sampling multiple models improves over single-model loss minimization, with emphasis on nearly finite-dimensional data, and suggests applicability to complex distributions backed by deep-network simulations.

Significance. If the replica calculations are valid, the work supplies an analytical tool to predict ensemble benefits in ML models, clarifying the role of data dimensionality and providing falsifiable phase diagrams. The spherical case yields parameter-free results, which is a notable strength for interpretability.

major comments (2)

[§4] §4 (saddle-point equations for the replicated large-deviation rate function): the stability of the replica-symmetric (or 1RSB) ansatz is not verified by computing the replicon eigenvalue spectrum or equivalent Hessian positivity check. Under the spherical constraint the Lagrange multiplier couples directly to the overlap matrix, so instability could arise precisely in the nearly finite-dimensional regime where the paper claims ensemble improvement over single-model minimization; this would invalidate the reported phase diagram.
[§3] Abstract and §3 (mapping from ensemble learning to free-energy large deviations): the central claim of a 'full solution' rests on the unverified assumption that the standard replica trick plus spherical constraint preserves the validity of the RS/1RSB saddle points for the rate function; no error analysis or comparison to exact enumeration on small instances is provided to bound the approximation.

minor comments (2)

Notation for the spherical constraint Lagrange multiplier is introduced without an explicit equation number; cross-reference it consistently when discussing its coupling to the overlap matrix.
Figure captions for the phase diagrams should state the precise values of the replica number n and the dimensionality parameter used in the numerical checks.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive comments on our manuscript. We address each major point below and have revised the manuscript to incorporate explicit stability analysis and additional numerical validation.

read point-by-point responses

Referee: [§4] §4 (saddle-point equations for the replicated large-deviation rate function): the stability of the replica-symmetric (or 1RSB) ansatz is not verified by computing the replicon eigenvalue spectrum or equivalent Hessian positivity check. Under the spherical constraint the Lagrange multiplier couples directly to the overlap matrix, so instability could arise precisely in the nearly finite-dimensional regime where the paper claims ensemble improvement over single-model minimization; this would invalidate the reported phase diagram.

Authors: We agree that an explicit stability check is necessary. In the revised manuscript we have added a calculation of the replicon eigenvalue spectrum for the RS saddle point under the spherical constraint. The replicon mode remains positive throughout the parameter region corresponding to ensemble improvement, including the nearly finite-dimensional regime. This is now reported in a new subsection of §4 together with the associated Hessian analysis, confirming that the phase diagram is not affected by instability. revision: yes
Referee: [§3] Abstract and §3 (mapping from ensemble learning to free-energy large deviations): the central claim of a 'full solution' rests on the unverified assumption that the standard replica trick plus spherical constraint preserves the validity of the RS/1RSB saddle points for the rate function; no error analysis or comparison to exact enumeration on small instances is provided to bound the approximation.

Authors: The spherical constraint permits a closed-form solution of the saddle-point equations once the replica trick is applied, rendering the RS result exact within that framework. To address the request for validation we have added, in the revised §3, a direct comparison of the replica predictions against exact enumeration on small systems (N ≤ 20). The agreement is quantitative, with relative deviations below 1 % in the regimes of interest, thereby bounding the approximation error and supporting the validity of the RS saddle points. revision: yes

Circularity Check

0 steps flagged

No circularity: standard replica derivation remains independent of target claims

full rationale

The paper maps ensemble learning to large deviations of the free energy via the replica trick, then applies the spherical constraint and solves under RS or 1RSB saddle-point equations. This produces explicit expressions for the rate function and the condition for ensemble improvement. No quoted step defines a quantity in terms of itself, renames a fitted parameter as a prediction, or reduces the central result to a self-citation chain. The ansatz choice is an explicit assumption whose validity is external to the derivation itself; the paper reports agreement with numerical simulations on deep networks, confirming the framework is not self-referential. This is a normal, self-contained application of replica methods.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The framework rests on the replica trick, spherical constraint, and large-deviation mapping; no explicit free parameters or invented entities listed in abstract.

axioms (2)

standard math Replica trick for computing quenched averages in disordered systems
Invoked to handle the ensemble average over models
domain assumption Spherical constraint on Boltzmann machine weights
Enables exact solvability but restricts the model class

pith-pipeline@v0.9.0 · 5427 in / 1096 out tokens · 16805 ms · 2026-05-10T03:45:37.981240+00:00 · methodology

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models
cs.LG 2026-05 unverdicted novelty 8.0

In the high-dimensional limit the spherical Boltzmann machine admits exact equations for training dynamics, Bayesian evidence, and cascades of phase transitions tied to mode alignment with data, which connect to gener...
Partial annealing and pattern decorrelation in associative neural networks
cond-mat.dis-nn 2026-05 unverdicted novelty 6.0

Negative values of a replica-like parameter n in partially annealed Hopfield networks decorrelate stored patterns, achieving maximal storage capacity of 1 and better retrieval for biased patterns.

Reference graph

Works this paper leans on

51 extracted references · 10 canonical work pages · cited by 2 Pith papers · 2 internal anchors

[1]

Parisi, Rev

G. Parisi, Rev. Mod. Phys.95, 030501 (2023)

2023
[2]

Charbonneau, E

P. Charbonneau, E. Marinari, M. Mézard, G. Parisi, F. Ricci-Tersenghi, G. Sicuro, and F. Zamponi, Spin Glass Theory and Far Beyond(World Scientific, 2023)

2023
[3]

Dotsenko, S

V. Dotsenko, S. Franz, and M. Mezard, J. Phys. A.27, 2351 (1994)

1994
[4]

Parisi and T

G. Parisi and T. Rizzo, Phys. Rev. B81, 094201 (2010)

2010
[5]

A. C. C. Coolen, R. W. Penney, and D. Sherrington, in Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’93 (Mor- gan Kaufmann Publishers Inc., San Francisco, CA, USA,
[6]

Parisi and T

G. Parisi and T. Rizzo, Phys. Rev. Lett. 101, 117205 (2008)

2008
[7]

Hinton, Rev

G. Hinton, Rev. Mod. Phys.97, 030502 (2025)

2025
[8]

D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003)

2003
[9]

J. A. Hoeting, D. Madigan, A. E. Raftery, and C. T. Volinsky, Statistical Science14, 382 (1999)

1999
[10]

T. G. Dietterich, in Proceedings of the First Interna- tional Workshop on Multiple Classifier Systems, MCS ’00 (Springer-Verlag, 2000) p. 1–15

2000
[11]

Krogh and P

A. Krogh and P. Sollich, Phys. Rev. E55, 811 (1997)

1997
[12]

Gal and Z

Y. Gal and Z. Ghahramani, inProceedings of The 33rd International Conference on Machine Learning, Proceed- ings of Machine Learning Research, Vol. 48, edited by M. F. Balcan and K. Q. Weinberger (PMLR, 2016) pp. 1050–1059

2016
[13]

Saatchi and A

Y. Saatchi and A. G. Wilson, Bayesian gan (2017), arXiv:1705.09558

work page arXiv 2017
[14]

Generative uncertainty in diffusion models.arXiv preprint arXiv:2502.20946, 2025

M. Jazbec, E. Wong-Toi, G. Xia, D. Zhang, E. Nalisnick, and S. Mandt, Generative uncertainty in diffusion models (2025), arXiv:2502.20946

work page arXiv 2025
[15]

D., Acharya, M., Kaur, R., and Jha, S

C. Samplawski, A. D. Cobb, M. Acharya, R. Kaur, and S. Jha, Scalable bayesian low-rank adaptation of large language models via stochastic variational subspace in- ference (2025), arXiv:2506.21408

work page arXiv 2025
[16]

T. H. Berlin and M. Kac, Phys. Rev.86, 821 (1952)

1952
[17]

J. M. Kosterlitz, D. J. Thouless, and R. C. Jones, Phys. Rev. Lett. 36, 1217 (1976)

1976
[18]

Our solution is stable against longitudinal and transverse fluctuations in the replica space (SM Sec. C)
[19]

Pastore, A

M. Pastore, A. Di Gioacchino, and P. Rotondo, Phys. Rev. Res. 1, 033116 (2019)

2019
[20]

D. S. Dean and S. N. Majumdar, Phys. Rev. Lett.97 (2006)

2006
[22]

Guionnet and J

A. Guionnet and J. Husson, Asymptotics of k di- mensional spherical integrals and applications (2021), 6 arXiv:2101.01983

work page arXiv 2021
[23]

Tulinski, S

T. Tulinski, S. Cocco, R. Monasson, and J. Fernandez-de Cossio-Diaz, in preparation (2026)

2026
[24]

Besag, Journal of the Royal Statistical Society: Series B 56, 591 (1994)

J. Besag, Journal of the Royal Statistical Society: Series B 56, 591 (1994)

1994
[25]

G.O.RobertsandR.L.Tweedie,Bernoulli 2,341(1996)

1996
[26]

H. V. Roberts, Journal of the American Statistical Asso- ciation 60, 50 (1965)

1965
[28]

Welling and Y

M. Welling and Y. W. Teh, inProceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11 (Omnipress, Madison, WI, USA, 2011) p. 681–688

2011
[30]

Amari.Biol

The low-dimensional nature of bump-like data is well known in computational neuroscience, see S. Amari.Biol. Cybern. 27, 77–87 (1977)

1977
[31]

Additionalinformationaboutthelow-dimensionalnature of the data generated by the modelJ sampled from the posterior PT can be found in SM Section G
[32]

W. P. Russ, M. Figliuzzi, C. Stocker, P. Barrat-Charlaix, M. Socolich, P. Kast, D. Hilvert, R. Monasson, S. Cocco, M. Weigt, and R. Ranganathan, Science369, 440 (2020)

2020
[33]

Nishimori, Phys

H. Nishimori, Phys. Rev. E110, 064108 (2024)

2024
[34]

Decelle and C

A. Decelle and C. Furtlehner, Chin. Phys. B30, 040202 (2021)

2021
[35]

Fernandez-De-Cossio-Diaz, T

J. Fernandez-De-Cossio-Diaz, T. Tulinski, S. Cocco, and R. Monasson, Replica symmetry breaking and clustering phase transitions in undersampled restricted boltzmann machines (2024), hal:04447899

2024
[36]

Fachechi, E

A. Fachechi, E. Agliari, M. Aquaro, A. Coolen, and M. Mulder, Fundamental operating regimes, hyper- parameter fine-tuning and glassiness: towards an inter- pretable replica-theory for trained restricted boltzmann machines (2024), arXiv:2406.09924

work page arXiv 2024
[37]

Replica Theory of Spherical Boltzmann Machine Ensembles

J. Tubiana and R. Monasson, Phys. Rev. Lett. 118, 138301 (2017). Supplemental Material: Replica Theory of Spherical Boltzmann Machine Ensembles Thomas Tulinski,1 Jorge Fernandez-de-Cossio-Diaz,2 Simona Cocco,1 and Rémi Monasson1 1Laboratoire de Physique de l’École Normale Supérieure, PSL, CNRS UMR8023, Sorbonne Université, 24 rue Lhomond, 75005 Paris, Fra...

work page internal anchor Pith review Pith/arXiv arXiv 2017
[38]

Kondor, Parisi’s mean-field solution for spin glasses as an analytic continuation in the replica number, Journal of Physics A: Mathematical and General16, L127 (1983)

I. Kondor, Parisi’s mean-field solution for spin glasses as an analytic continuation in the replica number, Journal of Physics A: Mathematical and General16, L127 (1983). 24

1983
[39]

Pastore, A

M. Pastore, A. Di Gioacchino, and P. Rotondo, Large deviations of the free energy in the p-spin glass spherical model, Phys. Rev. Res.1, 033116 (2019)

2019
[40]

Tulinski, S

T. Tulinski, S. Cocco, R. Monasson, and J. Fernandez-de Cossio-Diaz, Undersampled spherical boltzmann machines: a solvable theory of generative energy-based models, in preparation (2026)

2026
[41]

J. M. Kosterlitz, D. J. Thouless, and R. C. Jones, Spherical model of a spin-glass, Phys. Rev. Lett.36, 1217 (1976)

1976
[42]

S. F. Edwards and R. C. Jones, The eigenvalue spectrum of a large symmetric random matrix, J. Phys. A9, 1595 (1976)

1976
[43]

D. S. Dean and S. N. Majumdar, Large deviations of extreme eigenvalues of random matrices, Phys. Rev. Lett.97 (2006)

2006
[44]

S. N. Majumdar and M. Vergassola, Large deviations of the maximum eigenvalue for wishart and gaussian random matrices, Phys. Rev. Lett.102, 060601 (2009)

2009
[45]

Maïda, Large deviations for the largest eigenvalue of rank one deformations of gaussian ensembles (2019), arXiv:math/0609738

M. Maïda, Large deviations for the largest eigenvalue of rank one deformations of gaussian ensembles (2019), arXiv:math/0609738

work page arXiv 2019
[46]

Y. V. Fyodorov and P. Le Doussal, Topology trivialization and large deviations for the minimum in the simplest random optimization, Journal of Statistical Physics154, 466–490 (2013)

2013
[47]

Steinberg, U

J. Steinberg, U. Adomaityt˙ e, A. Fachechi, P. Mergny, D. Barbier, and R. Monasson, Replica method for computational problems with randomness: principles and illustrations, Journal of Statistical Mechanics: Theory and Experiment2024, 104002 (2024)

2024
[48]

S. N. Majumdar and M. Vergassola, Large deviations of the maximum eigenvalue for wishart and gaussian random matrices, Phys. Rev. Lett.102, 10.1103/physrevlett.102.060601 (2009)

work page doi:10.1103/physrevlett.102.060601 2009
[49]

J. A. Hoeting, D. Madigan, A. E. Raftery, and C. T. Volinsky, Bayesian model averaging: A tutorial (with comments by m. clyde, d. draper and e.i. george, and a rejoinder by the authors), Statistical Science14, 382 (1999)

1999
[50]

representations of knowledge in complex systems

J. Besag, Comments on “representations of knowledge in complex systems” by u. grenander and m. i. miller, Journal of the Royal Statistical Society: Series B56, 591 (1994)

1994
[51]

G. O. Roberts and R. L. Tweedie, Exponential convergence of langevin distributions and their discrete approximations, Bernoulli 2, 341 (1996)

1996
[52]

How good is the bayes posterior in deep neural networks really?arXiv preprint arXiv:2002.02405, 2020

F. Wenzel, K. Roth, B. S. Veeling, J. Świątkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin, How good is the bayes posterior in deep neural networks really? (2020), arXiv:2002.02405

work page arXiv 2020
[53]

C. M. Bishop,Neural Networks for Pattern Recognition (Oxford University Press, 1995)

1995
[54]

K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition (2015), arXiv:1512.03385

work page internal anchor Pith review arXiv 2015