REVIEW 2 major objections 5 minor 64 references
The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Activation functions, not just weights, decide which statistical interactions an RBM can express.
desk verdict A useful RBM review whose central theoretical claim is too strong—there is a concrete counterexample—so it needs a corrected statement before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the interaction expansion of the visible marginal, Eq. (2), in which the interaction of order $s$ is written as sums of $K_\mu$ evaluated at weighted sums of visible units, where $K_\mu(x)=\log \mathbb{E}_{z_\mu} e^{z_\mu x}$ is the cumulant generating function of the hidden unit distribution. The paper uses this expansion to derive the impossibility result for non-linear activations, and uses the linear-activation case to map the RBM onto a Hopfield network, whose storage-load transition becomes the overfitting transition in learning. It also functions as the language for comparing activation functions across applications.
What would settle it
Take a small RBM with a non-linear hidden activation, say a single ReLU hidden unit, and compute the coefficients of the visible interaction series exactly; if there is a choice of weights for which all coefficients of order three and higher vanish while the activation remains non-linear, the central impossibility claim is false. Equivalently, train a non-linear RBM on data generated by a pairwise model and check whether the fitted model shows strictly zero higher-order interactions beyond numerical noise; any non-zero higher-order coupling would support the paper's claim, while an exact zero would refute it.
Extended reading notes
Core claim
On the authors' account, the visible-unit marginal of an RBM is an exponential family whose interactions are set by the cumulant generating function $K_\mu(x)=\log \mathbb{E}_{z_\mu}\exp(z_\mu x)$ of the hidden units' distribution. Expanding the marginal in powers of the visible variables, every interaction order receives contributions from all hidden units, and the form of $K_\mu$ decides whether the coefficients beyond a given order can vanish. The central theoretical claim, taken from earlier work, is that no RBM with a non-linear activation can express only terms up to a prescribed order for any choice of weights; the only activation that suppresses all interactions beyond pairwise is the linear one, in which case the RBM is exactly a Hopfield model with stored patterns given by its weights. The same duality lets the Hopfield critical storage load $\alpha_c = 0.14$ be read as a bound beyond which RBM learning overfits. On the applied side, the paper claims that this is why protein and immunology applications need non-binary units with double ReLU potentials, while neural data applications often stay with binary sigmoid units.
Load-bearing premise
The whole theoretical story about activation functions controlling interaction orders rests on the interaction expansion in Eq. (2), which the paper imports from an earlier work without re-deriving it; if that expansion is incorrect or mis-stated, the central claim about non-linear activations losing control of interaction order collapses.
Editorial extensions
If this is right
- If the impossibility result holds, then any RBM trained with a ReLU, double-ReLU, or other non-linear hidden activation is, in principle, modelling arbitrarily high-order correlations, even when the data are pairwise; only linear hidden units guarantee a purely pairwise model.
- The Hopfield-RBM correspondence implies that increasing the number of hidden units in a linear RBM beyond the analogue of $\alpha_c$ will degrade generative performance through overfitting, giving a concrete failure mode rather than a mere heuristic.
- For protein-sequence modelling, the double-ReLU potential of Eq. (5) is presented as conferring the expressive power needed to capture high-order constraints among amino-acid positions, which is why it outperforms sigmoid units on those data.
- The compositional phase found analytically for RBMs with ReLU hidden units provides a mechanism by which visible configurations are represented by a small set of strongly active hidden units, relevant for interpreting latent representations of neural and protein data.
- Differential learning (diffRBM) exploits the interpretability of a few added hidden units to isolate the statistical motifs that distinguish immunogenic antigens from generic ones, a corollary of the same latent-space structure.
Reading between the lines
- Extension not in the paper: if the impossibility result is exact, then sparsity or regularization on weights cannot restore a purely pairwise RBM once a non-linear activation is chosen; the higher-order terms are structural, not an artifact of finite data.
- Extension not in the paper: comparing fitted interaction coefficients across activation functions for a fixed dataset would quantify how much of the 'higher-order' signal is genuine data structure versus an artefact of the chosen activation, a testable diagnostic.
- Extension not in the paper: the RBM-HM duality might transfer results on Hopfield storage of correlated or biased patterns directly to RBM learning on biased biological datasets, constraining when latent representations are expected to be trustworthy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This perspective reviews how the choice of activation function in Restricted Boltzmann Machines shapes their representational properties and their use in biological applications. After introducing the RBM formalism, the paper presents a theoretical discussion centered on Eq. (2), the interaction expansion of the visible marginal, and uses it to contrast linear hidden activations, which map to pairwise Hopfield-like models and a retrieval/overfitting transition, with non-linear activations, which are claimed to generate all interaction orders. The rest of the review surveys applications to neural recordings, protein-family modeling and design, immunology, and the diffRBM transfer-learning strategy, and closes with open problems on activation-function choice, learning dynamics, and noisy overparameterized learning.
Significance. If the theoretical statements are corrected, this will be a useful and readable synthesis that connects an active theoretical literature on RBMs to concrete biological applications. Its strengths are its clear organization, its broad and mostly accurate citation of the literature, and its informative treatment of the protein and immunology applications, particularly the diffRBM construction. The paper also provides an instructive toy illustration in Fig. 3, although the simulation details are missing. The main weakness is that the central theoretical claim about non-linear activations is overstated to the point of being false as written, and this affects the downstream framing of the protein-modeling discussion. Since the false statement is local and can be repaired by a genericity caveat, I do not regard the error as fatal to the review's overall value.
major comments (2)
- [Non-Sigmoid activation function, after Eq. (2)] The sentence 'there is no RBM, endowed with a non-linear activation, able to express only terms up to a given order, for any choice of connections w' is false as stated, and the counterexample comes from the paper's own framework. Take N=3 visible units, M=1 hidden unit, hidden prior ρ(z)=1/2(δ(z-1)+δ(z+1)), so that K(x)=log cosh(x), and take weights w1=1, w2=1, w3=-2. The visible marginal is proportional to exp[log cosh(v1+v2-2v3)], with vi in {0,1}. The third-order interaction for S={1,2,3}, obtained by the inclusion-exclusion expression equivalent to Eq. (2), is Σ_{T⊆S} (-1)^{3-|T|} log cosh(Σ_{i∈T} wi), which evaluates to 0 because the positive singleton terms log cosh(1)+log cosh(1)+log cosh(2) cancel against the negative pair terms log cosh(2)+log cosh(1)+log cosh(1), and the empty and full terms vanish. Since N=3, there are no interactions of order greater than 3, so this RBM expresses only terms up to second order despite having a non-linear activation and a valid weight choice. The correct statement is that non-linear activations generically generate all interaction orders, with possible exact cancellations at special weight configurations. Please revise the claim accordingly and adjust any text that relies on the universal version.
- [Protein Families, around Eq. (5)] The discussion of the double-ReLU potential in the protein section presents the ability to learn high-order correlations as a consequence of the general theoretical claim about non-linear activations. Since that general claim is falsified by the explicit counterexample above, the connection needs to be reformulated: the high-order-correlation behavior of the dReLU RBM in [3] is an empirical/modeling result for trained weights, not a consequence of the universal theorem stated in the theory section. Without this reformulation, the theoretical motivation offered for Eq. (5) is not valid.
minor comments (5)
- [Eq. (2)] The notation in Eq. (2) is difficult to parse: the summation 'Σ_{j1<...<j_{s-p}=1}' should be written as 'Σ_{1≤j1<...<j_{s-p}≤s}', and the argument of K should be explicitly identified as Σ_{l=1}^{s-p} w_{k_{j_l} μ}. Please also state that Eq. (2) is the standard inclusion-exclusion expansion of K_μ(Σ_i w_{iμ} v_i) so that readers can verify it.
- [Fig. 3] The toy simulation in Fig. 3 is not reproducible as reported: the learning rate, the number of gradient-descent steps, the initialization distribution, and the parameters defining p_LG(v) are not given. Since the figure is used to support a claim about convergence speed across activation functions, these details should be added to the caption or the main text.
- [Non-Sigmoid activation function (title and text)] The subsection title 'Non-Sigmoid activation function' is confusing because the text includes both linear activations and the standard sigmoidal/log-cosh case as examples of non-linear activations. Please clarify whether 'non-sigmoid' refers to activation functions different from the standard binary-unit sigmoid or to all non-linear activations.
- [Neural Data, TRBM sentence] The sentence 'a similar dynamical variant of RBM, the Recurrent Temporal RBM (RTRBM) can be trained efficiently and exactly' is imprecise; standard RTRBM training uses approximate inference and contrastive-divergence-style updates. Please clarify in what sense the training is exact, or soften the claim.
- [Phase diagram and compositionality] The statement that the RBM-HM duality relates 'supervised learning in HM' to 'unsupervised learning in RBM' could be misread as a claim about learning rules. The formal equivalence is between the visible marginal of a linear-activation RBM and the equilibrium distribution of a Hopfield model; please state this explicitly.
Circularity Check
No significant circularity: the central theoretical claim is a review-level citation to the authors' own prior published theorem, which is independent support, not a re-derivation from the manuscript's own inputs.
full rationale
This paper is a review/perspective, not a derivation of new results. The central theoretical claim about non-linear activations (Section 'Non-Sigmoid activation function') is explicitly attributed to [2] (Bulso & Roudi 2021), a prior paper co-authored by one of the present authors. This is a load-bearing self-citation for the review's narrative. However, the cited work is a published, peer-reviewed, parameter-free mathematical result with stated assumptions; it does not depend on the present paper's fitted values or data, and it is externally checkable. Under the rule that such citations are independent support, this self-citation does not constitute circularity. The manuscript does not derive new results from its own assumptions; it surveys published theory and applications. No equation is reused as its own input, and no fitted parameter is renamed as a prediction. The possible falsity of the claim (e.g., the suggestion of a weight configuration with vanishing third-order interactions) is a correctness concern, not a circularity concern, and should be weighed separately. Therefore no significant circularity is present.
Assumptions & free parameters
assumptions (4)
- standard math RBM energy function defined by Eq. (1) with Boltzmann distribution.
- standard math The marginal distribution over visible units can be expressed via cumulant generating function K as in Eq. (3b).
- domain assumption The formal equivalence between linear-activation RBMs and Hopfield models holds for binary weights.
- domain assumption Universal approximation property of RBMs with sufficiently many hidden units.
Cite this review
Pith. "Pith review of The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications." pith.science (2026). https://pith.science/paper/P2YJXC4D
@misc{pith2026250104387,
author = {Pith},
title = {Pith review of: The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2YJXC4D}},
note = {Machine review of arXiv:2501.04387}
}
read the original abstract
Restricted Boltzmann Machines are simple yet powerful neural networks. They can be used for learning structure in data, and are used as a building block of more complex neural architectures. At the same time, their simplicity makes them easy to use, amenable to theoretical analysis, yielding interpretable models in applications. Here, we focus on reviewing the role that the activation functions, describing the input-output relationship of single neurons in RBM, play in the functionality of these models. We discuss recent theoretical results on the benefits and limitations of different activation functions. We also review applications to biological data analysis, namely neural data analysis, where RBM units are mostly taken to have sigmoid activation functions and binary units, to protein data analysis and immunology where non-binary units and non-sigmoid activation functions have recently been shown to yield important insights into the data. Finally, we discuss open problems addressing which can shed light on broader issues in neural network research.
Figures
Reference graph
Works this paper leans on
- [3]
-
[1]
Smolensky P., Tech. Rep. Department of Computer Sci- ence, University of Colorado, Boulder (1986)
work page 1986
- [2]
-
[4]
and Bengio Y., Neural Comput., 20 (2008) 1631
Le Roux N. and Bengio Y., Neural Comput., 20 (2008) 1631
work page 2008
-
[5]
and Ghazi-Zahedi K., Journal of Machine Learning Research, 16 (2015) 2405
Mont´ufar G., Ay N. and Ghazi-Zahedi K., Journal of Machine Learning Research, 16 (2015) 2405
work page 2015
-
[6]
Ackley D. H., Hinton G. E. and Sejnowski T. J. , Cognitive Science, 9 (1985) 147
work page 1985
-
[7]
Hinton G. E. , Neural Comput., 14 (2002) 1771
work page 2002
-
[8]
Tieleman T., Proceedings of the 25th international con- ference on Machine learning , (2008) 1064
work page 2008
Show all 64 references
-
[9]
Desjardins G. et al. , Proceedings of the 13th interna- tional conference on artificial intelligence and statistics , (2010) 145
2010
-
[10]
and Igel C., Pattern Recognition, 47 (2014) 25
Fischer A. and Igel C., Pattern Recognition, 47 (2014) 25
2014
-
[11]
and Seoane B
Decelle A., Furtlehner C. and Seoane B. , J. Stat. Mech.: Theory and Experiment , 2022 (2022) 114009
2022
-
[12]
and Con- tucci P., Neural Networks, 34 (2012) 1
Barra A., Bernacchia A., Santucci E. and Con- tucci P., Neural Networks, 34 (2012) 1
2012
-
[13]
and Wilson M., Phys
Cossu G., Del Debbio L., Giani T., Khamseh A. and Wilson M., Phys. Rev. B , 100 (2019) 064304
2019
-
[14]
Decelle A., Furtlehner C., Navas G ´omez A. d. J. and Seoane B., SciPost Physics , 16 (2024) 095
2024
-
[15]
and Agliari E., Entropy, 23 (2021)
Marullo C. and Agliari E., Entropy, 23 (2021)
2021
-
[16]
and De Marzo G., Neural Networks, 148 (2022) 232
Agliari E., Alemanno F., Barra A. and De Marzo G., Neural Networks, 148 (2022) 232
2022
-
[17]
and Tantari D., Phys
Barra A., Genovese G., Sollich P. and Tantari D., Phys. Rev. E , 97 (2018) 022310
2018
-
[18]
J., Modeling brain function: The world of attrac- tor neural networks (Cambridge university press) 1989
Amit D. J., Modeling brain function: The world of attrac- tor neural networks (Cambridge university press) 1989
1989
-
[19]
and Agliari E., Europhysics Letters, 141 (2023) 11001
Alemanno F., Aquaro M., Kanter I., Barra A. and Agliari E., Europhysics Letters, 141 (2023) 11001
2023
-
[20]
Treves A., J. Phys. A: Math. Gen. , 23 (1990) 2631
1990
-
[21]
and van Hemmen J
K¨uhn R., B¨os S. and van Hemmen J. L., Phys. Rev. A, 43 (1991) 2084
1991
-
[22]
Treves A., J. Phys. A: Math. Gen. , 24 (1991) 327
1991
-
[23]
and van Mourik J
Bolle D., Kuhn R. and van Mourik J. , J. Phys. A: Math. Gen. , 26 (1993) 3149
1993
-
[24]
and Treves A
Sch¨onsberg F., Roudi Y. and Treves A. , Phys. Rev. Lett., 126 (2021) 018301
2021
-
[25]
and Tantari D., Phys
Barra A., Genovese G., Sollich P. and Tantari D., Phys. Rev. E , 96 (2017) 042156
2017
-
[26]
Agliari E., Leonelli F. E. and Marullo C., Applied Mathematics and Computation , 415 (2022) 126716
2022
-
[27]
E., Agliari E., Albanese L
Leonelli F. E., Agliari E., Albanese L. and Barra A., Neural Networks, 143 (2021) 314
2021
-
[28]
Krotov D., Nat. Rev. Phys. , 5 (2023) 366
2023
-
[29]
and M´ezard M
Lucibello C. and M´ezard M. , Phys. Rev. Lett. , 132 (2024) 077301
2024
-
[30]
and Monasson R
Tubiana J. and Monasson R. , Phys. Rev. Lett. , 118 (2017) 138301
2017
-
[31]
and Furtlehner C., J
Decelle A., Fissore G. and Furtlehner C., J. of Stat. Phys., 172 (2018) 1576
2018
-
[32]
and Troyer M., Science, 355 (2017) 602
Carleo G. and Troyer M., Science, 355 (2017) 602
2017
-
[33]
Hartmann M. J. and Carleo G., Phys. Rev. Lett., 122 (2019) 250502
2019
-
[34]
K¨oster U., Sohl-Dickstein J., Gray C. M. and Ol- shausen B. A. , PLOS Comp. Biol. , 10 (2014) 1
2014
-
[35]
K., Whitlock J
Olsen V. K., Whitlock J. R. and Roudi Y. , PLOS Comp. Biol. , 20 (2024) e1012074
2024
-
[36]
van der Plas T. L. et al. , Elife, 12 (2023) e83139
2023
-
[37]
Volpi R. et al. , Sci Rep, 10 (2020) 16549
2020
-
[38]
and Mora T
Gardella C., Marre O. and Mora T. , Proc. Natl. Acad. of Sci. , 115 (2018) 3267
2018
-
[39]
Sutskever I., Hinton G. E. and Taylor G. W. , Advances in neural information processing systems , 21 (2008)
2008
-
[40]
Bravi B. et al. , Cell Systems , 12 (2021) 195
2021
-
[41]
Bravi B. et al. , eLife, 12 (2023) e85126
2023
-
[42]
Quiroz Monnens S., Peters C., Smeets K., Hes- selink L. W. and Englitz B., eLife, 13 (2024)
2024
-
[43]
and Weigt M., Phys
Shimagaki K. and Weigt M., Phys. Rev. E , 100 (2019) 032128
2019
-
[44]
and Monasson R., Bioinformatics, 37 (2021) 4083
Malbranke C., Bikard D., Cocco S. and Monasson R., Bioinformatics, 37 (2021) 4083
2021
-
[45]
Tubiana J. et al. , PLOS Comp. Biol. , 19 (2023) e1010874
2023
-
[46]
Malbranke C. et al. , PLOS Comp. Biol. , 19 (2023) e1011621
2023
-
[47]
and Monasson R
Mauri E., Cocco S. and Monasson R. , Phys. Rev. Lett., 130 (2023) 158402
2023
-
[48]
and Rosset L., Phys
Decelle A., Seoane B. and Rosset L., Phys. Rev. E , 108 (2023) 014110
2023
-
[49]
Morcos F. et al. , Proc. Natl. Acad. Sci. , 108 (2011) E1293
2011
-
[50]
and Seoane B., IEEE Transactions on Pattern Analysis and Machine In- telligence, (2024)
Carbone A., Decelle A., Rosset L. and Seoane B., IEEE Transactions on Pattern Analysis and Machine In- telligence, (2024)
2024
-
[51]
Gioacchino A. D. et al. , PLOS Comp. Biol. , 18 (2022) e1010561
2022
-
[52]
, bioRxiv, (2024) 2023.05.10.540155
Fernandez-de-Cossio-Diaz et al. , bioRxiv, (2024) 2023.05.10.540155
2024
-
[53]
et al., PLOS Comp
Bravi B. et al., PLOS Comp. Biol. , 17 (2021) e1009297
2021
-
[54]
, Nature, 606 (2022) 389
Luksza M.et al. , Nature, 606 (2022) 389
2022
-
[55]
Bravi B., npj Vaccines, 9 (2024) 1
2024
-
[56]
and Bhardwaj N., Fron- tiers in Immunology , 11 (2020) 27
Roudko V., Greenbaum B. and Bhardwaj N., Fron- tiers in Immunology , 11 (2020) 27
2020
-
[57]
and Zamponi F., From Statis- tical Physics to Data-driven Modelling: With Applications to Quantitative Biology (Oxford University Press) 2022
Cocco S., Monasson R. and Zamponi F., From Statis- tical Physics to Data-driven Modelling: With Applications to Quantitative Biology (Oxford University Press) 2022
2022
-
[58]
and Solla S
Saad D. and Solla S. A., Phys. Rev. E, 52 (1995) 4225
1995
-
[59]
and Solla S
Saad D. and Solla S. A. , Phys. Rev. Lett. , 74 (1995) 4337
1995
-
[60]
and Rattray M., Phys
Saad D. and Rattray M., Phys. Rev. E, 57 (1998) 2170
1998
-
[61]
M., Krzakala F
Goldt S., Advani M., Saxe A. M., Krzakala F. and Zdeborov´a L., Advances in neural information process- ing systems , 32 (2019)
2019
-
[62]
et al., Advances in Neural Information Pro- cessing Systems, 34 (2021) 18137
Loureiro B. et al., Advances in Neural Information Pro- cessing Systems, 34 (2021) 18137
2021
-
[63]
and Furtlehner C
Decelle A., Fissore G. and Furtlehner C. , Euro- physics Letters, 119 (2017) 60001
2017
-
[64]
and Seoane B
Bachtis D., Biroli G., Decelle A. and Seoane B. , arXiv preprint arXiv:2405.14689 , (2024) . p-7
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.