REVIEW 4 major objections 5 minor 33 references
MEP-Net: Generating Solutions to Scientific Problems with Limited Knowledge by Maximum Entropy Principle
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read MEP-Net couples neural networks with the maximum entropy principle to reconstruct probability distributions from moment constraints alone.
desk verdict A useful reference review and a practical feature trick are buried under an unsupported claim that a KL regularizer implements maximum entropy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery has three moving parts. First, the total loss of Eq. (8) couples a constraint-matching term to the entropy term with weight $\lambda > 0$. Second, the entropy term of Eq. (9) is a fixed-point Kullback–Leibler divergence between the current output $\hat p_\theta$ and the previous iterate $\hat p_{\theta,\text{old}}$; it is always non-negative and reaches its minimum exactly at the fixed point $\hat p_\theta = \hat p_{\theta,\text{old}}$, which the paper reads as the convergence condition of the maximum-entropy iteration. Third, the observable library of Eq. (10) uses binomial functions $\varphi_{k_1,\ldots,k_d}(x) = \prod_j \binom{n}{k_j}x_j^{k_j}(1-x_j)^{n-k_j}$, whose combinatorial structure captures inter-variable relationships that polynomial, exponential, and trigonometric features miss. The classical anchor is the optimal exponential-family form $p^*(x) = \exp(\sum_i \lambda_i f_i(x))/Z$ of Eq. (5), which the network is designed to realize without ever solving for the Lagrange multipliers.
What would settle it
Train MEP-Net on one fixed set of binomial-feature constraints with several entropy weights $\lambda$, including $\lambda = 0$, and compare each converged output with the analytic maximum-entropy distribution $p^*(x) = \exp(\sum_i \lambda_i \varphi_{k_i}(x))/Z$ obtained by solving Eq. (5) for the same constraints. If Eq. (9) implements MEP, the $\lambda > 0$ outputs must coincide with $p^*$ and must have Shannon entropy at least as high as the $\lambda = 0$ output; a converged distribution that differs from $p^*$ or has lower entropy would show that the fixed-point KL term stabilizes training rather than maximizing entropy.
Extended reading notes
Core claim
The central claim is that the maximum entropy principle can be implemented by training a neural network with the loss $L_{\text{total}}(\theta) = L_{\text{constraint}}(\theta) + \lambda L_{\text{entropy}}(\theta)$ of Eq. (8), where the constraint term enforces the observed moments and the entropy term takes the fixed-point form $S[\hat p_\theta(x,t_k)] = \int \hat p_\theta(x,t_k)\, \ln\!\big(\hat p_\theta(x,t_k)/\hat p_{\theta,\text{old}}(x,t_k)\big)\, dx$ of Eq. (9). Because this Kullback–Leibler functional is non-negative and reaches zero exactly when the network output stops changing, the paper identifies its minimum with the entropy maximization of the classical MEP solution $p^*(x) = \exp(\sum_i \lambda_i f_i(x))/Z$ of Eq. (5). With binomial feature functions $\varphi_{k_1,\ldots,k_d}(x) = \prod_j \binom{n}{k_j} x_j^{k_j}(1-x_j)^{n-k_j}$ of Eq. (10) as the observables, which the authors find far more effective than polynomial, exponential, and trigonometric features, the converged network is claimed to generate the most unbiased distribution consistent with the data. The paper supports this with reconstructions of multimodal and high-dimensional distributions, the Schlögl model, and variational formulations of diffusion and phase separation.
Load-bearing premise
The load-bearing premise is that minimizing the fixed-point Kullback–Leibler divergence of Eq. (9) enforces the maximum entropy principle, so that the converged network output is the most unbiased distribution consistent with the observed moments.
Editorial extensions
If this is right
- The trained network outputs a normalized distribution directly, so the classical Lagrange-multiplier step of Eq. (5) is bypassed; maximum-entropy reconstruction becomes feasible in high dimensions where solving for the multipliers is hard.
- The fixed-point entropy loss doubles as a convergence monitor: it is zero exactly when the network output stops changing, so a vanishing entropy loss signals that training has reached its fixed point.
- Replacing the entropy term with a physical variational functional — the Rayleighian in the diffusion example, the free energy in the Allen–Cahn example — turns the same architecture into a deep solver for variational problems.
- Adding the entropy loss lowers the reconstruction error on the Schlögl chemical master equation by one to two orders of magnitude in mean squared error, evidence that the term does real work beyond the constraint fit.
Reading between the lines
- A testable consequence of the maximum-entropy claim: if the converged output really is the least biased distribution for the given constraints, it should not depend on the network's width, depth, or initialization.
- The fixed-point 'entropy' recipe is portable — any iterative deep solver, for inverse problems or dynamical systems, could adopt the same distance-from-the-previous-iterate term as a stabilizer with a built-in stopping criterion.
- The sharp improvement from switching to binomial observables hints that a problem's own closed moment hierarchy is the right feature library, so other sciences with known closure structures could import the idea directly.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MEP-Net, a neural network trained to output probability distributions that match a set of moment constraints, with an additional "entropy loss" defined in Eq. (9) as the KL divergence between the current network output and the previous iterate. The authors claim that this loss implements the maximum entropy principle, so the generated distributions are "the most unbiased one in physics," and they demonstrate the method on multimodal statistical distributions, the Schlögl chemical master equation, confined diffusion, and the Allen-Cahn equation. The paper also contains a tutorial review of maximum entropy derivations, including a large-deviations justification and tables of max-entropy distributions.
Significance. If the central claim were correct, the paper would offer a practical way to impose maximum entropy constraints in neural-network-based distribution reconstruction, which could be useful in stochastic chemical kinetics and other data-limited scientific settings. The review material in Section II and the appendices is largely standard but is presented clearly and could serve as a reference. However, the advertised connection to MEP is not established: the entropy loss in Eq. (9) is a smoothness regularizer, not an entropy functional, and the numerical experiments compare the method with and without this regularizer rather than testing maximum-entropy optimality. The binomial-feature selection is performed on the same examples used for evaluation, further weakening the generality of the reported results.
major comments (4)
- [III.A, Eq. (9)] The functional S[ˆp_θ] defined in Eq. (9) is the KL divergence D_KL(ˆp_θ || ˆp_θ,old), a functional of the current and previous iterates, whereas the Shannon entropy in Eq. (1) is S[p] = −∫ p ln p. Minimizing the loss in Eq. (8) therefore does not maximize entropy: at any fixed point with ˆp_θ = ˆp_θ,old the term is exactly zero, so the converged solution is selected by the constraint term and by the network's inductive bias, not by any entropy criterion. The text itself, immediately after Eq. (9), states that the entropy loss "aims to keep the changes in the MEP-Net slow and smooth," which is a regularization interpretation. No argument is given that the fixed point coincides with the exponential-family solution in Eq. (5), and the claim in Section III.B that MEP-Net produces "the most unbiased one in physics" is unsupported.
- [IV.B, IV.C, IV.D] The numerical demonstrations do not test the maximum-entropy claim. In Fig. 4(b) the comparison is between training with and without the entropy loss; a lower MSE in the presence of Eq. (9) is evidence that the term acts as a stabilizing regularizer, not that the output is max-entropy. Moreover, in Section IV.C the entropy loss is replaced by the Rayleighian and in Section IV.D by the free energy, which shows that Eq. (9) is treated as a generic physical variational regularizer rather than as a mathematical implementation of MEP. To support the paper's claim, the authors would need to compare the converged output against the actual max-entropy distribution over the same constraints (e.g., the exponential-family solution of Eq. (5) when the constraints are compatible with it) and show that the network selects it. No such comparison is reported.
- [III.B and IV.A] The binomial features are selected after testing polynomial, exponential, logarithmic, sine, cosine, square root and reciprocal features on the same benchmark distributions and rejecting them (Section III.B). This is a form of model selection on the test set, and the reported success of the binomial features on those examples is therefore not a predictive claim. The method should be validated on held-out problems or on a feature library fixed before evaluation before the paper can claim general reconstruction ability from limited information. Relatedly, the experiments are reconstruction tasks: the moment constraints in Eq. (8) are evaluated from the target distribution, so the network is fitted to the target's moments rather than generating new solutions from limited knowledge.
- [III.A, Eq. (8)] The method depends on several free choices — the weight λ, the number n of binomial terms, the number M of constraints, and the network architecture (two hidden layers of 50 nodes, Tanh and Softplus activations) — and the paper reports only that other choices were explored without matching the chosen configuration. No sensitivity analysis is given, and no convergence guarantee for the fixed-point iteration is provided. Because the claimed MEP property is a statement about the converged solution, the absence of a convergence result is a gap in the central argument.
minor comments (5)
- [II.A] There is a typo in the phrase "reconstruction of various classci probability distributions" in Section II.
- [IV] The paper uses inconsistent capitalization, e.g., "MEP-NEt" in the introduction to Section IV and "MEP-NET" in Section IV.A.2.
- [IV.D and V] The model name is misspelled as "Allen-Cahnn" in the Conclusion, and the text referring to Fig. 6 contains a placeholder "Fig. ??" for the MSE comparison.
- [Appendix A] There are typos such as "optiaml" and "itergration" in the derivations for the Laplace and Rayleigh distributions.
- [Data availability] The paper states that data are available from the corresponding author upon request; given the empirical nature of the work, releasing code and data would substantially improve reproducibility.
Circularity Check
Eq. (9) defines the 'entropy loss' as a KL divergence to the previous iterate, not Shannon entropy, so at convergence it cannot enforce maximum entropy; the claimed 'most unbiased' output is the moment-constrained network fit relabeled as an MEP result.
-
renaming known result
[Section III.A, Eq. (9); Section III.B]
"To solve this difficulty, we propose a practical form based on the concept of fixed-point iteration, S [ ˆpθ (⃗x,tk)] = Z Rd d⃗x ˆpθ (⃗x,tk)ln[ ˆpθ (⃗x,tk)/ ˆpθ (⃗x,tk)old] (9)... Based on the Kullback-Leibler divergence, it is straightforward to see that S [ ˆpθ (⃗x,tk)] is always non-negative and reaches its minimum if and only if ˆpθ (⃗x,tk) = ˆpθ (⃗x,tk)old... the entropy loss aims to keep the changes in the MEP-Net slow and smooth."
The paper defines its 'entropy function' S as a functional of both the current and the previous iterate, which is the Kullback-Leibler divergence D(p_hat || p_hat_old), not the Shannon entropy -∫ p ln p used in the MEP derivation of Eqs. (1)-(5). At any converged fixed point p_hat = p_hat_old the term is identically zero for every distribution, so it cannot select the maximum-entropy distribution among those satisfying the moment constraints. The later claim that 'the incorporation of the maximum entropy principle ensures that the generated probability distribution is the most unbiased one in physics' therefore attaches the MEP's unbiasedness guarantee to a regularizer by naming it 'entropy'; the central claim reduces to the label rather than to an entropy-maximizing objective.
-
fitted input called prediction
[Section III.A, Eq. (8); Section IV.B (Schlögl model)]
"Lconstraint(θ ) = 1/T Σ_{k=1}^T Σ_{i=1}^M ( E⃗X∼ ˆpθ [ fi(⃗X(tk))] − ˜fi(tk))2 ... the MEP-Net can reach a remarkable accuracy in generating the time-varying solutions to the Schlögl model, with the absolute errors below 1.2%."
The accuracy reported is the same moment-matching objective on which the network is trained: L_constraint directly penalizes the squared difference between moments of the network output and the data moments. Since the Eq. (9) 'entropy' term vanishes at any fixed point, the converged solution is selected by the constraint fit and network inductive bias, not by a maximum-entropy criterion. Presenting the output as 'generating' the solution therefore equates the prediction with the fitted input: the reported errors measure training convergence on the target moments rather than an independent generative or unbiasedness result.
full rationale
The background sections (II.A-II.C and Appendices A-B) contain standard maximum-entropy derivations and are not circular; the numerical demonstrations are reconstructions against known solutions and are self-contained in that narrow sense. However, the paper's central novelty is the claim that Eq. (9) implements the maximum entropy principle. That claim is circular by definitional substitution: Eq. (9) is a KL divergence to the previous iterate, explicitly described as keeping changes 'slow and smooth', and it vanishes at convergence, so it imposes no entropy selection on the final distribution. The paper then invokes the MEP's unbiasedness property to assert that the output is 'the most unbiased one in physics', importing a guarantee that applies to Shannon entropy, not to this fixed-point regularizer. The experimental successes reduce to fitting the specified moments, because the reported error is the training objective itself. The overall circularity is partial: the constraint-fitting component is a legitimate reconstruction task, but the 'maximum entropy' part of the claimed derivation is a renamed regularizer rather than an operative principle. Score 6 reflects that the central claim reduces by construction, while the numerical results retain some independent content as moment reconstructions.
Assumptions & free parameters
free parameters (4)
- lambda (entropy loss weight)
- n (degree of binomial features)
- M (number of binomial constraints)
- Network architecture (two hidden layers of 50 nodes, Tanh, Softplus)
assumptions (5)
- standard math Sanov's theorem and the contraction principle justify maximum entropy as the limit of empirical measures.
- domain assumption Neural networks can approximate the target probability distributions (universal approximation).
- ad hoc to paper Minimizing the fixed-point KL divergence in Eq. (9) implements the maximum entropy principle.
- ad hoc to paper Binomial features (Bernstein polynomials) are the right observables for reconstructing distributions.
- domain assumption Moment constraints estimated from one million Gillespie trajectories accurately represent the chemical master equation solution.
Cite this review
Pith. "Pith review of MEP-Net: Generating Solutions to Scientific Problems with Limited Knowledge by Maximum Entropy Principle." pith.science (2026). https://pith.science/paper/757VHL2X
@misc{pith2026241202090,
author = {Pith},
title = {Pith review of: MEP-Net: Generating Solutions to Scientific Problems with Limited Knowledge by Maximum Entropy Principle},
year = {2026},
howpublished = {\url{https://pith.science/paper/757VHL2X}},
note = {Machine review of arXiv:2412.02090}
}
read the original abstract
Maximum entropy principle (MEP) offers an effective and unbiased approach to inferring unknown probability distributions when faced with incomplete information, while neural networks provide the flexibility to learn complex distributions from data. This paper proposes a novel neural network architecture, the MEP-Net, which combines the MEP with neural networks to generate probability distributions from moment constraints. We also provide a comprehensive overview of the fundamentals of the maximum entropy principle, its mathematical formulations, and a rigorous justification for its applicability for non-equilibrium systems based on the large deviations principle. Through fruitful numerical experiments, we demonstrate that the MEP-Net can be particularly useful in modeling the evolution of probability distributions in biochemical reaction networks and in generating complex distributions from data.
Figures
Reference graph
Works this paper leans on
-
[1]
author author I. Goodfellow , author J. Pouget-Abadie , author M. Mirza , author B. Xu , author D. Warde-Farley , author S. Ozair , author A. Courville , \ and\ author Y. Bengio ,\ title title Generative adversarial networks , \ @noop journal journal Communications of the ACM \ volume 63 ,\ pages 139--144 ( year 2020 ) NoStop
work page 2020
-
[2]
author author D. P. \ Kingma ,\ title title Auto-encoding variational bayes , \ @noop journal journal arXiv preprint arXiv:1312.6114 \ ( year 2013 ) NoStop
arXiv 2013
-
[3]
author author A. Vaswani ,\ title title Attention is all you need , \ @noop journal journal Advances in Neural Information Processing Systems \ ( year 2017 ) NoStop
work page 2017
-
[4]
author author J. Ho , author A. Jain , \ and\ author P. Abbeel ,\ title title Denoising diffusion probabilistic models , \ @noop journal journal Advances in neural information processing systems \ volume 33 ,\ pages 6840--6851 ( year 2020 ) NoStop
work page 2020
-
[5]
author author Y. Song , author J. Sohl-Dickstein , author D. P. \ Kingma , author A. Kumar , author S. Ermon , \ and\ author B. Poole ,\ title title Score-based generative modeling through stochastic differential equations , \ @noop journal journal arXiv preprint arXiv:2011.13456 \ ( year 2020 ) NoStop
arXiv 2011
-
[6]
author author E. T. \ Jaynes ,\ title title Information theory and statistical mechanics , \ @noop journal journal Physical Review \ volume 106 ,\ pages 620 ( year 1957 a ) NoStop
work page 1957
-
[7]
author author E. T. \ Jaynes ,\ title title Information theory and statistical mechanics. ii , \ @noop journal journal Physical Review \ volume 108 ,\ pages 171 ( year 1957 b ) NoStop
work page 1957
-
[8]
author author C. Beck ,\ title title Generalised information and entropy measures in physics , \ @noop journal journal Contemporary Physics \ volume 50 ,\ pages 495--510 ( year 2009 ) NoStop
work page 2009
Show all 33 references
-
[9]
Kleidon \ and\ author R
author author A. Kleidon \ and\ author R. D. \ Lorenz ,\ @noop title Non-equilibrium thermodynamics and the production of entropy: life, earth, and beyond \ ( publisher Springer Science & Business Media ,\ year 2004 ) NoStop
2004
-
[10]
Harte ,\ @noop title Maximum entropy and ecology: a theory of abundance, distribution, and energetics \ ( publisher OUP Oxford ,\ year 2011 ) NoStop
author author J. Harte ,\ @noop title Maximum entropy and ecology: a theory of abundance, distribution, and energetics \ ( publisher OUP Oxford ,\ year 2011 ) NoStop
2011
-
[11]
Rocchi ,\ @noop title Reliability is a new science: Gnedenko was right \ ( publisher Springer ,\ year 2017 ) NoStop
author author P. Rocchi ,\ @noop title Reliability is a new science: Gnedenko was right \ ( publisher Springer ,\ year 2017 ) NoStop
2017
-
[12]
Prakash , author S
author author J. Prakash , author S. Mandal , author D. Razansky , \ and\ author V. Ntziachristos ,\ title title Maximum entropy based non-negative optoacoustic tomographic image reconstruction , \ @noop journal journal IEEE Transactions on Biomedical Engineering \ ( year 2019...
2019
-
[13]
Marchenko , author P
author author G. Marchenko , author P. Gagliardini , \ and\ author I. Horenko ,\ title title Towards a computationally tractable maximum entropy principle for nonstationary financial time series , \ @noop journal journal SIAM Journal on Financial Mathematics \ volume 9 ,\ page...
2018
-
[14]
Gresele \ and\ author M
author author L. Gresele \ and\ author M. Marsili ,\ title title On maximum entropy and inference , \ @noop journal journal Entropy \ volume 19 ,\ pages 642 ( year 2017 ) NoStop
2017
-
[15]
Haarnoja , author A
author author T. Haarnoja , author A. Zhou , author P. Abbeel , \ and\ author S. Levine ,\ title title Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , \ @noop journal journal arXiv preprint arXiv:1801.01290 \ ( year 2018 ) NoStop
2018 arXiv
-
[16]
Press \'e , author K
author author S. Press \'e , author K. Ghosh , author J. Lee , \ and\ author K. A. \ Dill ,\ title title Principles of maximum entropy and maximum caliber in statistical physics , \ @noop journal journal Reviews of Modern Physics \ volume 85 ,\ pages 1115 ( year 2013 ) NoStop
2013
-
[17]
author author C. Tsallis ,\ title title Possible generalization of boltzmann-gibbs statistics , \ @noop journal journal Journal of Statistical Physics \ volume 52 ,\ pages 479--487 ( year 1988 ) NoStop
1988
-
[18]
author author S. Y. \ Park \ and\ author A. K. \ Bera ,\ title title Maximum entropy autoregressive conditional heteroskedasticity model , \ @noop journal journal Journal of Econometrics \ volume 150 ,\ pages 219--230 ( year 2009 ) NoStop
2009
-
[19]
author author A. M. \ Kagan , author Y. V. \ Linnik , \ and\ author C. R. \ Rao ,\ @noop title Characterization problems in mathematical statistics \ ( publisher Wiley,New York ,\ year 1973 ) NoStop
1973
-
[20]
Tsallis , author S
author author C. Tsallis , author S. V. \ Levy , author A. M. \ Souza , \ and\ author R. Maynard ,\ title title Statistical-mechanical foundation of the ubiquity of l \'e vy distributions in nature , \ @noop journal journal Physical Review Letters \ volume 75 ,\ pages 3589 ( y...
1995
-
[21]
Anteneodo \ and\ author A
author author C. Anteneodo \ and\ author A. R. \ Plastino ,\ title title Maximum entropy approach to stretched exponential probability distributions , \ @noop journal journal Journal of Physics A: Mathematical and General \ volume 32 ,\ pages 1089 ( year 1999 ) NoStop
1999
-
[22]
author author P. Harremo \"e s ,\ title title Binomial and poisson distributions as maximum entropy distributions , \ @noop journal journal IEEE Transactions on Information Theory \ volume 47 ,\ pages 2039--2041 ( year 2001 ) NoStop
2001
-
[23]
Barzel \ and\ author O
author author B. Barzel \ and\ author O. Biham ,\ title title Binomial moment equations for stochastic reaction systems , \ @noop journal journal Physical Review Letters \ volume 106 ,\ pages 150602 ( year 2011 ) NoStop
2011
-
[24]
author author C. Jia ,\ title title Kinetic foundation of the zero-inflated negative binomial model for single-cell rna sequencing data , \ @noop journal journal SIAM Journal on Applied Mathematics \ volume 80 ,\ pages 1336--1355 ( year 2020 ) NoStop
2020
-
[25]
Doi ,\ title title Onsager’s variational principle in soft matter , \ @noop journal journal Journal of Physics: Condensed Matter \ volume 23 ,\ pages 284118 ( year 2011 ) NoStop
author author M. Doi ,\ title title Onsager’s variational principle in soft matter , \ @noop journal journal Journal of Physics: Condensed Matter \ volume 23 ,\ pages 284118 ( year 2011 ) NoStop
2011
-
[26]
Smadbeck \ and\ author Y
author author P. Smadbeck \ and\ author Y. N. \ Kaznessis ,\ title title A closure scheme for chemical master equations , \ @noop journal journal Proceedings of the National Academy of Sciences \ volume 110 ,\ pages 14261--14265 ( year 2013 ) NoStop
2013
-
[27]
Rathinam , author L
author author M. Rathinam , author L. R. \ Petzold , author Y. Cao , \ and\ author D. T. \ Gillespie ,\ title title Consistency and stability of tau-leaping schemes for chemical reaction systems , \ @noop journal journal Multiscale Modeling & Simulation \ volume 4 ,\ pages 867...
2005
-
[28]
author author D. T. \ Gillespie ,\ title title Exact stochastic simulation of coupled chemical reactions , \ @noop journal journal The journal of physical chemistry \ volume 81 ,\ pages 2340--2361 ( year 1977 ) NoStop
1977
-
[29]
Doi ,\ title title Onsager principle as a tool for approximation , \ @noop journal journal Chinese Physics B \ volume 24 ,\ pages 020505 ( year 2015 ) NoStop
author author M. Doi ,\ title title Onsager principle as a tool for approximation , \ @noop journal journal Chinese Physics B \ volume 24 ,\ pages 020505 ( year 2015 ) NoStop
2015
-
[30]
author author J. W. \ Cahn ,\ title title On spinodal decomposition , \ @noop journal journal Acta metallurgica \ volume 9 ,\ pages 795--801 ( year 1961 ) NoStop
1961
-
[31]
author author P. C. \ Fife ,\ @noop title Dynamics of internal layers and diffusive interfaces \ ( publisher SIAM ,\ year 1988 ) NoStop
1988
-
[32]
Ge , author S
author author H. Ge , author S. Press \'e , author K. Ghosh , \ and\ author K. A. \ Dill ,\ title title Markov processes follow from the principle of maximum caliber , \ @noop journal journal The Journal of chemical physics \ volume 136 ,\ pages 064108 ( year 2012 ) NoStop
2012
-
[33]
author author M. J. \ Hazoglou , author V. Walther , author P. D. \ Dixit , \ and\ author K. A. \ Dill ,\ title title Communication: Maximum caliber is a general variational principle for nonequilibrium statistical mechanics , \ @noop journal journal The Journal of Chemical Ph...
2015
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.