Properties of the Shannon, R\'{e}nyi and other entropies: dependence in parameters, robustness in distributions and extremes

Iryna Bodnarchuk; Kostiantyn Ralchenko; Yuliya Mishura

REVIEW 4 major objections 4 minor 23 references

Properties of the Shannon, R\'{e}nyi and other entropies: dependence in parameters, robustness in distributions and extremes

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives closed-form entropy formulas for six entropy families on five common distributions and proves their parameter monotonicity, convergence, and extremal properties.

desk verdict Useful entropy formula reference with correctable algebra slips in the Sharma-Mittal entries; send to referee after requiring the fixes. read the letter →

arxiv 2411.15817 v1 pith:HWQPHCBI submitted 2024-11-24 cs.IT math.ITmath.PR

Iryna Bodnarchuk , Yuliya Mishura , Kostiantyn Ralchenko This is my paper · ORCID

classification cs.ITmath.ITmath.PR MSC 94A1762B1060E05

keywords ShannonentropyRényigeneralizedTsallisSharma–MittalKullback–LeiblerdivergencegammadistributionHadamardinequality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to turn entropy computation into a formula lookup: for the gamma, exponential, chi-squared, Laplace and log-normal distributions it derives closed-form expressions for the six entropy functionals (Shannon, Rényi, the two generalized Rényi versions, Tsallis and Sharma–Mittal), for the modified Shannon entropy, and for the Kullback–Leibler divergence, all written directly in terms of the distribution parameters. It then reads parameter dependence off those formulas, giving a new proof that Poisson entropy increases with intensity, a proof that gamma Shannon entropy decreases in the rate $\lambda$ and increases in the shape $\mu$, and a full $\lambda$-profile for every exponential entropy. It also proves two convergence results: the conditional (zero-truncated) negative binomial entropy converges to the logarithmic entropy as $r\to0$, and the binomial entropy converges to the Poisson entropy when $np_n\to\lambda$. Finally, for Gaussian vectors with fixed variances it uses Hadamard's determinant inequality to locate the maximum and minimum of the Shannon entropy and to say which covariance structures realize them. If the formulas are correct, the paper gives a reference table that replaces numerical integration in applications using these entropy measures.

What carries the argument

The load-bearing machinery is a chain of explicit integral evaluations: each density is substituted into the defining entropy integrals, and the resulting Gamma integrals $\int_0^\infty x^{a-1}e^{-x}\,dx=\Gamma(a)$ and their differentiated versions produce closed forms in $\Gamma$, digamma $\psi$, and trigamma $\psi'$. Parameter monotonicity is then read from estimates such as $\psi'(z)=\sum_{n=0}^\infty (z+n)^{-2}$ and the bound $\psi'(z)<1/z^2+1/z$ used in Theorem 4.4. For the discrete convergence theorems, the mechanism is dominated convergence: the terms $p_n(k)\log(1/p_n(k))$ are bounded by summable sequences built from elementary binomial and negative-binomial inequalities. For the Gaussian extremal problem, the mechanism is Hadamard's inequality $\det A\le\prod_i a_{ii}$ for positive semidefinite matrices with fixed diagonal, with equality only for diagonal matrices; inserted into the Gaussian vector entropy formula $\frac n2(1+\log2\pi)+\frac12\log\det A$, it yields the maximum at independent components and the minimum at rank-one covariance.

What would settle it

For the gamma distribution with $\mu=2$, $\lambda=1$, $\alpha=2$, $\beta=3$, compute the Sharma–Mittal entropy numerically from the defining integral and compare it with Proposition 3.2(6): the integral gives $15/32$ and the printed formula gives $255/512$, so this single substitution is enough to decide whether that closed form is correct.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that all six entropy functionals plus the modified Shannon entropy and the Kullback–Leibler divergence are explicitly evaluable for the five named distribution families, and that the resulting expressions support exact statements about monotonicity, limits and extremes. The gamma calculations are representative: they express the Shannon entropy as $H_{\mathrm{SH}}^{(\mathrm{gamma})}(\lambda,\mu)=-\log\lambda+\log\Gamma(\mu)+\mu-\psi(\mu)(\mu-1)$, and the other gamma entropies through Gamma-function ratios; the exponential and chi-squared families then follow by specialization $\mu=1$ and $\lambda=1/2,\mu=\nu/2$, while the log-normal and Laplace cases are computed from moment formulas. The paper further claims that the gamma Shannon entropy decreases in $\lambda$ and increases in $\mu$ over the whole parameter range, that each exponential entropy has a precisely described $\lambda$-behaviour with identified zero crossings, that Shannon entropies converge along the two discrete distribution limits, and that among centered Gaussian vectors with fixed variances the entropy is maximized exactly by uncorrelated (independent) components and minimized by perfectly correlated components.

Load-bearing premise

Everything in Sections 4 and 5 that is built on the Section 3 formulas assumes every displayed closed-form expression is a correct simplification of its defining integral; if any one formula mis-simplifies a Gamma-function ratio, the monotonicity and convergence statements that rely on it inherit the error.

Editorial extensions

If this is right

For any of the five distributions, Shannon, Rényi, generalized Rényi, Tsallis and Sharma–Mittal entropies can be evaluated by substituting parameters into a closed formula, eliminating numerical quadrature in those cases.
The gamma monotonicity theorem fixes the sign of every partial derivative: increasing the rate $\lambda$ always lowers entropy and increasing the shape $\mu$ always raises it, for all $\lambda,\mu>0$.
The exponential results establish a strict ordering of entropy scales, with $H_R>H_{SH}$ for $\alpha<1$ and $H_R<H_{SH}$ for $\alpha>1$, and locate the exact zero crossings of Tsallis and Sharma–Mittal entropies.
The convergence theorems justify approximating binomial entropy by Poisson entropy and zero-truncated negative binomial entropy by logarithmic entropy, with explicit dominated-convergence bounds rather than numerical observation.
For Gaussian vectors with fixed variances, the entropy range is exactly the interval from the rank-one correlation case (determinant zero) to the diagonal case (determinant $\prod_i a_{ii}$), so every covariance matrix's entropy lies between these two computable extremes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same Gamma-integral technique should extend to other densities of the form $x^{a-1}e^{-bx}$ times a polynomial in $\log x$, such as Weibull or inverse-gamma laws; the paper does not compute those, but nothing in the method appears special to the five chosen families.
The two convergence theorems suggest Shannon entropy is continuous along the exhibited distributional limits, yet the paper does not prove a general continuity principle; finding the broadest class of discrete limits with entropy convergence is a natural next step.
The Gaussian vector extremal result, combined with the fractional Gaussian noise discussion, suggests a finite-dimensional sandwich for the entropy of Gaussian processes with prescribed marginal variances, though the paper only treats fixed-dimension vectors.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: the paper is a useful, mostly correct reference for entropy formulas, but its advertised reliability currently fails in exactly the place the stress-test flags: the Sharma-Mittal closed forms for gamma, chi-squared, and Laplace are wrong as printed. Those are localized algebra slips, not load-bearing for Sections 4–6, but they invalidate the paper's central claim until fixed.

What is genuinely new: explicit Tsallis, generalized Renyi, and Sharma-Mittal expressions for gamma, chi-squared, and lognormal that I haven't seen assembled elsewhere; a new, concise proof that Poisson entropy increases in λ; and two convergence results (binomial-to-Poisson, negative-binomial-to-logarithmic) proved by finding summable dominants. The proof of Poisson monotonicity is neat and the limit argument in Lemma 4.2 is correct. The lognormal derivations are clean and the KL divergence formulas are standard but correctly re-derived.

The soft spots are real but countable. I checked the stress-test's arithmetic: for gamma with µ=2, λ=1, α=2, β=3, the defining integral gives H_SM = 15/32 while Proposition 3.2(6) prints 255/512. The gamma-function ratio is inverted. Proposition 3.6(6) inherits the same inversion after the chi-squared substitution, and Proposition 3.9(6) drops a 2^{1-β} factor that J^{(La)} forces. Any reader using these formulas will get wrong numbers. Also, the proof of Proposition 5.4 is not complete as printed: after bounding 1/(Γ(r)(1-p^r)) it jumps to the final bound on P log(1/P) without showing the lower bound side; the displayed argument is a fragment. This is likely repairable but as it stands it is not a proof. The convergence theorem may still be true, but a referee should ask for the missing details.

The rest is solid. Section 4's dependence results on gamma entropy are correct, Section 6's determinant extremes are straightforward Hadamard applications, and the authors are honest about the unproved fGn hypothesis. The self-citations provide context rather than carrying the derivations.

Who this is for: applied information theorists and statistical physicists who want a single formula sheet. It is not a breakthrough, but it is a service. With the three corrections and the completed proof, it deserves publication. I would send it to a serious referee rather than desk-reject; the referee's job is narrow and mechanical.

Referee Report

4 major / 4 minor

Summary. The paper defines six entropy functionals (Shannon, Rényi, generalized Rényi, Tsallis, generalized Rényi with two parameters, and Sharma–Mittal), the Kullback–Leibler divergence, and a modified Shannon entropy, and derives closed-form expressions for gamma, exponential, chi-squared, Laplace, and log-normal distributions. It then studies parameter monotonicity for Poisson, gamma, and exponential entropies, proves convergence of Shannon entropies under negative-binomial-to-logarithmic and binomial-to-Poisson distributional limits, and gives extremal results for the determinant of covariance matrices of Gaussian vectors with fixed variances. The intended contribution is a reliable reference table of entropy formulas together with new monotonicity, robustness, and extremal results.

Significance. If the Section 3 table were correct, the paper would serve as a useful formula reference for information-theoretic and stochastic-modelling applications. The paper does contain genuine self-contained contributions: a new proof that Poisson entropy increases in its intensity, including an exact evaluation of the limiting derivative; a monotonicity proof for the two-parameter gamma Shannon entropy; dominated-convergence arguments for entropy limits; and clean determinant-extremal arguments for Gaussian vectors in Section 6. The Poisson proof is not circular and goes beyond the earlier argument in [5]. However, the paper's central formula-table claim is currently undermined by three algebraic errors in the Sharma–Mittal entries in Section 3, and the proof of Proposition 5.4 is not complete as printed.

major comments (4)

[Section 3.1, Proposition 3.2(6)] The printed Sharma–Mittal formula for the gamma distribution inverts the Gamma-function ratio. Using the formula for J = ∫ p^α(x)dx obtained in Proposition 3.2(2) and the definition (2.5) with q = (1−β)/(1−α), the correct expression is H_SM = (1/(1−β))[ λ^{β−1} α^{(α(1−μ)−1)(1−β)/(1−α)} Γ^{q}(α(μ−1)+1)/Γ^{αq}(μ) − 1 ]. The displayed formula in the paper has Γ^{αq}(μ) in the numerator and Γ^{q}(α(μ−1)+1) in the denominator. Direct evaluation of the defining integral for μ=2, λ=1, α=2, β=3 gives H_SM = 15/32, while the printed expression yields 255/512. This error directly contradicts the paper's advertised role as a reliable closed-form reference.
[Section 3.3, Proposition 3.6(6)] The chi-squared Sharma–Mittal formula inherits the same inverted Gamma-function ratio after substituting λ=1/2 and μ=ν/2 into the gamma formula. The corrected expression should contain Γ^{q}(α(ν/2−1)+1) divided by Γ^{αq}(ν/2), not the reciprocal placement displayed in the paper. This entry is therefore also quantitatively incorrect for generic parameter choices.
[Section 3.4, Proposition 3.9(6)] The Laplace Sharma–Mittal formula omits the factor 2^{1−β}. With J^{(La)} = λ^{α−1}/(2^{α−1}α) and q=(1−β)/(1−α), one has J^q = λ^{β−1} 2^{1−β} α^{(1−β)/(α−1)}. The printed formula lacks the factor 2^{1−β}, so it is wrong for every β ≠ 1, including the numerical check β=3.
[Section 5.1, Proposition 5.4] The proof of Proposition 5.4 is fragmented as printed. The opening paragraph states that both lower and upper bounds for P_{p,r}(k) are needed, but the displayed text after the first paragraph moves directly to an upper estimate, and the paragraph beginning 'Also, for 0<r<1...' together with the final domination series appears after the proof's QED marker. No explicit lower bound for P_{p,r}(k) is displayed. The proof should be rewritten as one continuous argument with the lower and upper bounds stated explicitly and the uniform-in-r domination shown in full.

minor comments (4)

[Proof of Proposition 3.2] In the proof of Proposition 3.2, after part 3, the sentence 'The statements 4)–6) immediately follow from (3.1)' cannot be correct, because (3.1) is the Shannon entropy formula. The Tsallis and Sharma–Mittal formulas follow from the integral J computed in part 2 (or from the corresponding expressions obtained there), so this cross-reference should be corrected.
[Typesetting] Several indicator functions are rendered as '/BD(0,+∞)(x)' or similar artifacts, e.g., in Definition 3.1; the typesetting should be cleaned to use standard indicator notation such as 1_{(0,+∞)}(x).
[References] Reference [17] contains corrupted text ('Witu/suppress la') and should be repaired; the correct name appears to be Wituła. A few other bibliographic entries also have minor capitalization or formatting issues.
[Lemma 4.2] The proof of Lemma 4.2 applies l'Hôpital's rule and term-by-term differentiation of a power series without explicitly justifying the interchange; this is standard for exponential series but should be stated for completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all closed forms are obtained by direct integration from the defining entropy formulas; self-citations are contextual or re-proved.

full rationale

The derivation chain is self-contained. Section 3 evaluates the defining integrals (2.1)-(2.5) directly for each distribution, and the resulting formulas are used in Sections 4-6. Theorem 4.4 is proved by differentiating the derived formula (3.1); Theorem 4.5 differentiates the derived exponential formulas (3.10)-(3.12); Propositions 5.4 and 5.6 use dominated convergence with explicit bounds; Section 6 relies on the external Hadamard inequality [17] and an explicit Gaussian construction. The only result attributed to prior work, the monotonicity of Poisson entropy, is explicitly re-proved in the paper with a new proof and Lemma 4.2, so the citation to [5] is not load-bearing. Self-citations [12] and [13] are contextual: [12] supplies already-known Gaussian entropy values and [13] is referenced for the fGn covariance and an explicitly labeled unproved hypothesis. No parameter is fitted to data and then renamed as a prediction, and no target result is assumed in place of a derivation. The algebraic simplification errors in Proposition 3.2(6), Proposition 3.6(6), and Proposition 3.9(6) noted by the reader are correctness defects, not circularity, because the derivations still start from the defining integrals rather than assuming the claimed formulas.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The derivations are self-contained from the entropy definitions plus classical analytic facts; there are no fitted constants. The only new construct is the modified Shannon entropy, which is a definition rather than an empirical entity.

assumptions (5)

domain assumption Convergence of the defining integrals for ∫p^α under the stated parameter restrictions
Gamma and chi-squared Rényi-type entropies require α(μ−1)>−1; the paper relies on this in items 2 to 6 of Proposition 3.2 and Proposition 3.6.
standard math Standard gamma and digamma identities, including ψ(z+1)=ψ(z)+1/z and the series representation for ψ′
Used in the proofs of Proposition 3.2 and in the gamma monotonicity proof of Theorem 4.4.
standard math Hadamard's determinant inequality for positive semidefinite matrices
Section 6.1 uses this to bound det(A) by the product of diagonal variances for a covariance matrix with fixed diagonal.
standard math Dominated convergence theorem for passing entropy limits through infinite sums
Propositions 5.4 and 5.6 rely on summable dominating sequences to exchange limits with infinite series.
standard math Jensen inequality for the non-negativity of Kullback-Leibler divergence
Remark 2.2 and Remark 3.3 use convexity of x log x to prove KL divergence is nonnegative.

invented entities (1)

Modified Shannon entropy H_SH,M
purpose: A rescaled entropy functional, defined via ~p(x)=p(x)/M, that is nonnegative for bounded densities.
Defined in Definition 2.3; it is not a normalized probability entropy because ∫~p dx=1/M is not 1, and no external benchmark is given for its usefulness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Properties of the Shannon, R\'{e}nyi and other entropies: dependence in parameters, robustness in distributions and extremes." pith.science (2026). https://pith.science/paper/HWQPHCBI

@misc{pith2026241115817,
  author       = {Pith},
  title        = {Pith review of: Properties of the Shannon, R\'enyi and other entropies: dependence in parameters, robustness in distributions and extremes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWQPHCBI}},
  note         = {Machine review of arXiv:2411.15817}
}

read the original abstract

We calculate and analyze various entropy measures and their properties for selected probability distributions. The entropies considered include Shannon, R\'enyi, generalized R\'enyi, Tsallis, Sharma-Mittal, and modified Shannon entropy, along with the Kullback-Leibler divergence. These measures are examined for several distributions, including gamma, chi-squared, exponential, Laplace, and log-normal distributions. We investigate the dependence of the entropy on the parameters of the respective distribution. We also study the convergence of Shannon entropy for certain probability distributions. Furthermore, we identify the extreme values of Shannon entropy for Gaussian vectors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

[5]

Braiman, A

V. Braiman, A. Malyarenko, Y. Mishura, and Y. A. Rudyk. Pr operties of Shannon and R´ enyi entropies of the Poisson distribution as the functions of intensity paramet er. Nonlinear Anal. Model. Control , 29(4):802–815, 2024

work page 2024
[1]

F. J. Anscombe. Sampling theory of the negative binomial and logarithmic series distributions. Biometrika, 37(3/4):358–382, 1950

work page 1950
[2]

Banna, Y

O. Banna, Y. Mishura, K. Ralchenko, and S. Shklyar. Fractional Brownian motion. Approximations and projections. Wiley-ISTE, 2019

work page 2019
[3]

R. Bellman. Introduction to Matrix Analysis . Classics in Applied Mathematics. SIAM, 2nd edition, 1997

work page 1997
[4]

Bianconi

G. Bianconi. The entropy of randomized network ensemble s. Europhysics Letters, 81(2):28005, dec 2007

work page 2007
[6]

Buryak and Y

F. Buryak and Y. Mishura. Convexity and robustness of the R´ enyi entropy.Mod. Stoch. Theory Appl. , 8(3):387–412, 2021

work page 2021
[7]

Cal` ı, M

C. Cal` ı, M. Longobardi, and J. Ahmadi. Some properties of cumulative Tsallis entropy. Physica A: Statistical Mechanics and its Applications , 486:1012–1021, 2017

work page 2017
[8]

D. Frenkel. Order through entropy. Nature materials, 14(1):9–12, 2015

work page 2015

Show all 23 references

[9]

I. S. Gradshteyn and I. M. Ryzhik. Table of integrals, series, and products . Elsevier/Academic Press, Ams- terdam, seventh edition, 2007. Edited by Alan Jeﬀrey and Dan iel Zwillinger

2007
[10]

Hadamard

J. Hadamard. R´ esolution d’une question relative aux determinants. Bull. Sciences Math. , 17:240–246, 1893
[11]

Z. He, H. Ahmadzade, K. Rezaei, H. Rezaei, and H. Naderi. Tsallis entropy of uncertain random variables and its application. Soft Computing , 25:11735–11743, 2021

2021
[12]

Malyarenko, Y

A. Malyarenko, Y. Mishura, K. Ralchenko, and Y. A. Rudyk . Properties of various entropies of Gaussian distribution and comparison of entropies of fractional pro cesses. Axioms, 12(11):1026, 2023

2023
[13]

Malyarenko, Y

A. Malyarenko, Y. Mishura, K. Ralchenko, and S. Shklyar . Entropy and alternative entropy functionals of fractional Gaussian noise as the functions of Hurst index . Fractional Calculus and Applied Analysis , 26(3):1052–1081, 2023

2023
[14]

Mauricio

S. Mauricio. Linking the negative binomial and logarit hmic series distributions via their associated series. Revista Colombiana de Estad ´ ıstica, 31:311–319, 2008

2008
[15]

Milev, P

M. Milev, P. N. Inverardi, and A. Tagliani. Moment infor mation and entropy evaluation for probability densities. Applied Mathematics and Computation , 218(9):5782–5795, 2012. PROPERTIES OF THE SHANNON, R ´ENYI AND OTHER ENTROPIES 25

2012
[16]

Mishura, K

Y. Mishura, K. Ralchenko, P. Zelenko, and V. Zubchenko. Properties of the entropic risk measure EVaR in relation to selected distributions. Mod. Stoch. Theory Appl. , 11(4):373–394, 2024

2024
[17]

R´ o˙ za´ nski, R

M. R´ o˙ za´ nski, R. Witu/suppress la, and E. Hetmaniok. More subtle versions of the Hadamard inequality. Linear Algebra and its Applications , 532:500–511, 2017

2017
[18]

V. P. Singh. Entropy theory and its application in environmental and wat er engineering. John Wiley & Sons, 2013

2013
[19]

R. L. Stratonovich. Theory of Information and its Value . Springer international Publishing: Cham, Switzer- land, 2020. Edited by R. V. Belavkin, P. M. Pardalos, and J. C. Principe

2020
[20]

A. Ullah. Entropy, divergence and distance measures wi th econometric applications. Journal of Statistical Planning and Inference , 49(1):137–162, 1996

1996
[21]

J. M. Van Campenhout and T. Cover. Maximum entropy and co nditional probability. IEEE Transactions on Information Theory, 27(4):483–489, 1981

1981
[22]

H. Yan, H. Li, and Q. Sun. Diﬀerential game analysis of mu lti-party rescue eﬀorts during major traﬃc accidents based on entropy optimization. Journal of Industrial and Management Optimization , 21(1):312– 334, 2025

2025
[23]

Zhang, F

F. Zhang, F. Meng, F. Ma, Q. Yin, and Y. Zhou. Time correla tion entropy: A novel multitemporal polsar feature and its application in salt lake classiﬁcation. IEEE Transactions on Geoscience and Remote Sensing , 62:1–14, 2024

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [5]

Braiman, A

V. Braiman, A. Malyarenko, Y. Mishura, and Y. A. Rudyk. Pr operties of Shannon and R´ enyi entropies of the Poisson distribution as the functions of intensity paramet er. Nonlinear Anal. Model. Control , 29(4):802–815, 2024

work page 2024

[2] [1]

F. J. Anscombe. Sampling theory of the negative binomial and logarithmic series distributions. Biometrika, 37(3/4):358–382, 1950

work page 1950

[3] [2]

Banna, Y

O. Banna, Y. Mishura, K. Ralchenko, and S. Shklyar. Fractional Brownian motion. Approximations and projections. Wiley-ISTE, 2019

work page 2019

[4] [3]

R. Bellman. Introduction to Matrix Analysis . Classics in Applied Mathematics. SIAM, 2nd edition, 1997

work page 1997

[5] [4]

Bianconi

G. Bianconi. The entropy of randomized network ensemble s. Europhysics Letters, 81(2):28005, dec 2007

work page 2007

[6] [6]

Buryak and Y

F. Buryak and Y. Mishura. Convexity and robustness of the R´ enyi entropy.Mod. Stoch. Theory Appl. , 8(3):387–412, 2021

work page 2021

[7] [7]

Cal` ı, M

C. Cal` ı, M. Longobardi, and J. Ahmadi. Some properties of cumulative Tsallis entropy. Physica A: Statistical Mechanics and its Applications , 486:1012–1021, 2017

work page 2017

[8] [8]

D. Frenkel. Order through entropy. Nature materials, 14(1):9–12, 2015

work page 2015