Pith. sign in

REVIEW 3 major objections 5 minor 67 references

To BEE or not to BEE: Estimating more than Entropy with Biased Entropy Estimators

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Chao-Shen and Chao-Wang-Jost entropy estimators converge to ground truth faster than 16 competing estimators across entropy, mutual information, and conditional mutual information, and stay more accurate as sample sizes grow.

desk verdict A useful entropy/MI benchmark whose CMI half is undermined by an algebraically wrong plug-in identity, so the 'regardless of measure' claim doesn't hold as printed. read the letter →

arxiv 2501.11395 v1 pith:MP7WPAPH submitted 2025-01-20 cs.IT cs.SEmath.IT

Ilaria Pia la Torre , David A. Kelly , Hector D. Menendez , David Clark This is my paper · ORCID
classification cs.ITcs.SEmath.IT MSC 94A1762B10
keywords entropyestimationbiasedestimatorsChao-ShenestimatorChao-Wang-Jostmutualinformationconditionalsamplesizedomain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Entropy, mutual information, and conditional mutual information are routinely needed in software engineering, but exact values require knowing the underlying probability distribution, which most practical settings do not provide. The paper runs 18 entropy estimators over 1000 randomly generated joint distributions of six domain sizes and twelve sample sizes, and measures how quickly each estimate flattens out to ground truth and how accurate it is. Its central finding is that the Chao-Shen and Chao-Wang-Jost estimators stabilise with fewer samples than the other 16 and keep the lowest error as samples grow, across all three Shannon measures. If that finding holds, engineers can collect substantially less data for information-flow analysis, feature selection, and leakage measurement without losing estimation quality.

What carries the argument

The carrying object is the pair of sample-coverage entropy estimators: Chao-Shen, a nonparametric estimator that corrects the plug-in count for unseen categories via estimated sample coverage, and Chao-Wang-Jost, its successor built on discovery rates of new categories. They are inserted into the plug-in entropy-difference identities that express mutual information and conditional mutual information as sums and differences of joint entropies: $\hat I_{\mathrm{est}}(X;Y) = \hat H_{\mathrm{est}}(X) + \hat H_{\mathrm{est}}(Y) - \hat H_{\mathrm{est}}(X,Y)$, and the analogous identity in equation (3) for conditional mutual information. The experimental machinery that separates estimators is the flattening-off criterion: a 'safe' sample size, or flattening-off point $F_p$, is the smallest sample size at which the change in mean squared error between consecutive sample sizes stays below 0.1 for all larger samples, and the ratio $F_p/k$ is used to compare how sample-size requirements scale with domain size.

What would settle it

Re-run the same 1000-repetition experiment for conditional mutual information using the standard identity $I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)$ and compare the flattening-off points and MSE rankings; if Chao-Shen and Chao-Wang-Jost no longer lead under the 0.1 flattening-off bound, the paper's 'regardless of measure' conclusion is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that two sample-coverage-based entropy estimators, Chao-Shen and Chao-Wang-Jost, dominate the 18-estimator field when the underlying distribution is unknown. In the experiments, their estimates flatten out to the ground truth at smaller sample sizes than the competition for entropy, mutual information, and conditional mutual information, and the advantage grows as the domain size increases from 256 to 262,144. The 'safe' sample size divided by domain size falls as domains grow for these two estimators, whereas the nearest rivals, GSB88 and SHU, show a roughly constant ratio for mutual information. For larger samples the two also have the lowest mean squared error, with Chao-Wang-Jost slightly ahead in very large domains. The paper's stated conclusion is that, with no distribution knowledge, these estimators are the best default choice for all three Shannon measures.

Load-bearing premise

The load-bearing assumption is that the code implements the correct entropy-combination identity for conditional mutual information; equation (3) in Section 3.2 as printed is not the standard identity, so the CMI results depend on whether the discrepancy is only in the text or in the implementation.

Editorial extensions

If this is right

  • Software engineers who estimate information leakage, feature relevance, or test-suite diversity can use the Chao-Shen and Chao-Wang-Jost estimators as default choices when the data distribution is unknown.
  • The flattening-off point gives a concrete stopping rule: once samples exceed the safe size for the chosen estimator, collecting more data yields less than 0.1 improvement in mean squared error, so data collection effort can be redirected.
  • For mutual information, the ratio of safe sample size to domain size stays near 0.5 for GSB88 and SHU, while it decreases for Chao-Shen and Chao-Wang-Jost, meaning the winning estimators become relatively cheaper as alphabets grow.
  • Estimators such as Bonachela, PYM, and ANSB fail to stabilise within the studied sample-size range and should be avoided when data are limited.
  • For small domains up to about 4096 values, MM, GSB88, and SHU can match or beat the winners on accuracy, but they lose that edge as domain sizes grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The conditional-mutual-information results can be rescued if equation (3) is a typographical error rather than a coding error: if the implementation uses the standard identity $I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)$, the rankings likely survive, but the paper should make that explicit.
  • Because the safe sample size is defined by an MSE change below 0.1, the exact ranking near the boundary is threshold-dependent; a stricter or looser flattening-off bound could shift the flattening-off points, so practitioners should treat the quoted ratios as indicative rather than exact.
  • The experiments use fully random joint distributions, so a natural next test is to apply Chao-Shen and Chao-Wang-Jost to real software artefacts, such as the TriangleType information-leakage channel or fuzzing traces, and see whether the sample-size savings persist under structured dependencies.
  • The exponential decay of $F_p/k$ for the winning estimators suggests a practical heuristic: for domains above a few thousand values, a sample of a few thousand observations may saturate estimation quality, which could be turned into a deployment guideline for leakage measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents an empirical comparison of 18 discrete entropy estimators applied to entropy, mutual information, and conditional mutual information, across six domain sizes and twelve sample sizes, with ground truth derived from randomly generated distributions in small, medium, and large entropy scenarios. The central claim is that the Chao-Shen and Chao-Wang-Jost estimators converge to the ground truth more quickly and more accurately across all three measures and all domain sizes, thereby enabling significant reductions in data collection effort for software engineering applications.

Significance. If the findings hold, this would be a practically valuable comparative study for software engineering and other fields, particularly because all estimators are implemented within the same Julia package (DiscreteEntropy.jl), and the authors provide a figshare link to the full code and evaluation outcomes. The use of a large number of randomly generated distributions, rather than a few hand-picked cases, is a strength. The entropy and mutual information results are internally coherent and the methodology for those measures is sound. However, the incorrect algebraic identity for conditional mutual information in Eq. (3) undermines all CMI-based conclusions and directly contradicts the abstract's 'regardless of the measure used' claim. This is a load-bearing issue that must be resolved before the paper can be accepted.

major comments (3)
  1. [Section 3.2, Eq. (3)] The stated identity for conditional mutual information is algebraically wrong. The paper gives I(X;Y|Z) = H(X,Z) + H(X,Y,Z) - H(Y,Z) - H(Z), but the standard identity is I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z). The printed expression differs from the correct one by 2H(X|Y,Z), a data-dependent quantity that is nonzero except when X is determined by (Y,Z). For example, with Z ~ Bernoulli(1/2) and X,Y conditionally independent fair Bernoulli variables given Z, the true CMI is 0, while the printed formula evaluates to 2 nats. Because all CMI results in Section 4 (Tables 3c, 5, 6a, 6b, and the motivating example in Section 2.1) inherit this formula, the 'regardless of the measure used' conclusion is unsupported as written. The authors must either demonstrate that the deposited code uses the correct identity (in which case Eq. (3) is a typo) or recompute all CMI results with the correct identity.
  2. [Section 4.2, Answer to RQ2 and Table 5] The claim that the safe-sample-size-to-domain-size ratio 'decreases exponentially' for CS and CW is not supported by the data presented in Table 5. The text itself notes that for CMI the ratio for CS is approximately constant ('it now also extends to CS'), and the CW CMI ratios are non-monotonic across k. An exponential decay cannot be inferred from a handful of points, some of which increase. The strong statement in the Answer to RQ2 that 'the amount of data required by CS and CW ... decreases, on average, exponentially with increasing k' should be replaced by a more cautious description of the observed trends.
  3. [Abstract and Section 4.3, Table 6a] The statement that CS and CW perform best 'regardless of domain size' is contradicted by the paper's own accuracy results. Table 6a and the accompanying text state that for H and CMI with k <= 4096, Miller-Madow, Grassberger (1988), and Schurmann either closely approach or are more accurate than CS and CW. The abstract and conclusions should be qualified to reflect that the advantage of CS and CW is most pronounced for large domains and for mutual information, not 'regardless of domain size.'
minor comments (5)
  1. [Section 4.1, entropy paragraph] The sentence 'These values confirm that their estimations converge slightly faster than CS and CW and require slightly larger samples' is self-contradictory; presumably 'slightly slower than CS and CW' was intended.
  2. [Table 5] The table is very difficult to read because the column headings are not clearly separated across the H, MI, and CMI groups. Please reformat it so that each measure's sub-columns are unambiguous and the values can be checked without reference to the running text.
  3. [Section 3.2] The citation 'Grassberger [36]' points to Kraskov, Stögbauer, and Grassberger (2004), not to Grassberger's own entropy papers [26,27]. Please correct the reference to avoid confusion.
  4. [Section 4.1] The flattening-off bound (Fb = 0.1) is a key parameter of the analysis, but its choice is not justified beyond an example. A sensitivity analysis with, say, Fb = 0.05 and Fb = 0.2 would strengthen the conclusions about 'safe' sample sizes.
  5. [Abstract] The phrase 'this discovery enables a significant reduction in data collection effort without compromising performance' overstates the scope of the findings, which are based on simulated distributions and specific experimental settings. Please temper the wording to reflect the conditions under which the recommendations apply.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ranking is an externally benchmarked empirical comparison, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is an empirical ranking of 18 entropy estimators over simulated joint distributions with known ground truth. The ranking is produced by measuring mean squared error and flattening-off points against those ground-truth values, so the outcome is determined by simulation data and estimator implementations, not by the paper's assumptions or by the estimators defining the target quantities. The estimators themselves are taken from external published sources and are not fit to the benchmark data; the paper does not fit any parameter to a subset of outcomes and then rename that fit as a prediction. The only self-references are the authors' DiscreteEntropy.jl implementation package and related prior work, but the package is used as a uniform implementation platform and the benchmark evidence is code-reproducible, externally comparable, and not justified by a self-citation chain. No uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion, and no known result is merely renamed. The apparent sign error in Eq. (3) for conditional mutual information is a substantive correctness and reproducibility threat, and if the code follows the printed formula the CMI results would need re-computation; however, that would be a mathematical error in the evaluation pipeline, not a circularity in which the conclusion is equivalent to its inputs by construction. Therefore, the paper's derivation chain is self-contained for the purpose of this circularity analysis, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The ledger records the experimental design choices that the ranking depends on: the 0.1 threshold, the unspecified distribution generator, and the plug-in identities, one of which is printed incorrectly.

free parameters (2)
  • flattening-off bound (Fb) = 0.1
    Ad hoc threshold on the change in MSE between adjacent sample sizes; defines every flattening-off point and the convergence rankings (Section 4.1). No sensitivity analysis is provided.
  • ground truth scenario definitions (S, M, L) = not specified
    The three scenarios are described only qualitatively (close to minimal, unstructured, close to maximal). The numerical recipe for generating such distributions is not given, so the benchmark's domain coverage is not reproducible.
assumptions (4)
  • domain assumption The plug-in entropy estimators, applied through Eqs. (2) and (3), produce valid estimates of mutual information and conditional mutual information.
    This is the core modeling assumption of the benchmark. For Eq. (3) it is false as printed: the correct identity is H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z).
  • domain assumption The implementations in DiscreteEntropy.jl correctly implement all 18 published estimators with uniform precision.
    All comparisons rely on one package written by two of the authors; implementation errors would shift rankings, and no independent verification or commit hash is given.
  • domain assumption Randomly generated 'general probability distributions' are representative of distributions software engineers face.
    The generation algorithm is not specified, so representativeness is assumed rather than demonstrated.
  • domain assumption Samples are drawn i.i.d. from each generated distribution.
    This is the standard estimation setting used throughout the experiments, stated in Section 3.2, and is not validated against dependent or sequential data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of To BEE or not to BEE: Estimating more than Entropy with Biased Entropy Estimators." pith.science (2026). https://pith.science/paper/MP7WPAPH

@misc{pith2026250111395,
  author       = {Pith},
  title        = {Pith review of: To BEE or not to BEE: Estimating more than Entropy with Biased Entropy Estimators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MP7WPAPH}},
  note         = {Machine review of arXiv:2501.11395}
}
read the original abstract

Entropy estimation plays a significant role in biology, economics, physics, communication engineering and other disciplines. It is increasingly used in software engineering, e.g. in software confidentiality, software testing, predictive analysis, machine learning, and software improvement. However accurate estimation is demonstrably expensive in many contexts, including software. Statisticians have consequently developed biased estimators that aim to accurately estimate entropy on the basis of a sample. In this paper we apply 18 widely employed entropy estimators to Shannon measures useful to the software engineer: entropy, mutual information and conditional mutual information. Moreover, we investigate how the estimators are affected by two main influential factors: sample size and domain size. Our experiments range over a large set of randomly generated joint probability distributions and varying sample sizes, rather than choosing just one or two well known probability distributions as in previous investigations. Our most important result is identifying that the Chao-Shen and Chao-Wang-Jost estimators stand out for consistently converging more quickly to the ground truth, regardless of domain size and regardless of the measure used. They also tend to outperform the others in terms of accuracy as sample sizes increase. This discovery enables a significant reduction in data collection effort without compromising performance.

Figures

Figures reproduced from arXiv: 2501.11395 by the authors.

Figure 2
Figure 2. Mean squared error of the estimations for the MI. The fastest converging methods are highlighted with [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Additional plots for entropy (row 1) and CMI estimations (row 2), with [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 50 canonical work pages

  1. [1]

    Allen and T.M

    E.B. Allen and T.M. Khoshgoftaar. 1999. Measuring coupling and cohesion: an information-theory approach. In Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403) . 119–127. doi:10.1109/METRIC.1999. 809733

    work page doi:10.1109/metric.1999 1999

  2. [2]

    Pierre-Olivier Amblard and Olivier Michel. 2012. The Relation between Granger Causality and Directed Information Theory: A Review. Entropy 15, 1 (Dec. 2012), 113–143. doi:10.3390/e15010113

    work page doi:10.3390/e15010113 2012

  3. [3]

    Hierons, and Mark Harman

    Kelly Androutsopoulos, David Clark, Haitao Dan, Robert M. Hierons, and Mark Harman. 2014. An analysis of the relationship between conditional entropy and failed error propagation in software testing. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY...

    arXiv 2014

  4. [4]

    András Antos and Ioannis Kontoyiannis. 2001. Convergence properties of functional estimates for discrete distributions. Random Structures & Algorithms 19 (2001). https://api.semanticscholar.org/CorpusID:55801502

    work page 2001

  5. [6]

    Evan Archer, Il Memming Park, and Jonathan W. Pillow. 2014. Bayesian Entropy Estimation for Countable Discrete Distributions. Journal of Machine Learning Research15, 81 (2014), 2833–2868. http://jmlr.org/papers/v15/archer14a.html

    work page 2014

  6. [7]

    Francesc Auli-Llinas. 2023. Fast and Efficient Entropy Coding Architectures for Massive Data Compression.Technologies 11, 5 (2023). doi:10.3390/technologies11050132

    work page doi:10.3390/technologies11050132 2023

  7. [8]

    G. P. Basharin. 1959. On a Statistical Estimate for the Entropy of a Sequence of Independent Random Variables.Theory of Probability & Its Applications 4, 3 (1959), 333–336. doi:10.1137/1104033 arXiv:https://doi.org/10.1137/1104033

    work page doi:10.1137/1104033 1959

  8. [9]

    Mario Beraha, Alberto Maria Metelli, Matteo Papini, Andrea Tirinzoni, and Marcello Restelli. 2019. Feature Selection via Mutual Information: New Theoretical Insights. arXiv:1907.07384

    arXiv 2019

Show all 67 references
  1. [10]

    Marcel Böhme, Valentin J. M. Manès, and Sang Kil Cha. 2020. Boosting fuzzer efficiency: an information theoretic perspective. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virt...

    2020

  2. [11]

    Juan A Bonachela, Haye Hinrichsen, and Miguel A Muñoz. 2008. Entropy estimates of small data sets. Journal of Physics A: Mathematical and Theoretical 41, 20 (April 2008), 202001. doi:10.1088/1751-8113/41/20/202001

    2008 doi

  3. [12]

    Marcel Böhme. 2018. STADS: Software Testing as Species Discovery. arXiv:1803.02130

    2018 arXiv

  4. [13]

    Anne Chao and Tsung-Jen Shen. 2003. Nonparametric estimation of Shannon’s diversity index when there are unseen species in sample. Environmental and Ecological Statistics 10 (12 2003), 429–443. doi:10.1023/A:1026096204727

    2003 doi

  5. [14]

    Anne Chao, Y. T. Wang, and Lou Jost. 2013. Entropy and the species accumulation curve: a novel entropy estimator via discovery rates of new species. Methods in Ecology and Evolution 4, 11 (2013), 1091–1100. doi:10.1111/2041-210X.12108 arXiv:https://besjournals.onlinelibrary.wi...

    2013 doi

  6. [15]

    Wainwright, and Michael I

    Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. 2018. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. arXiv:1802.07814

    2018 arXiv

  7. [16]

    Giovanni Cherubin, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. 2019. F-BLEAU: Fast Black-Box Leakage Estimation. 2019 IEEE Symposium on Security and Privacy (SP) (2019), 835–852. https://api.semanticscholar.org/ CorpusID:59599956

    2019

  8. [17]

    David Clark, Robert Feldt, Simon Poulding, and Shin Yoo. 2015. Information Transformation: An Underpinning Theory for Software Engineering. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , Vol. 2. 599–602. doi:10.1109/ICSE.2015.202

    2015 doi

  9. [18]

    Lianet Contreras Rodríguez, Evaristo José Madarro-Capó, Carlos Miguel Legón-Pérez, Omar Rojas, and Guillermo Sosa-Gómez. 2021. Selecting an effective entropy estimator for short sequences of bits and bytes with maximum entropy. Entropy 23, 5 (2021), 561

    2021

  10. [19]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA. To BEE or Not to BEE: Estimating more than Entropy with Biased Entropy Estimators 21

    2006

  11. [20]

    Juan De Gregorio, David Sánchez, and Raúl Toral. 2024. Entropy Estimators for Markovian Sequences: A Comparative Analysis. Entropy 26 (01 2024), 79. doi:10.3390/e26010079

    2024 doi

  12. [21]

    Ping Duan. 2014. Information Theory-based Approaches for Causality Analysis with Industrial Applications . Ph. D. Dissertation

    2014

  13. [22]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature 630 (2024), 625 – 630. https://api.semanticscholar.org/CorpusID:270615909

    2024

  14. [23]

    Farid Feyzi and Saeed Parsa. 2019. Inforence: effective fault localization based on information-theoretic analysis and statistical causal inference. Frontiers of Computer Science 13, 4 (June 2019), 735–759. doi:10.1007/s11704-017-6512-z

    2019 doi

  15. [24]

    J. A. Goguen and J. Meseguer. 1982. Security Policies and Security Models. In 1982 IEEE Symposium on Security and Privacy. 11–11. doi:10.1109/SP.1982.10014

    1982

  16. [25]

    Li Gong. 1990. A note on redundancy in encrypted messages. SIGCOMM Comput. Commun. Rev. 20, 5 (oct 1990), 18–22. doi:10.1145/381906.381918

    1990

  17. [26]

    Peter Grassberger. 1988. Finite sample corrections to entropy and dimension estimates. Physics Letters A 128, 6 (1988), 369–373. doi:10.1016/0375-9601(88)90193-4

    1988 doi

  18. [27]

    Peter Grassberger. 2003. Entropy Estimates from Insufficient Samplings. (07 2003)

    2003

  19. [28]

    Peter Grassberger. 2021. On Generalized Schürmann Entropy Estimators. arXiv:2111.11175 [cs.IT]

    2021 arXiv

  20. [29]

    Jean Hausser and Korbinian Strimmer. 2009. Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks. arXiv:0811.3579 [stat.ML]

    2009 arXiv

  21. [30]

    Hernández and Inés Samengo

    Damián G. Hernández and Inés Samengo. 2022. Inferring a Property of a Large System from a Small Number of Samples. Entropy 24, 1 (2022). doi:10.3390/e24010125

    2022 doi

  22. [31]

    Jonathan Heusser and Pasquale Malacaria. 2010. Quantifying information leaks in software. In Proceedings of the 26th Annual Computer Security Applications Conference (Austin, Texas, USA) (ACSAC ’10). Association for Computing Machinery, New York, NY, USA, 261–269. doi:10.1145/...

    2010

  23. [32]

    D Holste, Ivo Grosse, and Hanspeter Herzel. 1999. Bayes’ estimators of generalized entropies. Journal of Physics A: Mathematical and General 31 (01 1999), 2551. doi:10.1088/0305-4470/31/11/007

    1999 doi

  24. [33]

    David Kelly. 2022. Software Engineering with Incomplete Information . Ph. D. Dissertation. UCL (University College London)

    2022

  25. [34]

    David A Kelly and Ilaria Pia La Torre. 2024. DiscreteEntropy. jl: Entropy Estimation of Discrete Random Variables with Julia. Journal of Open Source Software 9, 103 (2024), 7334

    2024

  26. [35]

    Sanggyun Kim, David Putrino, Soumya Ghosh, and Emery N. Brown. 2011. A Granger Causality Measure for Point Process Models of Ensemble Neural Spiking Activity. PLOS Computational Biology 7, 3 (03 2011), 1–13. doi:10.1371/ journal.pcbi.1001110

    2011

  27. [36]

    Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Phys. Rev. E 69 (Jun 2004), 066138. Issue 6. doi:10.1103/PhysRevE.69.066138

    2004 doi

  28. [37]

    Krichevsky, V

    R. Krichevsky, V. Trofimov, and V. Trofimov. 1981. The performance of universal encoding. IEEE Transactions on Information Theory 27, 2 (1981), 199–207. doi:10.1109/TIT.1981.1056331

    1981

  29. [38]

    Haodong Li, Guosheng Xu, Liu Wang, Xusheng Xiao, Xiapu Luo, Guoai Xu, and Haoyu Wang. 2024. MalCertain: Enhancing Deep Neural Network Based Android Malware Detection by Tackling Prediction Uncertainty. InProceedings of the IEEE/ACM 46th International Conference on Software Eng...

    2024

  30. [39]

    Macke, Iain Murray, and Peter E

    Jakob H. Macke, Iain Murray, and Peter E. Latham. 2013. Estimation Bias in Maximum Entropy Models. Entropy 15, 8 (2013), 3109–3129. doi:10.3390/e15083109

    2013 doi

  31. [40]

    Ibrahim Mesecan, Daniel Blackwell, David Clark, Myra Cohen, and Justyna Petke. 2023. Keeping Secrets: Multi- objective Genetic Improvement for Detecting and Reducing Information Leakage. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Enginee...

    2023

  32. [41]

    Cohen, and Justyna Petke

    Ibrahim Mesecan, Daniel Blackwell, David Clark, Myra B. Cohen, and Justyna Petke. 2021. HyperGI: Automated Detection and Repair of Information Flow Leakage. arXiv:2108.12075 [cs.SE] https://arxiv.org/abs/2108.12075

    2021 arXiv

  33. [42]

    G Miller and G Miller. 1955. Note on the bias of information estimates. https://api.semanticscholar.org/CorpusID: 125662170

    1955

  34. [43]

    Verya Monjezi, Ashutosh Trivedi, Gang Tan, and Saeid Tizpaz-Niari. 2023. Information-Theoretic Testing and Debugging of Fairness Defects in Deep Neural Networks. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1571–1582. doi:10.1109/ICSE48619.2023.00136

    2023

  35. [44]

    Jugurta Montalvao, Romis Attux, and D.G. Silva. 2012. Simple entropy estimator for small datasets. Electronics Letters 48 (08 2012), 1059–1061. doi:10.1049/el.2012.2002

    2012

  36. [45]

    Ilya Nemenman, William Bialek, and Rob de Ruyter van Steveninck. 2004. Entropy and information in neural spike trains: Progress on the sampling problem. Physical Review E 69, 5 (May 2004). doi:10.1103/physreve.69.056111 22 Ilaria Pia la Torre, David A. Kelly, Héctor D. Menénde...

    2004 doi

  37. [46]

    Ilya Nemenman, Fariel Shafee, and William Bialek. 2002. Entropy and inference, revisited. arXiv:physics/0108025 [physics.data-an]

    2002 arXiv

  38. [47]

    Liam Paninski. 2003. Estimation of Entropy and Mutual Information. Neural Computation 15, 6 (06 2003), 1191–1253. doi:10.1162/089976603321780272

    2003 doi

  39. [48]

    Hierons, and David Clark

    Krishna Patel, Robert M. Hierons, and David Clark. 2022. An information theoretic notion of software testability. Information and Software Technology 143 (2022), 106759. doi:10.1016/j.infsof.2021.106759

    2022

  40. [49]

    Justyna Petke, David Clark, and William B. Langdon. 2021. Software robustness: a survey, a theory, and prospects. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) ...

    2021

  41. [50]

    Assaf Pinchas, Irad Ben-Gal, and Amichai Painsky. 2024. A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems. Entropy 26, 5 (2024). doi:10.3390/e26050369

    2024 doi

  42. [51]

    Christopher Quinn, Todd Coleman, Negar Kiyavash, and Nicholas Hatsopoulos. 2011. Estimating the directed infor- mation to infer causal relationships in ensemble neural spike train recordings. Journal of computational neuroscience 30 (02 2011), 17–44. doi:10.1007/s10827-010-0247-2

    2011 doi

  43. [52]

    Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. 2020. FR-Train: A Mutual Information-Based Approach to Fair and Robust Training. arXiv:2002.10234

    2020 arXiv

  44. [53]

    Mark S Roulston. 1999. Estimating the errors on measured entropy and mutual information. Physica D: Nonlinear Phenomena 125, 3 (1999), 285–294. doi:10.1016/S0167-2789(98)00269-3

    1999 doi

  45. [54]

    Donald B Rubin. 1981. The bayesian bootstrap. The annals of statistics (1981), 130–134

    1981

  46. [55]

    Thomas Schürmann. 2004. Bias analysis in entropy estimation. Journal of Physics A: Mathematical and General 37, 27 (June 2004), L295–L301. doi:10.1088/0305-4470/37/27/l02

    2004 doi

  47. [56]

    C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 3 (1948), 379–423. doi:10.1002/j.1538-7305.1948.tb01338.x

    1948

  48. [57]

    Qingkai Shi, Zhenyu Chen, Chunrong Fang, Yang Feng, and Baowen Xu. 2016. Measuring the Diversity of a Test Set With Distance Entropy. IEEE Transactions on Reliability 65, 1 (2016), 19–27. doi:10.1109/TR.2015.2434953

    2016

  49. [58]

    Ricardo De Matos Simoes and Frank Emmert-Streib. 2011. Influence of Statistical Estimators of Mutual Information and Data Heterogeneity on the Inference of Gene Regulatory Networks. PLoS ONE 6 (2011), 29279. Issue 12. doi:10.1371/journal.pone.0029279

    2011 doi

  50. [59]

    Alessandro Treves and Stefano Panzeri. 1995. The Upward Bias in Measures of Information Derived from Limited Data Samples. Neural Computation 7, 2 (1995), 399–407. doi:10.1162/neco.1995.7.2.399

    1995 doi

  51. [60]

    Stanislaw Trybula. 1958. Some Problems of Simultaneous Minimax Estimation. The Annals of Mathematical Statistics 29, 1 (1958), 245 – 253. doi:10.1214/aoms/1177706722

    1958

  52. [61]

    Vassilev and T

    A. Vassilev and T. A. Hall. 2014. The Importance of Entropy to Information Security.Computer 47, 02 (feb 2014), 78–81. doi:10.1109/MC.2014.47

    2014 doi

  53. [62]

    Robin Willink and Rod White. 2012. - 1-Disentangling Classical and Bayesian Approaches to Uncertainty Analysis. https://api.semanticscholar.org/CorpusID:15546936

    2012

  54. [63]

    Hirotoshi Yasuoka and Tachio Terauchi. 2010. Quantitative Information Flow - Verification Hardness and Possibilities. In 2010 23rd IEEE Computer Security Foundations Symposium

    2010

  55. [64]

    Shin Yoo, Mark Harman, and David Clark. 2013. Fault localization prioritization: Comparing information-theoretic and coverage-based approaches. ACM Trans. Softw. Eng. Methodol. 22, 3, Article 19 (jul 2013), 29 pages. doi:10.1145/ 2491509.2491513

    2013

  56. [65]

    Tao Yuan and S. Joe Qin. 2014. Root cause diagnosis of plant-wide oscillations using Granger causality. Journal of Process Control 24, 2 (2014), 450–459. doi:10.1016/j.jprocont.2013.11.009 ADCHEM 2012 Special Issue

    2014 doi

  57. [66]

    Xing Zhang, Jiongyi Chen, Chao Feng, Ruilin Li, Wenrui Diao, Kehuan Zhang, Jing Lei, and Chaojing Tang. 2022. DeFault: mutual information-based crash triage for massive crashes. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvan...

    2022

  58. [67]

    Zhiyi Zhang. 2012. Entropy Estimation in Turing’s Perspective. Neural computation 24 (02 2012), 1368–89. doi:10. 1162/NECO_a_00266

    2012

  59. [68]

    Behrouz Zolfaghari, Khodakhast Bibak, and Takeshi Koshiba. 2022. The Odyssey of Entropy: Cryptography. Entropy 24, 2 (2022). doi:10.3390/e24020266

    2022 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.