{"id":"ecd85f2a-dca2-4854-81e5-3f72d83e0661","arxiv_id":"2501.11395","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark of 18 biased entropy estimators on H, MI, and CMI concludes that Chao-Shen and Chao-Wang-Jost are fastest to converge and most accurate, though the CMI computation appears mis-specified.","lead":"Tests of 18 entropy estimators on entropy, mutual information, and conditional mutual information report that Chao-Shen and Chao-Wang-Jost converge fastest and are most accurate. The practical promise is smaller data-collection budgets for software analysis, but the paper's conditional mutual information formula is algebraically wrong, which invalidates that part of the evidence.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (3) for conditional mutual information has the wrong sign pattern, so every CMI result is suspect unless the deposited code is shown to implement the standard identity.","rationale":"The reader's weakest assumption is exactly the load-bearing spot: Eq. (3) is algebraically wrong, and the paper's headline claim explicitly covers 'regardless of the measure used,' which requires the CMI results to be valid. I checked the identity: the printed expression H(X,Z)+H(X,Y,Z)-H(Y,Z)-H(Z) differs from the correct H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z) by 2H(X|Y,Z), so the error is not a harmless sign slip in a definition; it changes every CMI estimate by a data-dependent term. A simple conditional-independence example gives true CMI 0 and printed-formula value 2 nats, making the failure concrete. The one check that settles the matter is to inspect the deposited Julia code, since the paper itself does not show the implementation. If the code uses the standard identity, this specific concern is a typo; if it uses the printed formula, the CMI results, including the RQ1/RQ3 answers and the abstract's universal claim, are invalid. I am not raising the arbitrary flattening-off bound, the unspecified distribution generator, or the missing error bars as the primary issue because those are methodological weaknesses that could be fixed in a revision, whereas a wrong formula in the core estimation procedure is directly falsifiable and is the single most load-bearing threat to the central claim. Given the reader already rejected the manuscript on this basis, my assessment leaves the verdict unchanged.","tokens_in":22653,"tokens_out":5644,"duration_ms":56603,"concrete_test":"Download the code linked in Section 3.2 (figshare.com/s/24fae7566ca00662d538), locate the conditional mutual information implementation, and run a unit check against the standard identity I(X;Y|Z)=H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z) on the distribution where Z~Bernoulli(1/2) and X,Y are conditionally independent fair Bernoulli variables given Z. If the implementation returns 0, the code is correct and the concern reduces to a typo in Eq. (3); if it returns 2 nats, the CMI experiments must be rerun with the corrected formula before the paper's cross-measure claim can be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that Chao-Shen and Chao-Wang-Jost converge more quickly and are more accurate 'regardless of ... the measure used.' For CMI, the paper's Eq. (3) states I(X;Y|Z) = H(X,Z)+H(X,Y,Z)-H(Y,Z)-H(Z). The standard identity is I(X;Y|Z) = H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z). The printed formula differs from the correct one by 2H(X|Y,Z), a data-dependent quantity that is nonzero except when X is determined by (Y,Z). For example, with Z~Bernoulli(1/2) and X,Y conditionally independent fair Bernoulli variables given Z, the true CMI is 0, while Eq. (3) evaluates to 2 nats. Because every CMI table and figure in Section 4 inherits this formula, the 'regardless of measure' conclusion is unsupported as written. The deposited Julia code could resolve the ambiguity: if it uses the correct identity, Eq. (3) is a typo and the CMI evidence stands; if it follows the printed formula, all CMI results require re-computation and the central claim collapses for one of the three measures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical comparison of 18 discrete entropy estimators applied to entropy, mutual information, and conditional mutual information, across six domain sizes and twelve sample sizes, with ground truth derived from randomly generated distributions in small, medium, and large entropy scenarios. The central claim is that the Chao-Shen and Chao-Wang-Jost estimators converge to the ground truth more quickly and more accurately across all three measures and all domain sizes, thereby enabling significant reductions in data collection effort for software engineering applications.","tokens_in":22724,"tokens_out":8281,"duration_ms":75531,"significance":"If the findings hold, this would be a practically valuable comparative study for software engineering and other fields, particularly because all estimators are implemented within the same Julia package (DiscreteEntropy.jl), and the authors provide a figshare link to the full code and evaluation outcomes. The use of a large number of randomly generated distributions, rather than a few hand-picked cases, is a strength. The entropy and mutual information results are internally coherent and the methodology for those measures is sound. However, the incorrect algebraic identity for conditional mutual information in Eq. (3) undermines all CMI-based conclusions and directly contradicts the abstract's 'regardless of the measure used' claim. This is a load-bearing issue that must be resolved before the paper can be accepted.","major_comments":[{"comment":"The stated identity for conditional mutual information is algebraically wrong. The paper gives I(X;Y|Z) = H(X,Z) + H(X,Y,Z) - H(Y,Z) - H(Z), but the standard identity is I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z). The printed expression differs from the correct one by 2H(X|Y,Z), a data-dependent quantity that is nonzero except when X is determined by (Y,Z). For example, with Z ~ Bernoulli(1/2) and X,Y conditionally independent fair Bernoulli variables given Z, the true CMI is 0, while the printed formula evaluates to 2 nats. Because all CMI results in Section 4 (Tables 3c, 5, 6a, 6b, and the motivating example in Section 2.1) inherit this formula, the 'regardless of the measure used' conclusion is unsupported as written. The authors must either demonstrate that the deposited code uses the correct identity (in which case Eq. (3) is a typo) or recompute all CMI results with the correct identity.","section":"Section 3.2, Eq. (3)"},{"comment":"The claim that the safe-sample-size-to-domain-size ratio 'decreases exponentially' for CS and CW is not supported by the data presented in Table 5. The text itself notes that for CMI the ratio for CS is approximately constant ('it now also extends to CS'), and the CW CMI ratios are non-monotonic across k. An exponential decay cannot be inferred from a handful of points, some of which increase. The strong statement in the Answer to RQ2 that 'the amount of data required by CS and CW ... decreases, on average, exponentially with increasing k' should be replaced by a more cautious description of the observed trends.","section":"Section 4.2, Answer to RQ2 and Table 5"},{"comment":"The statement that CS and CW perform best 'regardless of domain size' is contradicted by the paper's own accuracy results. Table 6a and the accompanying text state that for H and CMI with k <= 4096, Miller-Madow, Grassberger (1988), and Schurmann either closely approach or are more accurate than CS and CW. The abstract and conclusions should be qualified to reflect that the advantage of CS and CW is most pronounced for large domains and for mutual information, not 'regardless of domain size.'","section":"Abstract and Section 4.3, Table 6a"}],"minor_comments":[{"comment":"The sentence 'These values confirm that their estimations converge slightly faster than CS and CW and require slightly larger samples' is self-contradictory; presumably 'slightly slower than CS and CW' was intended.","section":"Section 4.1, entropy paragraph"},{"comment":"The table is very difficult to read because the column headings are not clearly separated across the H, MI, and CMI groups. Please reformat it so that each measure's sub-columns are unambiguous and the values can be checked without reference to the running text.","section":"Table 5"},{"comment":"The citation 'Grassberger [36]' points to Kraskov, Stögbauer, and Grassberger (2004), not to Grassberger's own entropy papers [26,27]. Please correct the reference to avoid confusion.","section":"Section 3.2"},{"comment":"The flattening-off bound (Fb = 0.1) is a key parameter of the analysis, but its choice is not justified beyond an example. A sensitivity analysis with, say, Fb = 0.05 and Fb = 0.2 would strengthen the conclusions about 'safe' sample sizes.","section":"Section 4.1"},{"comment":"The phrase 'this discovery enables a significant reduction in data collection effort without compromising performance' overstates the scope of the findings, which are based on simulated distributions and specific experimental settings. Please temper the wording to reflect the conditions under which the recommendations apply.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a serious load-bearing error in Eq. (3) that invalidates the CMI results as printed. However, the error is fixable within the scope of the paper: either show that the deposited code implements the correct identity (turning Eq. (3) into a typo) or recompute the CMI experiments with the correct formula. The entropy and MI analyses appear internally coherent and well-designed, and the study is otherwise a useful empirical contribution. With careful attention to the overclaims in the abstract and RQ2, the paper could be acceptable after major revision. I therefore disagree with the reader's blanket rejection and recommend a major revision, contingent on the authors resolving the CMI equation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take: this is a genuinely useful empirical benchmark for entropy and mutual information estimation, but the CMI half is compromised by an algebraically wrong identity in Eq. (3). The printed formula is I(X;Y|Z)=H(X,Z)+H(X,Y,Z)-H(Y,Z)-H(Z); the standard identity is H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z). The difference is 2H(X|Y,Z), which is nonzero except in degenerate cases. So every CMI number in the paper is suspect unless the deposited Julia code happens to use the correct identity. Since the abstract's \"regardless of measure\" claim explicitly includes CMI, this is load-bearing, not cosmetic.\n\nWhat the paper does well: it extends the usual entropy-only estimator comparisons to MI and CMI across 18 estimators, using randomized joint distributions, six domain sizes, and sample sizes from 2^3 to 2^14. Running all estimators from one package (DiscreteEntropy.jl) is a sensible uniformity decision, and the code and data are on figshare. The headline ranking — Chao-Shen and Chao-Wang-Jost converge faster and are more accurate — is plausible for entropy and MI and consistent with earlier coverage-based results in large-alphabet settings.\n\nSoft spots in proportion. The equation error is the big one; it is checkable against the code, so a typo is the likely explanation, but as printed the CMI results cannot be trusted. The ground-truth distribution generator is under-specified: the S/M/L scenarios are described verbally, not algorithmically. The flattening-off bound of 0.1 is arbitrary, and the reported Fp values depend on it. The accuracy analysis in Table 6b is restricted post hoc to samples above the CS/CW flattening point, which stacks the deck in their favor; the unrestricted averages in Table 6a already show other estimators beating CS/CW for small domains. No error bars, confidence intervals, or significance tests are attached to the 1000-run averages, and that matters when rankings are close.\n\nBottom line: the entropy and MI evidence is worth taking seriously, and the CMI flaw is probably fixable with one line if the code is right. This deserves a real referee. The reviewer should ask for a code check, a rerun of CMI if needed, a precise specification of the generator, and uncertainty quantification. I would not cite it as is, but I would want to see the revised version.","headline":"A useful entropy/MI benchmark whose CMI half is undermined by an algebraically wrong plug-in identity, so the 'regardless of measure' claim doesn't hold as printed.","tokens_in":23408,"tokens_out":3294,"would_cite":false,"duration_ms":32516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17","62B10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The Chao-Shen and Chao-Wang-Jost entropy estimators converge to ground truth faster than 16 competing estimators across entropy, mutual information, and conditional mutual information, and stay more accurate as sample sizes grow.","keywords":["entropy estimation","biased entropy estimators","Chao-Shen estimator","Chao-Wang-Jost estimator","mutual information","conditional mutual information","sample size","domain size"],"falsifier":"Re-run the same 1000-repetition experiment for conditional mutual information using the standard identity $I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)$ and compare the flattening-off points and MSE rankings; if Chao-Shen and Chao-Wang-Jost no longer lead under the 0.1 flattening-off bound, the paper's 'regardless of measure' conclusion is falsified.","tokens_in":22295,"feed_emoji":"📊","tokens_out":9221,"duration_ms":85661,"temperature":0.7,"pith_summary":"Entropy, mutual information, and conditional mutual information are routinely needed in software engineering, but exact values require knowing the underlying probability distribution, which most practical settings do not provide. The paper runs 18 entropy estimators over 1000 randomly generated joint distributions of six domain sizes and twelve sample sizes, and measures how quickly each estimate flattens out to ground truth and how accurate it is. Its central finding is that the Chao-Shen and Chao-Wang-Jost estimators stabilise with fewer samples than the other 16 and keep the lowest error as samples grow, across all three Shannon measures. If that finding holds, engineers can collect substantially less data for information-flow analysis, feature selection, and leakage measurement without losing estimation quality.","feed_headline":"Chao-Shen and Chao-Wang-Jost converge fastest on entropy, MI, and CMI","feed_subtitle":"Comparison of 18 estimators finds these two need fewer samples, cutting data collection costs in software analysis.","key_machinery":"The carrying object is the pair of sample-coverage entropy estimators: Chao-Shen, a nonparametric estimator that corrects the plug-in count for unseen categories via estimated sample coverage, and Chao-Wang-Jost, its successor built on discovery rates of new categories. They are inserted into the plug-in entropy-difference identities that express mutual information and conditional mutual information as sums and differences of joint entropies: $\\hat I_{\\mathrm{est}}(X;Y) = \\hat H_{\\mathrm{est}}(X) + \\hat H_{\\mathrm{est}}(Y) - \\hat H_{\\mathrm{est}}(X,Y)$, and the analogous identity in equation (3) for conditional mutual information. The experimental machinery that separates estimators is the flattening-off criterion: a 'safe' sample size, or flattening-off point $F_p$, is the smallest sample size at which the change in mean squared error between consecutive sample sizes stays below 0.1 for all larger samples, and the ratio $F_p/k$ is used to compare how sample-size requirements scale with domain size.","core_discovery":"On the paper's own terms, the discovery is that two sample-coverage-based entropy estimators, Chao-Shen and Chao-Wang-Jost, dominate the 18-estimator field when the underlying distribution is unknown. In the experiments, their estimates flatten out to the ground truth at smaller sample sizes than the competition for entropy, mutual information, and conditional mutual information, and the advantage grows as the domain size increases from 256 to 262,144. The 'safe' sample size divided by domain size falls as domains grow for these two estimators, whereas the nearest rivals, GSB88 and SHU, show a roughly constant ratio for mutual information. For larger samples the two also have the lowest mean squared error, with Chao-Wang-Jost slightly ahead in very large domains. The paper's stated conclusion is that, with no distribution knowledge, these estimators are the best default choice for all three Shannon measures.","pith_inferences":["The conditional-mutual-information results can be rescued if equation (3) is a typographical error rather than a coding error: if the implementation uses the standard identity $I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)$, the rankings likely survive, but the paper should make that explicit.","Because the safe sample size is defined by an MSE change below 0.1, the exact ranking near the boundary is threshold-dependent; a stricter or looser flattening-off bound could shift the flattening-off points, so practitioners should treat the quoted ratios as indicative rather than exact.","The experiments use fully random joint distributions, so a natural next test is to apply Chao-Shen and Chao-Wang-Jost to real software artefacts, such as the TriangleType information-leakage channel or fuzzing traces, and see whether the sample-size savings persist under structured dependencies.","The exponential decay of $F_p/k$ for the winning estimators suggests a practical heuristic: for domains above a few thousand values, a sample of a few thousand observations may saturate estimation quality, which could be turned into a deployment guideline for leakage measurement."],"forward_implications":["Software engineers who estimate information leakage, feature relevance, or test-suite diversity can use the Chao-Shen and Chao-Wang-Jost estimators as default choices when the data distribution is unknown.","The flattening-off point gives a concrete stopping rule: once samples exceed the safe size for the chosen estimator, collecting more data yields less than 0.1 improvement in mean squared error, so data collection effort can be redirected.","For mutual information, the ratio of safe sample size to domain size stays near 0.5 for GSB88 and SHU, while it decreases for Chao-Shen and Chao-Wang-Jost, meaning the winning estimators become relatively cheaper as alphabets grow.","Estimators such as Bonachela, PYM, and ANSB fail to stabilise within the studied sample-size range and should be avoided when data are limited.","For small domains up to about 4096 values, MM, GSB88, and SHU can match or beat the winners on accuracy, but they lose that edge as domain sizes grow."],"supporting_citations":[{"why":"Supplies the Chao-Shen sample-coverage estimator that the paper identifies as a top performer.","marker":"[13]"},{"why":"Supplies the Chao-Wang-Jost estimator, the other top performer, along with its species-accumulation derivation.","marker":"[14]"},{"why":"Defines the plug-in/maximum-likelihood baseline and bounds its bias, providing the comparison benchmark for all other estimators.","marker":"[47]"},{"why":"Provides the single Julia implementation of all 18 estimators, giving the uniform-precision comparison basis.","marker":"[34]"},{"why":"Sets the sampling protocol of power-of-two sample sizes and is the earlier estimator comparison that this study extends to MI and CMI.","marker":"[18]"},{"why":"Defines Shannon entropy and the mutual and conditional information measures being estimated.","marker":"[56]"}],"fun_headline_variants":["Chao-Shen and Chao-Wang-Jost converge fastest for entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost win on entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost are the most sample-efficient for entropy, MI, and CMI","With fewer samples, Chao-Shen and Chao-Wang-Jost nail entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost dominate entropy, MI, and CMI estimation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the code implements the correct entropy-combination identity for conditional mutual information; equation (3) in Section 3.2 as printed is not the standard identity, so the CMI results depend on whether the discrepancy is only in the text or in the implementation.","fun_headline_variants_meta":{"raw":{"variants":["Chao-Shen and Chao-Wang-Jost converge fastest for entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost win on entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost are the most sample-efficient for entropy, MI, and CMI","With fewer samples, Chao-Shen and Chao-Wang-Jost nail entropy, MI, and CMI","Chao-Shen and Chao-Wang-Jost dominate entropy, MI, and CMI estimation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002427,"raw_usage":{"total_tokens":9332,"prompt_tokens":949,"completion_tokens":8383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":8255}},"tokens_in":565,"tokens_out":8383,"duration_ms":50425,"temperature":1.0,"reasoning_tokens":8255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:18:50.874645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 1000-repetition experiment for conditional mutual information using the standard identity $I(X;Y|Z) = H(X,Z) + H(Y,Z) - H(X,Y,Z) - H(Z)$ and compare the flattening-off points and MSE rankings; if Chao-Shen and Chao-Wang-Jost no longer lead under the 0.1 flattening-off bound, the paper's 'regardless of measure' conclusion is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Chao-Shen sample-coverage estimator that the paper identifies as a top performer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Chao-Wang-Jost estimator, the other top performer, along with its species-accumulation derivation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the single Julia implementation of all 18 estimators, giving the uniform-precision comparison basis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets the sampling protocol of power-of-two sample sizes and is the earlier estimator comparison that this study extends to MI and CMI."}],"review_version":1}