How much is too much? Measuring divergence from Benford's Law with the Equivalent Contamination Proportion (ECP)

Manuel Cano-Rodriguez

REVIEW 3 major objections 5 minor 1 cited by

How much is too much? Measuring divergence from Benford's Law with the Equivalent Contamination Proportion (ECP)

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the Equivalent Contamination Proportion (ECP), defined as the fraction of contaminated observations needed to match an observed divergence statistic, provides a sample-size-robust, interpretable, and roughly…

desk verdict A genuinely new descriptive measure for Benford divergence with sound math, but missing null calibration undercuts its main interpretability claim. read the letter →

arxiv 2506.09915 v2 pith:OJIBWRFP submitted 2025-06-11 econ.EM

Manuel Cano-Rodriguez This is my paper · ORCID

classification econ.EM MSC 62F0362F0562G1062P20

keywords Benford'slawequivalentcontaminationproportiondivergencestatisticsmodelsforensicdataanalysissamplesizerobustnessgoodness-of-fiteffect

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Benford's Law says the leading digits of many real-world numbers follow a known distribution, and conformity with it is routinely used to flag manipulated or unreliable data. The usual test statistics—chi-squared, mean absolute deviation, Kolmogorov–Smirnov, and others—are sensitive to sample size and live on incomparable scales, so a large dataset can fail Benford for a trivial deviation while a small dataset can pass despite substantial irregularity. The paper proposes the Equivalent Contamination Proportion (ECP) as a remedy: the fraction f of a Benford-conforming sample of the same size that would have to be replaced by non-Benford values (uniform contamination in the baseline) for the expected divergence statistic to equal the value observed in the data. If the paper is right, researchers get a single 0-to-1 number with a substantive interpretation, one that does not drift with n and can be compared across studies that used different statistics. The paper develops closed-form and simulation-based ECP estimators for seven statistics and shows through reanalysis of three published studies that the measure can change how non-significance or significance is read.

What carries the argument

The machinery is the expected-value map f ↦ E[T(S)] built from the mean and variance of observed digit frequencies. Treating each digit's observed count as a Bernoulli average, the paper writes the probability of digit d as p_d = B_d + f·δ_d, where B_d is the Benford probability and δ_d is the difference between the contaminating distribution and Benford. This reduces E[χ²] and E[SSD] to explicit quadratics in f (with uniform contamination giving numerical coefficients), gives a folded-normal expectation for MAD, and supports a monotone simulation search for KS, Kuiper, and CvM. Inverting these maps at the observed statistic T(D) yields the ECP.

What would settle it

Simulate 10,000 datasets of moderate size in which a known fraction f of observations is drawn not from the uniform but from a concentrated distribution (say, all on leading digit 9), compute ECP by each of the seven estimation procedures, and check whether the seven estimates both track f and agree within a small band; the paper's own Table 3 shows exactly this failure mode for real data, so a controlled simulation would settle how often the congruence claim holds.

Watch

Extended reading notes

Core claim

The central discovery is a translation device. For any dataset D of size n and any divergence statistic T, the ECP is the value f in [0,1] such that the expected value of T on a synthetic same-size sample S equals the observed T(D), where a fraction f of S comes from a specified non-Benford distribution and the rest from Benford's law. Because the observed digit proportions in S are approximately normal, the expected value of quadratic statistics becomes a quadratic in f; under uniform contamination the paper obtains, for example, E[χ²_S] ≈ (n−1)·0.4017·f² + 3.6153·f + 8, with analogous expressions for SSD and a folded-normal expression for MAD. For non-additive statistics (ED, KS, Kuiper, CvM) it uses a first-order approximation for ED and an iterative simulation procedure. Simulations with 100,000 replications show ECP estimates tracking true contamination across sample sizes from 100 to 100,000, and correlations above roughly 0.84 across seven ECP estimators on 26 empirical datasets; the paper also reports that congruence degrades when digits deviate in uneven patterns. The claim is not that ECP is a new significance test—it is an interpretable parameter that complements tests, and in three reanalyses it flags cases where non-rejection masks meaningful deviation and where rejection reflects only excess power.

Load-bearing premise

The method's interpretability and cross-statistic agreement depend on the assumption that a dataset's real non-Benford irregularity can be represented as a uniform contamination of an otherwise Benford-conforming sample; when the true irregularity is concentrated on particular digits (for example, an excess of 9s), different statistics yield materially different ECP values, as the paper itself observes for several empirical datasets.

Editorial extensions

If this is right

Researchers can report a single ECP alongside any Benford test statistic, so a chi-squared of 18 with n = 5,000 and a MAD of 0.003 with n = 100,000 can be compared as equivalent contamination levels.
Sample-size-driven rejections lose their sting: a large dataset that rejects Benford at ECP = 1.42% is seen as near-conforming, while a small dataset that accepts at ECP ≈ 35% is seen as substantially divergent.
The closed-form quadratics allow power planning: given a contamination level one cares about, the paper's Table 2 gives the sample size needed for the expected statistic to reach a 95% or 99% threshold, or vice versa.
Retrospective application becomes routine whenever a paper reports a statistic and sample size, so published Benford findings can be translated to a common scale without re-running analyses.
Comparing different contamination models (non-uniform alternatives) extends ECP to second-digit, last-digit, or first-two-digit tests with the same machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If ECP is adopted as a standard complement, it effectively becomes an effect-size measure for Benford tests, allowing meta-analytic pooling of divergence magnitudes across accounting, finance, and economics studies that used different statistics; the paper does not draw this meta-analytic implication explicitly.
Because the baseline contamination distribution is uniform, ECP values for real datasets whose irregularities concentrate on particular digits will be sensitive to the chosen statistic; a sensitivity analysis varying the contaminating distribution would show how much of the congruence claim survives outside the uniform benchmark.
The same inversion logic could be applied to any null distribution, not just Benford's law, turning divergence statistics for other data-quality checks into a comparable contamination share—an extension the paper frames only for digit tests.
The thresholds in Table 2 suggest a practical design rule: studies with n below a few hundred cannot expect to detect contamination under roughly 10%, so ECP should accompany small-sample Benford analyses to avoid the acceptance fallacy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The ECP idea is genuinely new: instead of reporting bare chi-squared or MAD, you invert a contamination-mixture model and report the fraction f that would reproduce the observed statistic in expectation. The closed-form derivations for chi-squared and SSD are correct, and the folded-normal approximation for MAD is reasonable; the simulation-based route for KS/Kuiper/CvM is transparent and ships with code and data. As a purely descriptive complement to hypothesis tests, it has real potential.

The soft spot is that the paper never calibrates ECP against the null. Because ECP is a one-to-one transform of the observed statistic, it inherits all the sampling noise. Under pure Benford data with n=100, the 95th percentile of chi-squared is about 15.5, which inverts to ECP ≈39%—exactly what Table 2 Panel A reports. So a clean dataset will show ECP values above 30% roughly 10% of the time. Footnote 3 treats chi2=14 (ECP 34.6%) as "substantial contamination," and Section 4.1 calls the Amiram average of 34.73% at n=125 "substantial deviations"; both are within normal null variation. Without null quantiles or confidence intervals, the interpretability claim doesn't hold. This is a load-bearing issue, not a cosmetic one.

The second issue is the uniform contamination assumption. The ECP's cross-statistic stability only holds when the contamination pattern matches the model. Table 3 shows that for Constants, Atomic Weights, and Addresses, ECPs across statistics range by 30–40 percentage points. The paper honestly notes the variation, but the abstract's "consistent results under mild conditions" is stronger than what Table 3 shows.

The math is solid; the packaging is too assertive. The applications in Section 4 should be read as illustrative of the method's mechanics, not as evidence about the underlying datasets.

This belongs in a serious review process—the concept is worth having, and the derivations and code are reusable. But the author needs to add null calibration, report uncertainty, and soften the absolute-magnitude language. For an audience of auditors and empirical economists, the ECP is a good idea in need of a more careful presentation.

Referee Report

3 major / 5 minor

Summary. This paper introduces the Equivalent Contamination Proportion (ECP), defined as the proportion f of observations in a hypothetical Benford-conforming sample of the same size that would need to be replaced by draws from a specified non-Benford distribution for the expected value of a divergence statistic to equal the value observed in the empirical dataset. The author derives closed-form expressions for chi-squared and SSD, a folded-normal approximation for MAD, a Taylor approximation for ED, and a simulation-based method for KS, Kuiper, and CvM. He evaluates the ECP on simulated data with known contamination, reports ECP values for 26 empirical datasets, and reinterprets three published Benford studies (Amiram et al. 2015; Sifat et al. 2024; Eutsler et al. 2023). The paper claims that the ECP is sample-size robust, interpretable on a [0,1] scale, and broadly congruent across divergence statistics.

Significance. If properly supported, the ECP would be a useful effect-size measure for Benford analysis, addressing real problems of sample-size dependence and non-comparability across statistics. The algebraic derivations for chi-squared and SSD appear correct, the simulation procedures are clearly described, and the Python code plus retrospective applications are constructive. However, the paper's central interpretability claim currently lacks null calibration: because the ECP is a monotone transform of the observed statistic, it inherits the sampling variability of that statistic, and the manuscript itself shows that pure Benford data of moderate size can produce ECP values above 30% well within null variability. The cross-statistic congruence claim is also only partially supported, with several real datasets showing spreads of 20–50 percentage points across statistics. The strengths of the paper are its transparency, derivations, and reproducible code; the main weakness is the gap between the strength of the abstract's claims and the statistical evidence provided.

major comments (3)

[§4.1, footnote 3, Table 2 Panel A] The manuscript treats ECP values around 34–35% at n≈100–125 as indicating 'substantial contamination', but Table 2 Panel A shows that for chi-squared at n=100 the ECP corresponding to the 95th percentile of the null distribution is 39.14%. The average ECP of 34.73% reported for Amiram et al. (2015) at n=125 is therefore close to or below the null percentile, and footnote 3's ECP of 34.6% for chi²=14, n=100 is below the null 95th percentile as well. Since the ECP is a monotone transform of T(D), it inherits the sampling noise in T(D); point estimates must be accompanied by a null sampling distribution or confidence intervals before they can be interpreted as absolute contamination magnitudes. Supplying this calibration and reinterpreting the applications accordingly is necessary to support the abstract's claim that the ECP is a continuous, interpretable measure of divergence.
[§3.1, Table 1] The simulation exercise is a self-consistency check rather than an independent test of sample-size robustness. Because the ECP is defined as the inverse of E[T(S)] under the same uniform contamination model used to generate Table 1, recovering a pre-specified f from the average statistic is implied by construction. The table provides no information on the sampling distribution of the ECP under the null or under mis-specified contamination, and therefore does not establish that ECP magnitudes are comparable across n. Table 2 Panel A in fact shows that the ECP value at a fixed null quantile varies strongly with n: for chi-squared, the 95th-percentile ECP is 39.14% at n=100 but 13.23% at n=1000. The 'robust to sample size' claim should be qualified as point-estimate recovery under a correct model, not as distributional invariance.
[§3.2, Table 3] The congruence claim is not supported for a substantial subset of real datasets. The four datasets explicitly identified in the text have ECP spreads that are difficult to call 'broadly congruent': Constants (17.47% to 63.33%), Specific Heat (18.57% to 44.09%), Atomic Weights (42.89% to 73.75%), and Addresses (0% to 23.52%). The 'mild conditions' under which cross-statistic congruence is claimed are never formalized, and the Addresses dataset shows that a localized digit anomaly can produce zero ECP from SSD/MAD/ED while yielding non-negligible ECPs from KS, Kuiper, and chi-squared. The paper needs either a formal condition under which congruence holds or a definition of acceptable congruence, rather than relying on pairwise correlations that can be driven by the large number of zero-ECP datasets.

minor comments (5)

[Table 1 footnote] The word 'correspondint' should be 'corresponding' in the table note.
[Figure 1 caption] The caption lists '50,00' among the sample sizes; this should be '5,000' or '50,000', depending on the intended value.
[§4.1, Table 5] The row label 'Botomm 1% of line items' contains a typo and should read 'Bottom'.
[§4.2, Table 6] The column heading 'V olume' contains a stray space and should read 'Volume'.
[§1, p. 5] 'This easy of computation' should be 'This ease of computation'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ECP is a transparent definitional inversion, and the claimed properties are empirically evaluated rather than derived from the definition.

full rationale

The ECP is explicitly introduced as an inversion: Eq. (1) defines it as the f satisfying E[T(S)] = T(D), and the appendix derives E[T(S)] under an explicit Bernoulli/uniform contamination model. Those derivations do not import the target claim: each statistic's expectation is computed from stated distributional assumptions, and the congruence across statistics is checked on 26 external empirical datasets, where the paper openly reports nontrivial spread (Constants, Spec. Heat, Atomic Wgt., Addresses). The Section 3.1 simulation, in which ECP estimates recover the contamination proportion f, is a self-consistency/calibration check because the simulated data are generated from the very uniform-contamination model used to define the ECP; but the paper does not use this as independent evidence that real data are uniformly contaminated, and it explicitly warns that the ECP depends on the chosen contamination model and is not a test statistic. The sample-size robustness claim is an analytic consequence of inverting the n-specific expectation formulas, not a hidden reuse of the conclusion. Minor self-citations (e.g., Cano-Rodríguez et al. 2025) appear only in support of conventions or threshold references and are not load-bearing for the central definition or inversion. The reviewer's null-calibration critique of interpreting ECP magnitudes is a statistical validity concern, not a circularity concern. No specific equation reduces to its own input by construction.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The method introduces one hand-chosen convergence parameter and relies on several modeling assumptions, most importantly the uniform contamination distribution and the appropriateness of BL as a benchmark. These are not derived from first principles and limit the generality of the ECP.

free parameters (1)

Significance level for convergence of simulation-based ECP = not stated in paper (likely 5%)
The iterative simulation method in Appendix A.5 stops when the difference between simulated mean statistic and observed statistic is not statistically significant at a predefined significance level; this choice affects the estimated ECP and is not reported or varied in sensitivity analysis.

assumptions (6)

domain assumption Uniform distribution as the contamination distribution
The ECP is defined and estimated under the assumption that contaminated observations follow a uniform distribution over the nine leading digits. The paper justifies this as a neutral benchmark, but if actual data irregularities have a different distribution, the ECP's numeric value is not the true contamination proportion. This assumption is invoked throughout Section 2 and the Appendix.
domain assumption Benford's Law is the appropriate null distribution for unmanipulated data
The whole framework treats BL as the benchmark and interprets contamination as a mixture that replaces BL-conforming values with non-BL values. If the underlying data generation does not follow BL, as the author notes for COVID-19 death counts in Section 4.3, then the ECP is not meaningful as a measure of manipulation.
domain assumption Independence of the n digit observations in the hypothetical sample
The derivation of E[O_d] and Var[O_d] in Appendix equations (3)-(5) models the leading digits as n independent Bernoulli draws. Real-world datasets can have within-dataset dependence; the ECP would then not exactly match the reported expectation.
domain assumption Monotonicity and continuity of E[T(S)] as a function of f
The inversion that defines the ECP requires E[T(S)] to increase continuously and monotonically in f between f=0 and f=1. This is assumed for all seven statistics, explicitly for the simulation method in Appendix A.5, and is not proven for KS, Kuiper, or CvM.
standard math Normal approximation for the distribution of digit proportions
The MAD derivation uses the folded normal distribution, which requires the Central Limit Theorem approximation for the digit proportions O_d in Appendix A.3. This is standard for moderate n but is an approximation.
standard math First-order Taylor approximation for the expected Euclidean distance
Appendix A.4 approximates E[ED] as sqrt(E[SSD]) using a first-order expansion; the paper notes this is imprecise for small n and f.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How much is too much? Measuring divergence from Benford's Law with the Equivalent Contamination Proportion (ECP)." pith.science (2026). https://pith.science/paper/OJIBWRFP

@misc{pith2026250609915,
  author       = {Pith},
  title        = {Pith review of: How much is too much? Measuring divergence from Benford's Law with the Equivalent Contamination Proportion (ECP)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OJIBWRFP}},
  note         = {Machine review of arXiv:2506.09915}
}

read the original abstract

Conformity with Benford's Law is widely used to detect irregularities in numerical datasets, particularly in accounting, finance, and economics. However, the statistical tools commonly used for this purpose (such as Chi-squared, MAD, or KS) suffer from three key limitations: sensitivity to sample size, lack of interpretability of their scale, and the absence of a common metric that allows for comparison across different statistics. This paper introduces the Equivalent Contamination Proportion (ECP) to address these issues. Defined as the proportion of contamination in a hypothetical Benford-conforming sample such that the expected value of the divergence statistic matches the one observed in the actual data, the ECP provides a continuous and interpretable measure of deviation (ranging from 0 to 1), is robust to sample size, and offers consistent results across different divergence statistics under mild conditions. Closed-form and simulation-based methods are developed for estimating the ECP, and, through a retrospective analysis of three influential studies, it is shown how the ECP can complement the information provided by traditional divergence statistics and enhance the interpretation of results.

Figures

Figures reproduced from arXiv: 2506.09915 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Manipulation testing based on Benford's Law for discrete scores
stat.ME 2026-07 reject novelty 5.0 of 10

A Benford's-Law-based framework for RDD manipulation testing that selects a bandwidth and tests score symmetry around the cutoff, but it tests global symmetry rather than local manipulation.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

Statistical models and the Benford hypothesis: a unified framework,

Barabesi, L., Cerioli, A., and Di Marzio, M. (2023), “Statistical models and the Benford hypothesis: a unified framework,” Test, Springer Science and Business Media Deutschland GmbH, 32, 1479–1507. https://doi.org/10.1007/S11749-023-00881-Y. Berger, A., and Hill, T. P. (2011), “A basic theory of Benford’s law,” Probability Surveys, 8, 1–126. https://doi.o...

arXiv 2023
[37]

The significant digit law in statistical physics,

https://doi.org/10.1029/2010GL044830. Shao, L., and Ma, B. Q. (2010), “The significant digit law in statistical physics,” Physica A: Statistical Mechanics and its Applications, North-Holland, 389, 3109–3116. https://doi.org/10.1016/J.PHYSA.2010.04.021

work page doi:10.1029/2010gl044830 2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Statistical models and the Benford hypothesis: a unified framework,

Barabesi, L., Cerioli, A., and Di Marzio, M. (2023), “Statistical models and the Benford hypothesis: a unified framework,” Test, Springer Science and Business Media Deutschland GmbH, 32, 1479–1507. https://doi.org/10.1007/S11749-023-00881-Y. Berger, A., and Hill, T. P. (2011), “A basic theory of Benford’s law,” Probability Surveys, 8, 1–126. https://doi.o...

arXiv 2023

[2] [37]

The significant digit law in statistical physics,

https://doi.org/10.1029/2010GL044830. Shao, L., and Ma, B. Q. (2010), “The significant digit law in statistical physics,” Physica A: Statistical Mechanics and its Applications, North-Holland, 389, 3109–3116. https://doi.org/10.1016/J.PHYSA.2010.04.021

work page doi:10.1029/2010gl044830 2010