REVIEW 3 major objections 4 minor 16 references
JSD gets an operational reading: 1/d vs 1/d² samples
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · glm-5.2
2026-07-08 11:20 UTC pith:LCFH7V3W
load-bearing objection The 1/d² majority-vote scaling is loose — the true scaling is 1/d — but the paper's proofs are correct and the 1/d LLR bound is a clean, self-contained derivation. the 3 major comments →
Sample complexity bounds for the Jensen-Shannon divergence
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central mechanism is a chain of inequalities linking the Chernoff information (which governs the Bayes error rate of the LLR classifier) to the JSD: C(P,Q) ≥ B(P,Q) ≥ H²(P,Q) ≥ log(2)·D_JS(P,Q), where B is the Bhattacharyya distance and H² is the squared Hellinger distance. Substituting this lower bound into the standard exponential error bound p_e ≤ e^{-N·C(P,Q)} yields the 1/d scaling for the LLR classifier. For the majority-vote classifier, the proof combines Hoeffding's inequality on the binomial majority count with a known single-sample Bayes error bound p_e ≤ ½(1−d) from Lin (1991), then applies the inequality −log(y) ≥ 1−y to extract the 1/d² scaling. The paper shows this 1/d² is
What carries the argument
The inequality chain C(P,Q) ≥ B(P,Q) ≥ H²(P,Q) ≥ log(2)·D_JS(P,Q), combined with the Chernoff bound p_e ≤ e^{-N·C(P,Q)} for the LLR classifier, and Hoeffding's inequality combined with Lin's bound p_e ≤ ½(1−d) for the majority-vote classifier.
Load-bearing premise
The majority-vote bound (1/d²) depends on Lin's 1991 bound p_e ≤ ½(1−d), which is an equality only for specific distribution pairs and can be loose for others. Since the 1/d² scaling is derived by feeding this bound into Hoeffding's inequality, the tightness of the 1/d² claim depends on whether Lin's bound is representative of typical distribution pairs.
What would settle it
Construct a family of distribution pairs where Lin's bound p_e ≤ ½(1−d) is far from tight (i.e., where the actual single-sample Bayes error is much smaller than ½(1−d)). For such pairs, the majority-vote classifier should need far fewer than O(1/d²) samples, contradicting the claim that 1/d² is the characteristic scaling.
If this is right
- The 1/d vs 1/d² contrast quantifies exactly how much information is lost when per-sample evidence is hard-quantized to a single bit before aggregation, giving a concrete cost metric for distributed sensing architectures.
- Any system that must compress, communicate, or commit to per-sample decisions independently (federated estimation, biological neural coding, sensor networks) can use the 1/d² bound to assess the penalty relative to the centralized 1/d optimum.
- Intermediate strategies—soft quantization, confidence-weighted voting, transmitting a few bits per sample—should achieve sample complexities between 1/d and 1/d², interpolating between the two extremes the paper brackets.
- The bound provides a practical rule of thumb: if JSD between two distributions is 0.1 bits, the optimal classifier needs roughly 10/0.1 ≈ 100 samples per log(1/ε) factor, while a majority-vote scheme needs roughly 2/0.01 = 200 samples per log(1/ε) factor.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript derives two sample-complexity bounds for binary hypothesis testing in terms of the Jensen-Shannon divergence $d = D_{JS}(P,Q)$: (1) for the log-likelihood-ratio (LLR) classifier, $N ge log(1/epsilon)/(d log 2)$ samples suffice (Theorem 1); (2) for a majority-vote classifier built from Bayes-optimal single-sample decisions, $N ge 2log(1/epsilon)/d^2$ samples suffice (Theorem 2). Theorem 1 chains Chernoff $ge$ Bhattacharyya $ge$ Hellinger $ge$ JSD via three self-contained lemmas. Theorem 2 combines Lin's (1991) single-sample Bayes-error bound $p_e le frac12(1-d)$ with Hoeffding's inequality. The proofs are mathematically correct and the sufficient conditions are valid.
Significance. The paper provides a clean, self-contained derivation of the LLR sample-complexity bound with explicit constants, and the majority-vote bound is a new sufficient condition. The parameter-free nature of both derivations is a strength. However, the paper's central narrative—a qualitative contrast between $1/d$ and $1/d^2$ scaling—requires qualification, as detailed below. The contribution is real but its framing overstates what the bounds imply.
major comments (3)
- The framing of 'distinct scalings' (abstract, Discussion) is misleading because the $1/d^2$ majority-vote bound is a worst-case result over distribution pairs with a given JSD, and is loose by a factor of $Theta(1/d)$ for many common distribution families. Concretely: for $P=(frac12+delta, frac12-delta)$, $Q=(frac12-delta, frac12+delta)$, one has $d approx 2delta^2/ln 2$ and $p_e = frac12 - delta = frac12 - Theta(sqrt{d})$, so the majority-vote error exponent is $-log(2sqrt{p_e(1-p_e)}) approx Theta(d)$, giving a true sample complexity of $Theta(1/d)$—the same scaling as the LLR classifier. The $1/d^2$ bound arises because Lin's bound $p_e le frac12(1-d)$ is tight for some distributions (e.g., $P=(1-epsilon, epsilon, 0,ldots)$, $Q=(1-epsilon, 0, epsilon, 0,ldots)$, where $d=epsilon$ and $p_e = frac12(1-d)$ exactly) but loose for others. The paper should state explicitly that Theorem 2 is
- Eq. (35) and the surrounding text claim the bound (34) is 'tight to leading order as $d to 0$.' This is tight only relative to the intermediate expression (32), which itself inherits the looseness of Lin's bound (29). For distributions where $p_e = frac12 - Theta(sqrt{d})$, the denominator of (32) is $Theta(d)$, not $Theta(d^2)$, so the overall bound (34) is loose by $Theta(1/d)$ in the exponent. The tightness claim should be qualified: the bound is tight for worst-case distributions (where $p_e = frac12(1-d)$ holds with equality) but not for all distribution pairs.
- Discussion, final paragraph: the claim that 'the $1/d^2$ penalty is the price of locality' is overstated. For distribution pairs where the per-sample evidence is spread uniformly (e.g., the binary symmetric example above), the majority-vote classifier achieves $Theta(1/d)$ scaling—no penalty relative to the LLR classifier. The $1/d^2$ penalty arises specifically when the single-sample Bayes error is close to $frac12 - Theta(d)$ rather than $frac12 - Theta(sqrt{d})$, which occurs when the distributions differ on rare events (sparse perturbations). The paper should discuss this distribution-dependence rather than presenting the penalty as universal.
minor comments (4)
- Introduction: the paper states that sample complexity 'has been shown to be given by the Hellinger distance [11, 12], up to multiplicative constants.' It would help to state the resulting $Theta(1/H^2)$ scaling explicitly and note that Lemma 3 implies $H^2 ge log(2) cdot d$, connecting this known result to Theorem 1.
- Eq. (9): the Chernoff information is defined with natural logarithm, while JSD in Eq. (1) uses $log_2$. The bound (10) mixes these conventions ($C ge log(2) cdot D_{JS}$). This is correct but could confuse readers; a brief note on the convention would help.
- The paper does not discuss whether the $1/d^2$ bound could be improved by using a tighter bound on $p_e$ that incorporates additional distributional information (e.g., TV distance or alphabet size). A sentence acknowledging that Lin's bound is the tightest possible bound on $p_e$ in terms of JSD alone would strengthen the presentation.
- Reference [12] (Pensia, Jog & Loh, 2025) is cited as 'arXiv (2025)'; please update with publication details if available.
Simulated Author's Rebuttal
We thank the referee for a careful and substantive reading. The referee's core observation—that the 1/d² majority-vote bound is a worst-case result over distribution pairs with a given JSD, and is loose by Θ(1/d) for common families—is correct. We will revise the manuscript to qualify the framing accordingly. The mathematical results themselves are unaffected; what needs to change is the narrative around them.
read point-by-point responses
-
Referee: The framing of 'distinct scalings' is misleading because the 1/d² majority-vote bound is a worst-case result, loose by Θ(1/d) for many common distribution families (e.g., binary symmetric). The paper should state explicitly that Theorem 2 is a worst-case bound.
Authors: We agree. The referee's example is correct: for P=(1/2+δ, 1/2−δ), Q=(1/2−δ, 1/2+δ), one has d ≈ 2δ²/ln 2 and p_e = 1/2 − δ = 1/2 − Θ(√d), so the majority-vote error exponent is −log(2√(p_e(1−p_e))) = Θ(d), giving true sample complexity Θ(1/d)—the same scaling as the LLR classifier. The 1/d² bound in Theorem 2 arises because Lin's bound p_e ≤ ½(1−d) is tight for some distributions (e.g., sparse perturbations where the distributions differ on rare events) but loose for others. We will revise the abstract, the Results section header, and the Discussion to state explicitly that Theorem 2 provides a worst-case sufficient condition over all distribution pairs with a given JSD, and that for many common families the majority-vote classifier achieves Θ(1/d) scaling. The phrase 'distinct scalings' will be replaced with language that accurately reflects this qualification. revision: yes
-
Referee: Eq. (35) and the surrounding text claim the bound (34) is 'tight to leading order as d→0.' This is tight only relative to the intermediate expression (32), which itself inherits the looseness of Lin's bound. The tightness claim should be qualified.
Authors: The referee is correct. The expansion in (35) shows that (34) is tight relative to (32), but (32) itself inherits the looseness of Lin's bound (29). For distributions where p_e = ½ − Θ(√d), the denominator of (32) is Θ(d), not Θ(d²), so the overall bound (34) is loose by Θ(1/d) in the exponent. We will revise the text around (35) to state that the bound is tight to leading order relative to the intermediate expression (32), and that this tightness holds for worst-case distributions where Lin's bound is achieved with equality, but not for all distribution pairs. We will add the referee's binary symmetric example as an explicit illustration. revision: yes
-
Referee: The claim that 'the 1/d² penalty is the price of locality' is overstated. For distribution pairs where per-sample evidence is spread uniformly, majority-vote achieves Θ(1/d) scaling—no penalty. The 1/d² penalty arises specifically for sparse perturbations. The paper should discuss this distribution-dependence.
Authors: We agree that the statement as written is too strong. The 1/d² penalty is not universal; it arises specifically when the single-sample Bayes error is ½ − Θ(d) (i.e., when Lin's bound is tight), which occurs for distributions that differ on rare events. For distributions where the per-sample evidence is spread uniformly—such as the binary symmetric example—the majority-vote classifier incurs no asymptotic penalty relative to the LLR classifier. We will revise the final paragraph of the Discussion to replace the universal claim with a distribution-dependent discussion: the 1/d² bound is the worst-case price of locality, realized for sparse-perturbation distributions, while for uniformly spread differences the penalty vanishes. We will also note that the degree of penalty depends on the relationship between p_e and d, which varies across distribution families. revision: yes
Circularity Check
No circularity found. The derivation is self-contained, parameter-free, and relies on external results (Chernoff, Hoeffding, Lin 1991) with no self-citation chain.
full rationale
The paper's two main results are derived from standard external results and self-contained proofs, with no fitted parameters, no self-citations, and no definitional circularity. Theorem 1 chains Chernoff's bound (eq 8, from [14]) through three lemmas (eqs 14, 15, 20) that are each proved from scratch via elementary calculus and the inequality log(y) ≤ y−1. The final bound N ≥ log(1/ε)/(d·log 2) (eq 23) follows by direct substitution. No step reduces to its own input by construction. Theorem 2 combines Hoeffding's inequality (eq 28, from [16]) with Lin's 1991 bound p_e ≤ ½(1−d) (eq 29, from [3], an external citation by different authors) and then applies the same log(y) ≤ y−1 inequality to arrive at N ≥ 2log(1/ε)/d² (eq 34). All references are to external works by different authors. The skeptic's concern that Lin's bound is loose and the true majority-vote scaling may be 1/d rather than 1/d² is a correctness/tightness issue, not a circularity issue — the paper does not define any quantity in terms of its own output, does not fit parameters to data, and does not invoke self-cited results as load-bearing premises. The derivation chain is genuinely self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (4)
- standard math Chernoff bound on Bayes error: p_e^(N) ≤ e^{−N·C(P,Q)}
- standard math Hoeffding's inequality for bounded random variables
- domain assumption Single-sample Bayes error bound: p_e ≤ ½(1−D_JS(P,Q))
- standard math log(y) ≤ y−1 for y > 0
read the original abstract
The Jensen-Shannon divergence (JSD) is a symmetric and bounded measure of the dissimilarity of two probability distributions, which has become a standard tool in statistics, information theory, and machine learning. We complement the understanding of its mathematical properties by presenting an analysis of the amount of data that is needed to distinguish between two distributions, given the value of JSD between them. We find the number of independent and identically distributed samples that suffice for a classifier to determine which of two distributions generated observed data at a desired error rate, for two complementary classifiers: we show that for the log-likelihood-ratio classifier, a sample size that grows as the inverse JSD is sufficient, whereas for a majority-vote classifier assembled from independent single-sample decisions, the sufficient size grows as the squared inverse JSD. These distinct scalings offer operational readings of JSD values and their translation into distinguishability in different contexts.
Reference graph
Works this paper leans on
-
[1]
Kullback, S. & Leibler, R. A. On Information and Sufficiency.The Annals of Mathematical Statistics22,79–86 (1951)
work page 1951
-
[2]
Cover, T. M. & Thomas, J. A.Elements of Information Theory2nd ed. (Wiley, 2005)
work page 2005
-
[3]
Lin, J. Divergence Measures Based on the Shannon Entropy.IEEE Trans- actions on Information Theory37,145–151 (1991)
work page 1991
-
[4]
Grosse, I.et al.Analysis of symbolic sequences using the Jensen-Shannon divergence.Phys. Rev. E65,041905 (2002)
work page 2002
-
[5]
Sims, G. E., Jun, S.-R., Wu, G. A. & Kim, S.-H. Alignment-free genome comparison with feature frequency profiles (FFP) and optimal resolutions. Proceedings of the National Academy of Sciences106,2677–2682 (2009)
work page 2009
-
[6]
Goodfellow, I. J.et al.Generative Adversarial Nets.Advances in Neural Information Processing Systems27,2672–2680 (2014). 9
work page 2014
-
[7]
Csisz´ ar, I. Information-type measures of difference of probability distri- butions and indirect observations.Studia Scientiarum Mathematicarum Hungarica2,299–318 (1967)
work page 1967
-
[8]
Csisz´ ar, I. & Shields, P. Information Theory and Statistics: A Tutorial. Foundations and Trends in Communications and Information Theory1 (Nov. 2004)
work page 2004
-
[9]
Amari, S.-I.α-Divergence Is Unique, Belonging to Bothf-Divergence and Bregman Divergence Classes.IEEE Transactions on Information Theory 55,4925–4931 (2009)
work page 2009
-
[10]
Endres, D. & Schindelin, J. A new metric for probability distributions. IEEE Transactions on Information Theory49,1858–1860 (2003)
work page 2003
-
[11]
The complexity of massive data set computations.PhD thesis, University of California, Berkeley(2002)
Bar-Yossef, Z. The complexity of massive data set computations.PhD thesis, University of California, Berkeley(2002)
work page 2002
-
[12]
Pensia, A., Jog, V. & Loh, P.-L. The Sample Complexity of Simple Binary Hypothesis Testing.arXiv(2025)
work page 2025
-
[13]
Neyman, J. & Pearson, E. S. On the Problem of the Most Efficient Tests of Statistical Hypotheses.Philosophical Transactions of the Royal Society of London. Series A231,289–337 (1933)
work page 1933
-
[14]
Chernoff, H. A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the Sum of Observations.The Annals of Mathematical Statistics 23,493–507 (1952)
work page 1952
-
[15]
Chair, Z. & Varshney, P. K. Optimal Data Fusion in Multiple Sensor De- tection Systems.IEEE Transactions on Aerospace and Electronic Systems 22,98–101 (1986)
work page 1986
-
[16]
Hoeffding, W. Probability Inequalities for Sums of Bounded Random Vari- ables.Journal of the American Statistical Association58,13–30 (1963). 10
work page 1963
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.