Pith. sign in

REVIEW 3 major objections 4 minor 16 references

JSD gets an operational reading: 1/d vs 1/d² samples

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · glm-5.2

2026-07-08 11:20 UTC pith:LCFH7V3W

load-bearing objection The 1/d² majority-vote scaling is loose — the true scaling is 1/d — but the paper's proofs are correct and the 1/d LLR bound is a clean, self-contained derivation. the 3 major comments →

arxiv 2607.06270 v1 pith:LCFH7V3W submitted 2026-07-07 cs.IT math.IT

Sample complexity bounds for the Jensen-Shannon divergence

classification cs.IT math.IT
keywords Jensen-Shannon divergencesample complexitybinary hypothesis testinglog-likelihood-ratio classifiermajority voteChernoff informationBhattacharyya distanceHellinger distance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks a direct question: given that the Jensen-Shannon divergence (JSD) between two distributions P and Q equals d, how many independent samples does a classifier need to tell them apart at error rate ε? It proves two answers that bracket a spectrum. For the log-likelihood-ratio (LLR) classifier—the Bayes-optimal rule that weighs the full evidence of each observation—N ≥ log(1/ε)/(d·log 2) samples suffice, so the requirement scales as 1/d. For a majority-vote classifier that hard-quantizes each sample to a single bit and then aggregates, N ≥ 2·log(1/ε)/d² suffices, so the requirement scales as 1/d². The extra factor of 1/d is the price of discarding the magnitude of per-sample evidence before combining decisions.

Core claim

The central mechanism is a chain of inequalities linking the Chernoff information (which governs the Bayes error rate of the LLR classifier) to the JSD: C(P,Q) ≥ B(P,Q) ≥ H²(P,Q) ≥ log(2)·D_JS(P,Q), where B is the Bhattacharyya distance and H² is the squared Hellinger distance. Substituting this lower bound into the standard exponential error bound p_e ≤ e^{-N·C(P,Q)} yields the 1/d scaling for the LLR classifier. For the majority-vote classifier, the proof combines Hoeffding's inequality on the binomial majority count with a known single-sample Bayes error bound p_e ≤ ½(1−d) from Lin (1991), then applies the inequality −log(y) ≥ 1−y to extract the 1/d² scaling. The paper shows this 1/d² is

What carries the argument

The inequality chain C(P,Q) ≥ B(P,Q) ≥ H²(P,Q) ≥ log(2)·D_JS(P,Q), combined with the Chernoff bound p_e ≤ e^{-N·C(P,Q)} for the LLR classifier, and Hoeffding's inequality combined with Lin's bound p_e ≤ ½(1−d) for the majority-vote classifier.

Load-bearing premise

The majority-vote bound (1/d²) depends on Lin's 1991 bound p_e ≤ ½(1−d), which is an equality only for specific distribution pairs and can be loose for others. Since the 1/d² scaling is derived by feeding this bound into Hoeffding's inequality, the tightness of the 1/d² claim depends on whether Lin's bound is representative of typical distribution pairs.

What would settle it

Construct a family of distribution pairs where Lin's bound p_e ≤ ½(1−d) is far from tight (i.e., where the actual single-sample Bayes error is much smaller than ½(1−d)). For such pairs, the majority-vote classifier should need far fewer than O(1/d²) samples, contradicting the claim that 1/d² is the characteristic scaling.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The 1/d vs 1/d² contrast quantifies exactly how much information is lost when per-sample evidence is hard-quantized to a single bit before aggregation, giving a concrete cost metric for distributed sensing architectures.
  • Any system that must compress, communicate, or commit to per-sample decisions independently (federated estimation, biological neural coding, sensor networks) can use the 1/d² bound to assess the penalty relative to the centralized 1/d optimum.
  • Intermediate strategies—soft quantization, confidence-weighted voting, transmitting a few bits per sample—should achieve sample complexities between 1/d and 1/d², interpolating between the two extremes the paper brackets.
  • The bound provides a practical rule of thumb: if JSD between two distributions is 0.1 bits, the optimal classifier needs roughly 10/0.1 ≈ 100 samples per log(1/ε) factor, while a majority-vote scheme needs roughly 2/0.01 = 200 samples per log(1/ε) factor.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript derives two sample-complexity bounds for binary hypothesis testing in terms of the Jensen-Shannon divergence $d = D_{JS}(P,Q)$: (1) for the log-likelihood-ratio (LLR) classifier, $N ge log(1/epsilon)/(d log 2)$ samples suffice (Theorem 1); (2) for a majority-vote classifier built from Bayes-optimal single-sample decisions, $N ge 2log(1/epsilon)/d^2$ samples suffice (Theorem 2). Theorem 1 chains Chernoff $ge$ Bhattacharyya $ge$ Hellinger $ge$ JSD via three self-contained lemmas. Theorem 2 combines Lin's (1991) single-sample Bayes-error bound $p_e le frac12(1-d)$ with Hoeffding's inequality. The proofs are mathematically correct and the sufficient conditions are valid.

Significance. The paper provides a clean, self-contained derivation of the LLR sample-complexity bound with explicit constants, and the majority-vote bound is a new sufficient condition. The parameter-free nature of both derivations is a strength. However, the paper's central narrative—a qualitative contrast between $1/d$ and $1/d^2$ scaling—requires qualification, as detailed below. The contribution is real but its framing overstates what the bounds imply.

major comments (3)
  1. The framing of 'distinct scalings' (abstract, Discussion) is misleading because the $1/d^2$ majority-vote bound is a worst-case result over distribution pairs with a given JSD, and is loose by a factor of $Theta(1/d)$ for many common distribution families. Concretely: for $P=(frac12+delta, frac12-delta)$, $Q=(frac12-delta, frac12+delta)$, one has $d approx 2delta^2/ln 2$ and $p_e = frac12 - delta = frac12 - Theta(sqrt{d})$, so the majority-vote error exponent is $-log(2sqrt{p_e(1-p_e)}) approx Theta(d)$, giving a true sample complexity of $Theta(1/d)$—the same scaling as the LLR classifier. The $1/d^2$ bound arises because Lin's bound $p_e le frac12(1-d)$ is tight for some distributions (e.g., $P=(1-epsilon, epsilon, 0,ldots)$, $Q=(1-epsilon, 0, epsilon, 0,ldots)$, where $d=epsilon$ and $p_e = frac12(1-d)$ exactly) but loose for others. The paper should state explicitly that Theorem 2 is
  2. Eq. (35) and the surrounding text claim the bound (34) is 'tight to leading order as $d to 0$.' This is tight only relative to the intermediate expression (32), which itself inherits the looseness of Lin's bound (29). For distributions where $p_e = frac12 - Theta(sqrt{d})$, the denominator of (32) is $Theta(d)$, not $Theta(d^2)$, so the overall bound (34) is loose by $Theta(1/d)$ in the exponent. The tightness claim should be qualified: the bound is tight for worst-case distributions (where $p_e = frac12(1-d)$ holds with equality) but not for all distribution pairs.
  3. Discussion, final paragraph: the claim that 'the $1/d^2$ penalty is the price of locality' is overstated. For distribution pairs where the per-sample evidence is spread uniformly (e.g., the binary symmetric example above), the majority-vote classifier achieves $Theta(1/d)$ scaling—no penalty relative to the LLR classifier. The $1/d^2$ penalty arises specifically when the single-sample Bayes error is close to $frac12 - Theta(d)$ rather than $frac12 - Theta(sqrt{d})$, which occurs when the distributions differ on rare events (sparse perturbations). The paper should discuss this distribution-dependence rather than presenting the penalty as universal.
minor comments (4)
  1. Introduction: the paper states that sample complexity 'has been shown to be given by the Hellinger distance [11, 12], up to multiplicative constants.' It would help to state the resulting $Theta(1/H^2)$ scaling explicitly and note that Lemma 3 implies $H^2 ge log(2) cdot d$, connecting this known result to Theorem 1.
  2. Eq. (9): the Chernoff information is defined with natural logarithm, while JSD in Eq. (1) uses $log_2$. The bound (10) mixes these conventions ($C ge log(2) cdot D_{JS}$). This is correct but could confuse readers; a brief note on the convention would help.
  3. The paper does not discuss whether the $1/d^2$ bound could be improved by using a tighter bound on $p_e$ that incorporates additional distributional information (e.g., TV distance or alphabet size). A sentence acknowledging that Lin's bound is the tightest possible bound on $p_e$ in terms of JSD alone would strengthen the presentation.
  4. Reference [12] (Pensia, Jog & Loh, 2025) is cited as 'arXiv (2025)'; please update with publication details if available.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and substantive reading. The referee's core observation—that the 1/d² majority-vote bound is a worst-case result over distribution pairs with a given JSD, and is loose by Θ(1/d) for common families—is correct. We will revise the manuscript to qualify the framing accordingly. The mathematical results themselves are unaffected; what needs to change is the narrative around them.

read point-by-point responses
  1. Referee: The framing of 'distinct scalings' is misleading because the 1/d² majority-vote bound is a worst-case result, loose by Θ(1/d) for many common distribution families (e.g., binary symmetric). The paper should state explicitly that Theorem 2 is a worst-case bound.

    Authors: We agree. The referee's example is correct: for P=(1/2+δ, 1/2−δ), Q=(1/2−δ, 1/2+δ), one has d ≈ 2δ²/ln 2 and p_e = 1/2 − δ = 1/2 − Θ(√d), so the majority-vote error exponent is −log(2√(p_e(1−p_e))) = Θ(d), giving true sample complexity Θ(1/d)—the same scaling as the LLR classifier. The 1/d² bound in Theorem 2 arises because Lin's bound p_e ≤ ½(1−d) is tight for some distributions (e.g., sparse perturbations where the distributions differ on rare events) but loose for others. We will revise the abstract, the Results section header, and the Discussion to state explicitly that Theorem 2 provides a worst-case sufficient condition over all distribution pairs with a given JSD, and that for many common families the majority-vote classifier achieves Θ(1/d) scaling. The phrase 'distinct scalings' will be replaced with language that accurately reflects this qualification. revision: yes

  2. Referee: Eq. (35) and the surrounding text claim the bound (34) is 'tight to leading order as d→0.' This is tight only relative to the intermediate expression (32), which itself inherits the looseness of Lin's bound. The tightness claim should be qualified.

    Authors: The referee is correct. The expansion in (35) shows that (34) is tight relative to (32), but (32) itself inherits the looseness of Lin's bound (29). For distributions where p_e = ½ − Θ(√d), the denominator of (32) is Θ(d), not Θ(d²), so the overall bound (34) is loose by Θ(1/d) in the exponent. We will revise the text around (35) to state that the bound is tight to leading order relative to the intermediate expression (32), and that this tightness holds for worst-case distributions where Lin's bound is achieved with equality, but not for all distribution pairs. We will add the referee's binary symmetric example as an explicit illustration. revision: yes

  3. Referee: The claim that 'the 1/d² penalty is the price of locality' is overstated. For distribution pairs where per-sample evidence is spread uniformly, majority-vote achieves Θ(1/d) scaling—no penalty. The 1/d² penalty arises specifically for sparse perturbations. The paper should discuss this distribution-dependence.

    Authors: We agree that the statement as written is too strong. The 1/d² penalty is not universal; it arises specifically when the single-sample Bayes error is ½ − Θ(d) (i.e., when Lin's bound is tight), which occurs for distributions that differ on rare events. For distributions where the per-sample evidence is spread uniformly—such as the binary symmetric example—the majority-vote classifier incurs no asymptotic penalty relative to the LLR classifier. We will revise the final paragraph of the Discussion to replace the universal claim with a distribution-dependent discussion: the 1/d² bound is the worst-case price of locality, realized for sparse-perturbation distributions, while for uniformly spread differences the penalty vanishes. We will also note that the degree of penalty depends on the relationship between p_e and d, which varies across distribution families. revision: yes

Circularity Check

0 steps flagged

No circularity found. The derivation is self-contained, parameter-free, and relies on external results (Chernoff, Hoeffding, Lin 1991) with no self-citation chain.

full rationale

The paper's two main results are derived from standard external results and self-contained proofs, with no fitted parameters, no self-citations, and no definitional circularity. Theorem 1 chains Chernoff's bound (eq 8, from [14]) through three lemmas (eqs 14, 15, 20) that are each proved from scratch via elementary calculus and the inequality log(y) ≤ y−1. The final bound N ≥ log(1/ε)/(d·log 2) (eq 23) follows by direct substitution. No step reduces to its own input by construction. Theorem 2 combines Hoeffding's inequality (eq 28, from [16]) with Lin's 1991 bound p_e ≤ ½(1−d) (eq 29, from [3], an external citation by different authors) and then applies the same log(y) ≤ y−1 inequality to arrive at N ≥ 2log(1/ε)/d² (eq 34). All references are to external works by different authors. The skeptic's concern that Lin's bound is loose and the true majority-vote scaling may be 1/d rather than 1/d² is a correctness/tightness issue, not a circularity issue — the paper does not define any quantity in terms of its own output, does not fit parameters to data, and does not invoke self-cited results as load-bearing premises. The derivation chain is genuinely self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are introduced—all bounds are parameter-free given P, Q, and ε. No invented entities. The axioms are standard information-theoretic results (Chernoff bound, Hoeffding, concavity of log) plus Lin's (1991) single-sample Bayes error bound, which is the one non-trivial external dependency.

axioms (4)
  • standard math Chernoff bound on Bayes error: p_e^(N) ≤ e^{−N·C(P,Q)}
    Eq. (8), standard result from Cover & Thomas ch. 11.9 [2].
  • standard math Hoeffding's inequality for bounded random variables
    Eq. (28), applied to binary indicators E_i. Ref [16].
  • domain assumption Single-sample Bayes error bound: p_e ≤ ½(1−D_JS(P,Q))
    Eq. (29), from Lin (1991) ref [3]. This is the load-bearing external result for Theorem 2.
  • standard math log(y) ≤ y−1 for y > 0
    Used in Lemma 2 (eq. 15) and in eq. (33). Standard concavity of log.

pith-pipeline@v1.1.0-glm · 10725 in / 2471 out tokens · 431344 ms · 2026-07-08T11:20:06.118344+00:00 · methodology

0 comments
read the original abstract

The Jensen-Shannon divergence (JSD) is a symmetric and bounded measure of the dissimilarity of two probability distributions, which has become a standard tool in statistics, information theory, and machine learning. We complement the understanding of its mathematical properties by presenting an analysis of the amount of data that is needed to distinguish between two distributions, given the value of JSD between them. We find the number of independent and identically distributed samples that suffice for a classifier to determine which of two distributions generated observed data at a desired error rate, for two complementary classifiers: we show that for the log-likelihood-ratio classifier, a sample size that grows as the inverse JSD is sufficient, whereas for a majority-vote classifier assembled from independent single-sample decisions, the sufficient size grows as the squared inverse JSD. These distinct scalings offer operational readings of JSD values and their translation into distinguishability in different contexts.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    & Leibler, R

    Kullback, S. & Leibler, R. A. On Information and Sufficiency.The Annals of Mathematical Statistics22,79–86 (1951)

  2. [2]

    Cover, T. M. & Thomas, J. A.Elements of Information Theory2nd ed. (Wiley, 2005)

  3. [3]

    Divergence Measures Based on the Shannon Entropy.IEEE Trans- actions on Information Theory37,145–151 (1991)

    Lin, J. Divergence Measures Based on the Shannon Entropy.IEEE Trans- actions on Information Theory37,145–151 (1991)

  4. [4]

    Grosse, I.et al.Analysis of symbolic sequences using the Jensen-Shannon divergence.Phys. Rev. E65,041905 (2002)

  5. [5]

    E., Jun, S.-R., Wu, G

    Sims, G. E., Jun, S.-R., Wu, G. A. & Kim, S.-H. Alignment-free genome comparison with feature frequency profiles (FFP) and optimal resolutions. Proceedings of the National Academy of Sciences106,2677–2682 (2009)

  6. [6]

    J.et al.Generative Adversarial Nets.Advances in Neural Information Processing Systems27,2672–2680 (2014)

    Goodfellow, I. J.et al.Generative Adversarial Nets.Advances in Neural Information Processing Systems27,2672–2680 (2014). 9

  7. [7]

    Information-type measures of difference of probability distri- butions and indirect observations.Studia Scientiarum Mathematicarum Hungarica2,299–318 (1967)

    Csisz´ ar, I. Information-type measures of difference of probability distri- butions and indirect observations.Studia Scientiarum Mathematicarum Hungarica2,299–318 (1967)

  8. [8]

    & Shields, P

    Csisz´ ar, I. & Shields, P. Information Theory and Statistics: A Tutorial. Foundations and Trends in Communications and Information Theory1 (Nov. 2004)

  9. [9]

    Amari, S.-I.α-Divergence Is Unique, Belonging to Bothf-Divergence and Bregman Divergence Classes.IEEE Transactions on Information Theory 55,4925–4931 (2009)

  10. [10]

    & Schindelin, J

    Endres, D. & Schindelin, J. A new metric for probability distributions. IEEE Transactions on Information Theory49,1858–1860 (2003)

  11. [11]

    The complexity of massive data set computations.PhD thesis, University of California, Berkeley(2002)

    Bar-Yossef, Z. The complexity of massive data set computations.PhD thesis, University of California, Berkeley(2002)

  12. [12]

    & Loh, P.-L

    Pensia, A., Jog, V. & Loh, P.-L. The Sample Complexity of Simple Binary Hypothesis Testing.arXiv(2025)

  13. [13]

    & Pearson, E

    Neyman, J. & Pearson, E. S. On the Problem of the Most Efficient Tests of Statistical Hypotheses.Philosophical Transactions of the Royal Society of London. Series A231,289–337 (1933)

  14. [14]

    A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the Sum of Observations.The Annals of Mathematical Statistics 23,493–507 (1952)

    Chernoff, H. A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the Sum of Observations.The Annals of Mathematical Statistics 23,493–507 (1952)

  15. [15]

    & Varshney, P

    Chair, Z. & Varshney, P. K. Optimal Data Fusion in Multiple Sensor De- tection Systems.IEEE Transactions on Aerospace and Electronic Systems 22,98–101 (1986)

  16. [16]

    Probability Inequalities for Sums of Bounded Random Vari- ables.Journal of the American Statistical Association58,13–30 (1963)

    Hoeffding, W. Probability Inequalities for Sums of Bounded Random Vari- ables.Journal of the American Statistical Association58,13–30 (1963). 10