Pith. sign in

REVIEW 7 minor 35 references

Data-adaptive binning keeps local permutation tests valid and powerful.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:28 UTC pith:75DV6DRD

load-bearing objection Rigorous extension of local permutation tests to adaptive bins, with a genuinely new product-form Type I error bound and a constant-factor power comparison to the oracle; the central smoothness condition is explicit and the proof seems sound.

arxiv 2607.20751 v1 pith:75DV6DRD submitted 2026-07-22 stat.ME math.STstat.TH

Local permutation tests for conditional independence: an adaptive binning perspective

David Chen , Rohan Hore , Rina Foygel Barber This is my paper
classification stat.ME math.STstat.TH MSC 62G1062H15
keywords conditional independence testinglocal permutation testdata-adaptive binningtotal variation distanceType I errorNeyman-Pearson powerlinear confounder modelfinite-sample validity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that local permutation tests for conditional independence remain approximately valid when the bins defining the permutations are chosen adaptively from the observed confounder values. Its central finite-sample guarantee bounds the conditional Type I error by α plus a product of within-bin total-variation distances, so the test is valid whenever at least one of the conditional distributions of X|Z or Y|Z is approximately constant inside each bin. For test statistics that decompose as a sum over bins, the paper gives a sharper bound that permits larger bins and weaker smoothness. On the power side, it shows that an adaptive local permutation test can match the signal-to-noise ratio of the oracle likelihood-ratio test up to a constant factor, even with only two observations per bin, and that constant bin sizes recover the same detection boundary as growing bin sizes in a linear confounder model.

Core claim

The central claim is that the adaptive local permutation test—where bins are built from Z alone and a bin-symmetric statistic is evaluated under all within-bin permutations—has conditional Type I error at most α + 4Σ_k (m_k − 1) times the product of within-bin total-variation distances for the X|Z and Y|Z conditional distributions. This product structure makes the excess error small when either conditional law is smooth, giving a doubly robust validity guarantee. The paper further proves that with oracle knowledge of the likelihood ratio, there is a bin statistic using only two points per bin whose signal-to-noise ratio is at least one quarter of the oracle test's signal-to-noise ratio, up t

What carries the argument

The engine is the group of bin-preserving permutations: after sorting the data into data-adaptive bins, the test permutes X values within each bin and compares the observed statistic to its permuted distribution. The key identity is a transposition-wise bound: swapping two indices i, j within a bin moves the joint law of permuted (X, Y) by at most 4 times the product of the total-variation distances between P(X|Z_i) and P(X|Z_j) and between P(Y|Z_i) and P(Y|Z_j); summing over transpositions yields the product-type Type I error bound. For power, the corresponding machinery is the signal-to-noise ratio SNR_LPT of a linearly decomposable bin statistic, which is compared to the oracle SNR throug

Load-bearing premise

The load-bearing premise is that within each bin the conditional distributions of X|Z and Y|Z are approximately constant—in the power theorem this appears as condition (9), which requires the average within-bin squared deviations to be small compared with the squared signal strength; when this fails, the Type I error can be inflated even though the test statistic itself is well behaved.

What would settle it

Simulate the paper's Experiment 1 null model with Z uniform on [0, θ] and X, Y independent and uniform on [Z, Z+1], set θ = n^{3/4}, and run the adaptive local permutation test with two points per bin; if the rejection rate converges to the nominal α rather than exceeding it by an amount reflecting the within-bin total-variation product, the central Type I error bound would be false. The same experiment at θ = n^{1/2} should show approximate validity, matching the paper's reported curves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can use balanced, data-adaptive bins—such as equal-size bins or sorted pairs—instead of fixed grids without losing approximate validity, provided within-bin conditional variation is small.
  • The product form of the Type I error bound implies a double-robustness property: it suffices that either the X-side or the Y-side conditional distribution is nearly constant within bins.
  • Even the extreme choice of two observations per bin loses only a constant factor of oracle signal strength, so there is little statistical reason to pay the validity cost of large bins.
  • For linearly decomposable statistics, the validity range extends to wider bins and weaker smoothness, yielding a concrete rule of thumb: choose the number of bins K to grow faster than n^{2/5}.
  • In the linear confounder model, equally sized bins maximize power, and the detection threshold of the adaptive-LPT coincides with that of the oracle likelihood-ratio benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 1/4 constant in the oracle-power theorem is a conservative upper bound; the paper's own linear-model calculation at m=2 gives a constant 1/√2 ≈ 0.707, suggesting the typical power loss is much smaller than the worst-case bound.
  • The product-TV bound suggests a directly testable design rule: choose binning to minimize the product of within-bin total-variation deviations for X|Z and Y|Z, rather than the sum, since only the product appears in the Type I error guarantee.
  • Because the validity theorem covers any bin-symmetric statistic, the framework should extend to kernel- or distance-based bin statistics; the open question is whether their signal-to-noise ratio can be compared to the oracle in the same way as the covariance-based statistics analyzed here.
  • The proof that nearest-neighbor total-variation products can vanish arbitrarily slowly makes the gap to full distribution-freeness precise; turning that rate explicit for a given model class would tell practitioners exactly when adaptive binning can be trusted.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 7 minor

Summary. The paper studies local permutation tests (LPT) for conditional independence and extends them to data-adaptive binning. Section 3 gives finite-sample Type I error bounds: Theorem 1 shows that for any bin-symmetric statistic and any partition constructed from Z, the conditional rejection probability is bounded by alpha plus a product-type total-variation term; Theorem 2 improves this for linearly decomposable statistics using a Berry-Esseen argument. Section 4 develops a power analysis benchmarked against an oracle likelihood-ratio test: Theorem 6 shows that an oracle-informed LPT with two points per bin achieves SNR at least a constant fraction of the oracle SNR, and Theorem 8 gives an explicit SNR formula in a linear confounder model, leading to the conclusion that equal-size bins are optimal and that increasing bin size has diminishing returns. The theoretical claims are supported by three simulation experiments, including a comparison of adaptive versus fixed binning.

Significance. If correct, this is a valuable contribution to conditional independence testing. Theorem 1's product structure in the Type I error bound is a genuine improvement over earlier sum-type bounds and yields a double-robustness interpretation. The power analysis is also substantial: Theorem 8 provides explicit, interpretable formulas for SNR under a concrete model, and the conclusion that constant bin sizes can match the detection boundary of the oracle test is practically useful. The paper ships detailed proofs in the appendix, reproducible simulation code, and explicitly states the main limitation (within-bin smoothness condition (9)). The formal theorems are stated with precise technical conditions, and the informal versions in the main text are clearly flagged as such.

minor comments (7)
  1. [Section 4.2.1, after Theorem 6] Typo: 'in signal strengt' should be 'in signal strength'.
  2. [Section 5.2, Figure 4 caption] The caption says 'The black dotted line is the oracle test (Theorem 8)', but the oracle test is Theorem 7. Theorem 8 is the LPT power result.
  3. [Section 2.2.2 and Experiment 3] The Gaussian kernel is written as exp(-(u-v)^2) 2 , which appears to be missing the divisor: it should be exp(-(u-v)^2 / 2).
  4. [Theorems 4-6] These are labeled 'Informal' in the main text but are cited later as theorems. It would help to include a sentence in the main text pointing to the precise formal statements (Theorems 13, 15, and 16) and to the assumptions A1-A8.
  5. [Theorem 2] The quantity eps_n is defined as an expectation over X,Y given Z, so it is not a data-only quantity. The statement would be clearer if this were noted explicitly, since a reader may otherwise think it is computable from the sample alone.
  6. [Corollary 1] The phrase 'a.e.' is terse. It should be stated that the bound holds on the event {max_k max_{i,j in B_k} d(Z_i,Z_j) <= h_n}, which is assumed to occur almost surely.
  7. [Figures 1 and 2] The axis labels such as 'delta_n << sqrt(eps_n delta_n)' are somewhat cryptic. A one-sentence explanation of what '<<' means in the legend would improve readability.

Circularity Check

0 steps flagged

No significant circularity detected; the paper's bounds and power analyses are derived from explicit assumptions and constructions rather than from the conclusions being tested.

full rationale

I checked the claimed derivation chain, focusing on the load-bearing steps: Theorem 1's finite-sample Type I error bound, Theorem 2's refined bound for linearly decomposable statistics, Theorem 6's oracle-informed LPT power comparison, and Theorem 8's SNR formula in the linear confounder model. Theorem 1 is a direct mathematical inequality: conditional on Z, the permutation p-value is approximately uniform when the product of within-bin total variation distances of P_{X|Z} and P_{Y|Z} is small. The proof (Lemmas 1 and 2) uses only bin-symmetry of the statistic, the group structure of bin-preserving permutations, and elementary total-variation bounds. There is no parameter fitted to a subset of data and then relabeled as a prediction; the bound is stated as a theorem with an explicit δ_n term. Theorem 2 follows the same structure: it adds linear decomposability and uses a Berry-Esseen-type argument, with ε_n measuring the range-to-variance ratio of the bin statistics. Again, this is a finite-sample inequality, not a fitted result. The power analysis likewise derives rather than assumes its conclusions. Theorem 4 computes oracle power from the Neyman-Pearson lemma and an explicit SNR_ORC. Theorem 5 expresses LPT power in terms of an analogous SNR_LPT. Theorem 6 constructs a concrete LPT statistic from the oracle log-likelihood ratio and proves, under stated vanishing-error assumptions (Assumptions A7 and A8), that SNR_LPT is within a constant factor of SNR_ORC. This is an existence result with an explicit construction, not a circular definition. In the linear confounder model, Theorem 8 obtains its SNR_LPT expression by explicit moment calculations for the covariance-type bin statistic. The within-bin smoothness conditions (8) and (9) are stated assumptions; they are not derived from the SNR formula, and the theorem does not hide them. The paper even flags in Section 5.2 that large bins can violate these conditions and cause severe Type I error inflation, which confirms that the assumptions are load-bearing but explicit. The only self-citations appear as background references (e.g., Hore et al. 2025, Barber et al. 2020, Berrett et al. 2020) and are not used as the unique justification for any central claim. No uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in via self-citation: the data-adaptive binning is defined directly in Section 2. The paper is self-contained against external benchmarks and simulation res

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claims rest on explicit smoothness assumptions, i.i.d. sampling, bin symmetry, and standard probabilistic inequalities. No free parameters are fitted to data; the constants in the bounds are explicit and not tuned to the simulations.

axioms (6)
  • domain assumption Observations are drawn i.i.d. from an unknown joint distribution P and the null hypothesis X⊥⊥Y|Z holds.
    The validity results (Theorems 1 and 2) are statements about distributions satisfying the conditional independence null.
  • domain assumption Conditional distributions P_{X|Z} and P_{Y|Z} are Lipschitz in total variation (Definition 3) for the smoothness corollaries.
    This is needed to translate the TV terms δ_n into the bin-diameter bound h_n and to obtain asymptotic validity.
  • domain assumption The conditional distributions admit densities with respect to a common σ-finite measure (Theorem 3).
    This assumption is stated explicitly in Theorem 3 and is used in the proof of the convergence of the nearest-neighbor TV product.
  • domain assumption Test statistics are bin-symmetric (Definition 2) and bins are constructed using only Z values.
    Bin-symmetry is necessary for the permutation distribution to be calibrated; the restriction that bins depend only on Z ensures the null distribution is not distorted.
  • standard math Berry–Esseen / CLT-type approximations for sums of independent non-identical random variables hold in the specified regimes.
    Theorem 2 uses Berry–Esseen (Lemma 8) and Theorems 4, 5, and 8 use Lyapunov-type conditions to justify the Gaussian power approximations.
  • domain assumption In Section 4.3, the data follow the linear confounder model X=f1(Z)+β1 U+ε1, Y=f2(Z)+β2 U+ε2 with independent Gaussian U, ε1, ε2.
    The Gaussianity is used to compute the oracle likelihood-ratio SNR and to derive the exact variance formulas in Theorem 8.

pith-pipeline@v1.3.0-alltime-deepseek · 68726 in / 9901 out tokens · 89165 ms · 2026-08-01T09:28:29.162419+00:00 · methodology

0 comments
read the original abstract

In this work, we study the problem of testing conditional independence between random variables $X$ and $Y$ given a confounder $Z$. The local permutation test (LPT) offers a principled approach to this problem by partitioning the $Z$-space into pre-specified bins, and permuting the $X$ and $Y$ data within each bin, to assess the significance of an observed test statistic. However, when the partitions are pre-fixed, the resulting partition can be poorly balanced, as some bins may contain most of the samples while others contain only a few. This motivates the use of data-adaptive binning strategies, such as equisized bins with a fixed (typically small) number of points. We study this natural and practically important extension of LPT, providing finite-sample bounds on the Type I error for an arbitrary test statistic, providing stronger validity results than previously known. We also show that LPT attains power comparable to the oracle likelihood ratio tests derived from the Neyman-Pearson lemma. Within a linear confounder model class, we further analyze the effect of bin size and demonstrate that constant bin sizes can match the performance of partitions with growing bin-size. These results, further supported by extensive numerical simulations, position the proposed data-adaptive strategy as both practically implementable and statistically efficient.

Figures

Figures reproduced from arXiv: 2607.20751 by David Chen, Rina Foygel Barber, Rohan Hore.

Figure 1
Figure 1. Figure 1: An illustration of the discussion of Section [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of the discussion of Section [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The plot displays Type I error rate of adaptive-LPT in Experiment 1, averaged over 1000 inde [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Type I error (left) and power (right) in Experiment 2 (Section [PITH_FULL_IMAGE:figures/full_fig_p023_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Demonstration of different binning procedures in Experiment 3 (see Section [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Results for Experiment 3 (see Section 5.3), showing Type I error (left) and power (right), averaged over 1000 independent trials; standard error bars are shown, but are not easily visible as they are smaller than the points. Solid lines represent a data-adaptive binning strategy, whereas dashed lines represent a fixed-partition binning strategy. 6 Conclusion In this paper, we propose data-adaptive binning … view at source ↗
Figure 7
Figure 7. Figure 7: We construct a Cantor-like set S = T∞ m=1 Sm and consider the Z1 which fall into the gaps Tm,ℓ. As long as we choose the lengths sm of the gaps Tm,ℓ to decay sufficiently slowly, we will get slow convergence in expectation for the total variation distance between X1 and Xn,1. To get an intuitive grasp on why the rate of removal matters, consider widening the lowest-most removals T3,• and narrowing the earl… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 3 linked inside Pith

  1. [1]

    2012 , publisher=

    Categorical data analysis , author=. 2012 , publisher=

    2012

  2. [2]

    The Annals of Statistics , volume=

    A simple measure of conditional dependence , author=. The Annals of Statistics , volume=. 2021 , publisher=

    2021

  3. [3]

    The Annals of Statistics , volume=

    Robust inference with knockoffs , author=. The Annals of Statistics , volume=. 2020 , publisher=

    2020

  4. [4]

    and Wang, Yi and Barber, Rina Foygel and Samworth, Richard J

    Berrett, Thomas B. and Wang, Yi and Barber, Rina Foygel and Samworth, Richard J. , TITLE =. J. R. Stat. Soc. Ser. B. Stat. Methodol. , FJOURNAL =. 2020 , NUMBER =

    2020

  5. [5]

    Cand\`es, Emmanuel and Fan, Yingying and Janson, Lucas and Lv, Jinchi , TITLE =. J. R. Stat. Soc. Ser. B. Stat. Methodol. , FJOURNAL =. 2018 , NUMBER =. doi:10.1111/rssb.12265 , URL =

    doi:10.1111/rssb.12265 2018

  6. [6]

    Maathuis and Markus Kalisch and Thomas S

    Diego Colombo and Marloes H. Maathuis and Markus Kalisch and Thomas S. Richardson , title =. The Annals of Statistics , number =. 2012 , doi =

    2012

  7. [7]

    IEEE transactions on neural networks and learning systems , volume=

    Significance tests of feature relevance for a black-box learner , author=. IEEE transactions on neural networks and learning systems , volume=. 2022 , publisher=

    2022

  8. [8]

    arXiv preprint arXiv:1810.08693 , year=

    The total variation distance between high-dimensional Gaussians with the same mean , author=. arXiv preprint arXiv:1810.08693 , year=

    Pith/arXiv arXiv

  9. [9]

    Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume =

    Evans, Dafydd , title =. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , volume =. 2008 , month =

    2008

  10. [10]

    , author=

    035: The Distribution of the Partial Correlation Coefficient. , author=

  11. [11]

    Advances in neural information processing systems , volume=

    Equality of opportunity in supervised learning , author=. Advances in neural information processing systems , volume=

  12. [12]

    Biometrika , volume=

    Conservative hypothesis tests and confidence intervals using importance sampling , author=. Biometrika , volume=. 2012 , publisher=

    2012

  13. [13]

    arXiv preprint arXiv:2501.06133 , year=

    Testing conditional independence under isotonicity , author=. arXiv preprint arXiv:2501.06133 , year=

    Pith/arXiv arXiv

  14. [14]

    Journal of the American Statistical Association , volume=

    A two-sample conditional distribution test using conformal prediction and weighted rank sum , author=. Journal of the American Statistical Association , volume=. 2024 , publisher=

    2024

  15. [15]

    CoRR , volume=

    Jensen Hwa and Qingyu Zhao and Aditya Lahiri and Adnan Masood and Babak Salimi and Ehsan Adeli , title=. CoRR , volume=. 2024 , cdate=

    2024

  16. [16]

    , author=

    Estimating high-dimensional directed acyclic graphs with the PC-algorithm. , author=. Journal of Machine Learning Research , volume=

  17. [17]

    The Annals of Statistics , volume=

    Local permutation tests for conditional independence , author=. The Annals of Statistics , volume=. 2022 , publisher=

    2022

  18. [18]

    2009 , publisher=

    Probabilistic graphical models: principles and techniques , author=. 2009 , publisher=

    2009

  19. [19]

    Precis Med , volume=

    Annual review of statistics and its application , author=. Precis Med , volume=

  20. [20]

    The Annals of Statistics , volume=

    The Projected Covariance Measure for assumption-lean variable significance testing , author=. The Annals of Statistics , volume=

  21. [21]

    The Annals of Statistics , volume=

    Reconciling model-X and doubly robust approaches to conditional independence testing , author=. The Annals of Statistics , volume=. 2024 , publisher=

    2024

  22. [22]

    Sutherland and Victor Veitch and Arthur Gretton , title=

    Roman Pogodin and Namrata Deka and Yazhe Li and Danica J. Sutherland and Victor Veitch and Arthur Gretton , title=. 2023 , cdate=

    2023

  23. [23]

    International Conference on Artificial Intelligence and Statistics , pages=

    Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2018 , organization=

    2018

  24. [24]

    Advances in neural information processing systems , volume=

    Model-powered conditional independence test , author=. Advances in neural information processing systems , volume=

  25. [25]

    The hardness of conditional independence testing and the generalised covariance measure , author=

  26. [26]

    Biometrics , volume=

    Nonparametric variable importance assessment using machine learning techniques , author=. Biometrics , volume=. 2021 , publisher=

    2021

  27. [27]

    Advances in neural information processing systems , volume=

    A kernel statistical test of independence , author=. Advances in neural information processing systems , volume=

  28. [28]

    Van Erven, Tim and Harremos, Peter , journal=. R. 2014 , publisher=

    2014

  29. [29]

    2003 , publisher=

    Abstract Algebra , author=. 2003 , publisher=

    2003

  30. [30]

    Gaussian Hilbert Spaces , publisher=

    Janson, Svante , year=. Gaussian Hilbert Spaces , publisher=

  31. [31]

    2018 , publisher=

    Algebraic inequalities , author=. 2018 , publisher=

    2018

  32. [32]

    Journal of the American Statistical Association , volume=

    A new coefficient of correlation , author=. Journal of the American Statistical Association , volume=. 2021 , publisher=

    2021

  33. [33]

    Biometrika , volume=

    A reduction formula for normal multivariate integrals , author=. Biometrika , volume=

  34. [34]

    Philosophical Transactions of the Royal Society of London

    On the application of the theory of error to cases of normal distribution and normal correlation , author=. Philosophical Transactions of the Royal Society of London. Series A , volume=

  35. [35]

    arXiv preprint arXiv:2103.11038 , year=

    Stochastic comparisons, differential entropy and varentropy for distributions induced by probability density functions , author=. arXiv preprint arXiv:2103.11038 , year=

    Pith/arXiv arXiv