Pith. sign in

REVIEW 2 major objections 3 minor 31 references

Even non-exchangeable Monte Carlo copies can give p-values that are valid up to a factor of 2.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:53 UTC pith:ZAI6H4R6

load-bearing objection The finite-m factor-of-2 p-value bound is new and the main proof is clean; the balanced-permutation application is oversold on an unproven asymptotic claim, but that is a fixable overstatement, not a flaw in the central theorem. the 2 major comments →

arxiv 2607.23010 v1 pith:ZAI6H4R6 submitted 2026-07-25 stat.ME math.STstat.TH

Monte Carlo testing: non-asymptotic guarantees without joint exchangeability

Rina Foygel Barber , Aaditya Ramdas This is my paper
classification stat.ME math.STstat.TH MSC 62F0362F40
keywords empirical p-valueMonte Carlo testingexchangeabilityMCMCposterior predictive p-valuepermutation testp*-variablefinite-sample guarantee
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper closes a gap in Monte Carlo hypothesis testing: when copies of the data are drawn from a null-compatible sampling kernel but are not jointly exchangeable with the observed data, the usual empirical p-value can lose type-I error control entirely. The authors prove that one structural condition restores a finite-sample guarantee: if, under the null, the data and a copy are conditionally i.i.d. given some latent variable, then the empirical p-value satisfies P{p_m ≤ α} ≤ 2α for every number of copies m and every α. The result needs no mixing assumptions and explains a striking small-sample phenomenon: parallel MCMC with an even number of steps in a reversible chain is protected, while odd steps are not. The same argument gives factor-of-two validity to Bayesian posterior predictive p-values and to constrained permutation tests built by composing two constrained permutations.

Core claim

The central claim is Theorem 1: for a forward–backward probability kernel P relative to a null Q0 — meaning that X∼Q0 and X′|X∼P(·|X) admit a latent Y such that X and X′ are conditionally i.i.d. given Y — the empirical p-value p_m = (1 + Σ 1{T(X_i) ≥ T(X)})/(m+1) satisfies P{p_m ≤ α} ≤ 2α for every α∈[0,1] and every finite m. The proof shows that p_m is a p*-variable (bounded below uniform in the decreasing-convex order), first for the derandomized limit p∞ and then for finite m via a binomial coupling: binomial mixing preserves the order, and a binomial with a uniform success probability is itself uniform on {0,…,m}, which is what produces the factor of two. A direct corollary is that rejec

What carries the argument

The forward–backward probability kernel is the load-bearing object: a sampling kernel P is forward–backward relative to Q0 when X and X′|X∼P(·|X) are conditionally i.i.d. given a latent Y. This makes the derandomized p∞ = P{T(X′)≥T(X)|X} a conditional expectation of a genuinely valid p-value, so p∞ is a p*-variable; the proof then transfers the p*-property to the Monte Carlo p_m using stochastic-order preservation under binomial mixing (a binomial with a uniform random success probability is uniform on its support). This machinery explains the even/odd dichotomy for reversible MCMC: two steps of a reversible chain are exactly one forward and one backward step through the midpoint state.

Load-bearing premise

The guarantee stands or falls on the forward–backward condition: under the null, the observed data and a Monte Carlo copy must be conditionally independent and identically distributed given some latent variable, a condition strictly stronger than pairwise exchangeability that can fail for even innocent-looking samplers.

What would settle it

Run the paper's autoregressive example with ρ = −0.8, s = 3 (odd), m = 1000, α = 0.05 for 10,000 trials: the empirical type-I error should exceed 0.10 (the 2α line), demonstrating that pairwise exchangeability alone is insufficient. Then run the same example with s = 4 (even): the empirical type-I error must remain below 0.10; any simulation showing P{p_m ≤ α} > 2α for an even-step reversible chain would refute Theorem 1.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For a reversible Markov chain, running s=2r steps before collecting each copy makes the naive parallel MCMC p-value valid up to a factor of 2 even if the chain has not mixed; for odd s this guarantee is lost and the type-I error can be arbitrarily large.
  • Bayesian posterior predictive p-values inherit the factor-of-2 finite-sample guarantee for any m, not only in the m→∞ limit, by taking the latent model parameter θ as the conditioning variable.
  • Constrained permutation tests whose constraint set is not a subgroup regain a factor-of-2 bound when each copy is formed by composing two independent constrained permutations rather than applying one constrained permutation.
  • When the sampling kernel is close in total variation to the conditional null sampler, the multiplicative 2α bound improves to an additive α+ε bound, so mixing and forward–backward structure are complementary rather than competing safeguards.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The theorem can be read as a design rule for samplers: any procedure expressible as one forward step followed by one backward step in a (possibly latent) Markov chain automatically buys a uniform factor-of-two robustness, so even-step composition is a cheap guard against unknown mixing quality.
  • Using p_m/2 as a reported p-value converts the guaranteed bound into an ordinary level-α test; the cost is a factor of two in power, but it is paid uniformly across all m and all mixing regimes, which could simplify practical reporting.
  • A natural testable extension is the serial MCMC sampler, where copies are drawn along a single trajectory and are not conditionally independent given X; the paper identifies this as an open problem, and a similar factor-of-two result there would substantially broaden the computational settings covered.
  • The balanced-permutation bound is governed by Bal_n, the maximal family of permutations whose pairwise ratios are balanced, and Bal_n = n−1 exactly when a Hadamard matrix of order n exists — connecting the statistical guarantee to a classical combinatorial existence question.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper studies Monte Carlo testing when Monte Carlo copies are generated from a null-compatible kernel P, but the observed data and copies are not jointly exchangeable. The main result (Theorem 1) proves that if P is a forward-backward kernel — i.e., under the null there exists a latent variable Y such that X and a draw X' | X ~ P are conditionally i.i.d. given Y — then the empirical p-value p_m satisfies P(p_m ≤ α) ≤ 2α for every finite m and every α. This is applied to reversible MCMC samplers with an even number of steps, to forward-backward compositions for non-reversible chains, to posterior predictive p-values, and to constrained and balanced permutation tests. A second theorem gives additive α+ε-type bounds under total-variation closeness to a conditional null kernel. The paper also contains examples showing that pairwise exchangeability or stationarity alone is not enough for validity.

Significance. The central result is significant and clean: it gives a parameter-free, non-asymptotic Type-I error bound in an intermediate regime between exact null simulation and joint exchangeability. The proof strategy — representing p∞ as a conditional expectation of a superuniform variable, then transferring the p*-variable property to finite m via a binomial stochastic-order argument — is elegant and appears correct. The applications to reversible MCMC and posterior predictive checking are genuinely useful and clearly explained. The paper also provides reproducible code and honest limitations (prespecified statistic, forward-backward condition). However, the balanced-permutation application is advertised in the abstract but rests on unproved quantitative claims about Bal_n and on a uniformity step that is asserted rather than proved; this part needs substantial repair before the paper can be accepted as is.

major comments (2)
  1. [Section 3.3.2 / Appendix A.3.3] The advertised factor-2 interpretation for balanced permutation tests is not supported by the proven results. Proposition 10 gives P{p_m ≤ α} ≤ 2α(1−1/Bal_n) + 2/Bal_n, and the text concludes that the inflation is 'essentially bounded by a factor of 2' because 'we expect Bal_n ≍ n'. Lemma 18 in Appendix A.3.3 proves only the upper bound Bal_n ≤ n−1, with equality iff a Hadamard matrix of dimension n exists. No lower bound beyond the trivial Bal_n ≥ 1 is proved. If Bal_n is O(1) for some n (as could happen if the Hadamard conjecture fails), the bound degrades to roughly α + O(1) and the factor-2 conclusion disappears. The abstract's mention of balanced permutation tests is therefore stronger than what is established. Please either prove a general lower bound, or explicitly state the factor-2 interpretation as conditional on a conjecture and adjust the abstract and Section 3.3.2 accordingl
  2. [Appendix A.3.1, paragraph defining S̃_n,bal] In the proof of Proposition 10, after defining S̃_n,bal, the paper asserts: 'If (σ_1,...,σ_{Bal_n}) ∼ Unif(S̃_n,bal) then σ_i ∘ σ_j^{-1} ∼ Unif(S_n,bal), for each i≠j.' This uniformity claim is essential: it is what allows P* to be written as (1−1/Bal_n)P + (1/Bal_n)δ_X, and hence allows the forward-backward machinery to be applied to P*. The claim is not implied by the exchangeability of the σ_i's, and it does not follow from the definition of Bal_n alone; it requires a counting argument. Without a proof of this uniformity, Proposition 10 is not established. Please supply a rigorous justification or replace the proof.
minor comments (3)
  1. [Section 2.1, Fact 14] Fact 14 — that p = P{T(X')≥T(X)|X,Y} is a valid p-value conditional on Y — is stated as standard but not proved. A one-line verification (e.g., by conditioning on the value of X) would improve readability and make the paper more self-contained.
  2. [Section 3.1, Proposition 6, Example 5] The verification that Example 5 defines a forward-backward kernel is terse, especially for odd r. Spelling out the joint distribution of (X,Y,X') in the odd-r case would prevent confusion about the composition order.
  3. [Section 3.3.1] The distance metric d is not defined before Proposition 8 and Example 9. The reader can infer that d is an arbitrary fixed metric on the design-feature space, but the manuscript should state this explicitly.

Circularity Check

0 steps flagged

No significant circularity: Theorem 1 is self-contained and rests on external p*-variable lemmas; the only caveats are non-load-bearing self-citations and an unproven interpretive claim about Bal_n.

full rationale

The central derivation is not circular. Theorem 1 assumes the forward-backward condition (7) and proves p_m is a p*-variable by writing p_infty = E[tilde_p_infty | X], where tilde_p_infty is superuniform conditional on the latent Y (Fact 14), then transferring to finite m via the binomial stochastic-order lemmas (Prop. 11 and Lemma 13). The p*-variable bound (15) is imported from Wang (2024), Rueschendorf (1982), Meng (1994), and Vovk-Wang (2020), which are external results, not the authors' own prior theorems. The applications verify condition (7) by explicit construction rather than defining the conclusion into the premise: Proposition 7 chooses Y=theta for posterior predictive draws, and Examples 3-5 choose Y as the midpoint of a reversible or composed Markov chain. The only self-citation that might look load-bearing is the side note in Proposition 8 ('See also Ramdas et al. [2023, Theorems 3.1,3.2] for results related to this proposition'), but Proposition 8 is fully proved in the paper via an explicit reversible two-step chain, so the citation is not load-bearing. The paper also honestly flags the prespecified-statistic limitation in Section 5, which is a scope restriction, not a circular step. The balanced-permutation application contains an unproven interpretive claim: the paper states 'we expect Bal_n ≍ n' after proving only the upper bound Bal_n <= n-1 in Lemma 18, with equality iff a Hadamard matrix exists. This is a missing proof / potential overstatement, but it is not a circular step, because Proposition 10's bound is explicitly stated in terms of Bal_n and does not assume the asymptotic. Therefore the derivation chain is self-contained and no circular reduction occurs.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No free parameters are fitted. The central theorem rests on the forward-backward condition and standard Borel regularity; applications add domain assumptions such as reversibility or availability of π^{-1}. The balanced-permutation application additionally assumes Bal_n grows like n, which is only conjectured.

axioms (6)
  • standard math X is a standard Borel space, providing standard regularity for conditional distributions.
    Stated in footnote 3; needed for Fact 14 and conditional expectations.
  • domain assumption P is compatible with the null, i.e., if X∼Q0 and X'|X∼P(·|X), then X'∼Q0 (Eq. 6).
    Defines the class of sampling kernels under study; if X' is not null, the p-value is not testing H0.
  • domain assumption Forward-backward condition (Eq. 7): there exists a latent Y such that X and X' are conditionally i.i.d. given Y.
    The central structural assumption; fails for odd-step reversible chains and general nonreversible chains, and is exactly what Theorem 1 requires.
  • domain assumption Test statistic T is prespecified and fixed before seeing copies.
    Section 5 limitation; the proof evaluates T separately on X and each copy, and the guarantee does not cover symmetric refitting across all copies.
  • domain assumption For MCMC applications, the chain has stationary distribution Q0 and either is reversible or has an exactly simulable time-reversal kernel π^{-1}.
    Required for Examples 3–5; in non-reversible chains without reverse moves the forward-backward construction is not implementable.
  • domain assumption For balanced permutations, the practical interpretation assumes Bal_n ≍ n.
    Appendix A.3.3 only proves Bal_n ≤ n−1 with equality iff a Hadamard matrix exists; the claimed factor-2 practical bound depends on the unproven Hadamard conjecture.

pith-pipeline@v1.3.0-alltime-deepseek · 20923 in / 29516 out tokens · 266747 ms · 2026-08-01T03:53:18.580563+00:00 · methodology

0 comments
read the original abstract

In hypothesis testing, Monte Carlo tests are usually justified either by exact null simulation or by joint exchangeability of the observed data and its simulated copies. This leaves a gap for common computational procedures, such as parallel MCMC sampling initialized at the observed data, where each copy may be marginally null and even pairwise exchangeable with the observation, but the full collection is not jointly exchangeable. In such cases the usual empirical p-value can be invalid when the chain has not mixed, while exactly exchangeable constructions such as the Besag--Clifford hub-and-spoke sampler may suffer from high conditional Monte Carlo variability. We give finite-sample guarantees for this intermediate regime. If, under the null, the observed data $X$ and a copy $X'\sim P(\cdot\mid X)$ are conditionally i.i.d.\ given a latent variable, then for any prespecified statistic and any finite number $m$ of conditionally independent Monte Carlo copies, the resulting empirical p-value obeys $\mathbb P\{p_m\le \alpha\}\le 2\alpha$. This guarantee requires no mixing conditions and holds for any number of copies $m$, and it explains finite-sample oscillatory behavior in inference via MCMC sampling. In addition, we further show that the guarantee provides insights into inference problems arising in other settings, including inference on Bayesian models (recovering a classical result showing validity up to a factor of $2$ for posterior predictive p-values), and inference via balanced permutation tests.

Figures

Figures reproduced from arXiv: 2607.23010 by Aaditya Ramdas, Rina Foygel Barber.

Figure 1
Figure 1. Figure 1: An example of Type-I error under the naive sampling scheme (2). The figure [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of the naive sampling strategy (2) (left) and Besag and Clifford [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The left plots show the marginal distribution of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An example of Type-I error under the naive sampling scheme (2), when the Markov [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An illustration of the forward–backward probability kernels [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 2 canonical work pages

  1. [1]

    Testing goodness-of-fit and conditional independence with approximate co-sufficient sampling

    Rina Foygel Barber and Lucas Janson. Testing goodness-of-fit and conditional independence with approximate co-sufficient sampling. The Annals of Statistics, 50 0 (5): 0 2514--2544, 2022. doi:10.1214/22-AOS2187

    work page doi:10.1214/22-aos2187 2022

  2. [2]

    Berrett, Yi Wang, Rina Foygel Barber, and Richard J

    Thomas B. Berrett, Yi Wang, Rina Foygel Barber, and Richard J. Samworth. The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society: Series B, 82 0 (1): 0 175--197, 2020. doi:10.1111/rssb.12340

    doi:10.1111/rssb.12340 2020

  3. [3]

    Generalized M onte C arlo significance tests

    Julian Besag and Peter Clifford. Generalized M onte C arlo significance tests. Biometrika, 76 0 (4): 0 633--642, 1989

    1989

  4. [4]

    Conditioning on posterior samples for flexible frequentist goodness-of-fit testing

    Ritwik Bhaduri, Aabesh Bhattacharyya, Rina Foygel Barber, and Lucas Janson. Conditioning on posterior samples for flexible frequentist goodness-of-fit testing. Biometrika, page asag019, 2026

    2026

  5. [5]

    Panning for gold: `model-x' knockoffs for high dimensional controlled variable selection

    Emmanuel Cand \`e s, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold: `model-x' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B, 80 0 (3): 0 551--577, 2018. doi:10.1111/rssb.12265

    doi:10.1111/rssb.12265 2018

  6. [6]

    Sequential monte carlo methods for permutation tests on truncated data

    Yuguo Chen and Jun S Liu. Sequential monte carlo methods for permutation tests on truncated data. Statistica Sinica, pages 857--872, 2007

    2007

  7. [7]

    Testing the mantel statistic with a spatially-constrained permutation procedure

    Julie Crabot, Sylvie Clappe, St \'e phane Dray, and Thibault Datry. Testing the mantel statistic with a spatially-constrained permutation procedure. Methods in Ecology and Evolution, 10 0 (4): 0 532--540, 2019

    2019

  8. [8]

    Statistical problems involving permutations with restricted positions

    Persi Diaconis, Ronald Graham, and Susan P Holmes. Statistical problems involving permutations with restricted positions. Lecture Notes-Monograph Series, pages 195--222, 2001

    2001

  9. [9]

    Real analysis and probability

    Richard M Dudley. Real analysis and probability. Chapman and Hall/CRC, 2018

    2018

  10. [10]

    Nonparametric methods for doubly truncated data

    Bradley Efron and Vahe Petrosian. Nonparametric methods for doubly truncated data. Journal of the American Statistical Association, 94 0 (447): 0 824--834, 1999

    1999

  11. [11]

    An introduction to probability theory and its applications, Volume 2, volume 2

    William Feller. An introduction to probability theory and its applications, Volume 2, volume 2. John Wiley & Sons, 1991

    1991

  12. [12]

    Mathematics of a lady tasting tea

    Ronald A Fisher. Mathematics of a lady tasting tea. The World of Mathematics, 3 0 (part 8): 0 1514--1521, 1956

    1956

  13. [13]

    Posterior predictive assessment of model fitness via realized discrepancies

    Andrew Gelman, Xiao-Li Meng, and Hal Stern. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6 0 (4): 0 733--760, 1996

    1996

  14. [14]

    Exact testing with random permutations

    Jesse Hemerik and Jelle Goeman. Exact testing with random permutations. Test, 27 0 (4): 0 811--825, 2018

    2018

  15. [15]

    Dahl, and Gunnhildur H \"o gnad \'o ttir Steinbakk

    Nils Lid Hjort, Fredrik A. Dahl, and Gunnhildur H \"o gnad \'o ttir Steinbakk. Post-processing posterior predictive p values. Journal of the American Statistical Association, 101 0 (475): 0 1157--1174, 2006

    2006

  16. [16]

    Hadamard matrices and their applications

    Kathy Horadam. Hadamard matrices and their applications. Princeton University Press, 2012

    2012

  17. [17]

    Markov chain monte carlo significance tests

    Michael Howes. Markov chain monte carlo significance tests. Statistical Science, 41 0 (1): 0 188--200, 2026. doi:10.1214/24-STS958

    work page doi:10.1214/24-sts958 2026

  18. [18]

    On divergences and informations in statistics and information theory

    Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52 0 (10): 0 4394--4412, 2006

    2006

  19. [19]

    Posterior predictive p-values

    Xiao-Li Meng. Posterior predictive p-values. The Annals of Statistics, 22 0 (3): 0 1142--1160, 1994

    1994

  20. [20]

    Taking population stratification into account by local permutations in rare-variant association studies on small samples

    Jimmy Mullaert, Matthieu Bouaziz, Yoann Seeleuthner, Benedetta Bigio, Jean-Laurent Casanova, Alexandre Alca \" s, Laurent Abel, and Aur \'e lie Cobat. Taking population stratification into account by local permutations in rare-variant association studies on small samples. Genetic epidemiology, 45 0 (8): 0 821--829, 2021

    2021

  21. [21]

    Significance tests which may be applied to samples from any populations

    Edwin JG Pitman. Significance tests which may be applied to samples from any populations. Supplement to the Journal of the Royal Statistical Society, 4 0 (1): 0 119--130, 1937

    1937

  22. [22]

    Permutation tests using arbitrary permutation distributions

    Aaditya Ramdas, Rina Foygel Barber, Emmanuel J Cand \`e s, and Ryan J Tibshirani. Permutation tests using arbitrary permutation distributions. Sankhya A, 85 0 (2): 0 1156--1177, 2023

    2023

  23. [23]

    Fundamentals of Stein's method

    Nathan Ross. Fundamentals of Stein's method . Probability Surveys, 8: 0 210 -- 293, 2011

    2011

  24. [24]

    Donald B. Rubin. Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, 12 0 (4): 0 1151--1172, 1984

    1984

  25. [25]

    Random variables with maximum sums

    Ludger R \"u schendorf. Random variables with maximum sums. Advances in Applied Probability, 14 0 (3): 0 623--632, 1982

    1982

  26. [26]

    On the dispersion of mixtures

    Tore Schweder. On the dispersion of mixtures. Scandinavian Journal of Statistics, pages 165--169, 1982

    1982

  27. [27]

    On mixtures from exponential families

    Moshe Shaked. On mixtures from exponential families. Journal of the Royal Statistical Society: Series B (Methodological), 42 0 (2): 0 192--198, 1980

    1980

  28. [28]

    Properties of balanced permutations

    Lucinda K Southworth, Stuart K Kim, and Art B Owen. Properties of balanced permutations. Journal of Computational Biology, 16 0 (4): 0 625--638, 2009

    2009

  29. [29]

    Combining p-values via averaging

    Vladimir Vovk and Ruodu Wang. Combining p-values via averaging. Biometrika, 107 0 (4): 0 791--808, 2020

    2020

  30. [30]

    Testing with p*-values: Between p-values, mid p-values, and e-values

    Ruodu Wang. Testing with p*-values: Between p-values, mid p-values, and e-values. Bernoulli, 30 0 (2): 0 1313--1346, 2024

    2024

  31. [31]

    Approximate co-sufficient sampling with regularization

    Wanrong Zhu and Rina Foygel Barber. Approximate co-sufficient sampling with regularization. arXiv preprint arXiv:2309.08063, 2023

    arXiv 2023