REVIEW 3 major objections 5 minor 19 references
An MCMC hypothesis test to check a claimed sampler: applied to a claimed sampler for the G-Wishart distribution
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A widely used G-Wishart sampler fails an exact exchangeability test, so it draws from the wrong distribution.
desk verdict A clean, valid sampler-audit test with strong evidence against Lenkoski's G-Wishart sampler, worth a serious referee despite a slightly over-categorical central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Proposition 1: if $x$ is drawn from $p$ and $y$ is obtained after $r$ updates of any transition kernel satisfying detailed balance with $p$, then $x$ and $y$ are exchangeable. The paper turns this symmetry into a test: starting Metropolis--Hastings from claimed samples, a correct sampler makes initial and final states exchangeable, while a biased sampler produces a burn-in shift. The test statistic compares empirical low quantiles of $\ln|Q|$ between initial and final states, and the p-value comes from randomly flipping the two entries within each row of the paired data. The same exchangeability logic also diagnoses why the claimed sampler's fixed-point construction fails: its justification relies on matching conditional distributions, which requires a positivity condition that the family of positive definite matrices does not satisfy.
What would settle it
A third implementation of the Lenkoski algorithm, written independently from the original paper and applied to the ten-node decomposable graph with $s=100{,}000$ and $q=999{,}999$, should also give p-values at the $1/(q+1)$ floor; if instead the p-values are scattered across $(0,1)$, the rejections are an artifact of a shared implementation bug and the central claim collapses.
Extended reading notes
Core claim
The central claim, stated plainly, is that the Lenkoski (2013) direct sampler, despite being widely used as an exact G-Wishart generator, produces samples from a different distribution. The evidence is the exchangeability test described above: with $s=10{,}000$ claimed samples followed by $r=3m$ updates of a detailed-balance block-Gibbs kernel, all five p-values for both ten-node graphs are at the smallest possible value $1/(q+1)$, and the small non-decomposable graph also produces very small p-values. For the small decomposable graph the deviation is smaller, but an $s=100{,}000$ run still rejects. A control exact sampler for decomposable graphs shows no such rejection. The paper attributes the failure to the invalid use of conditional-specification arguments on the non-rectangular positive-definite support, and notes that two independent implementations give the same result.
Load-bearing premise
The conclusion depends on the two implementations of the Lenkoski algorithm being faithful to the algorithm as intended; if both implementations share the same misunderstanding, the test would be rejecting a distorted version rather than the actual sampler.
Editorial extensions
If this is right
- Bayesian inference in Gaussian graphical models that uses the Lenkoski sampler inside an exchange algorithm is not exactly targeting the intended posterior; the induced bias is unknown and could persist in the limiting distribution.
- The new test provides a general way to validate any claimed sampler whenever a detailed-balance kernel for the target is available, with no reference sampler or analytic moments needed.
- A large sample size can be necessary to expose a wrong sampler: the small decomposable graph looks fine at $s=10{,}000$ but is rejected at $s=100{,}000$, so non-rejection at one sample size is weak evidence of correctness.
- Running the Lenkoski fixed-point iterations to machine-precision convergence does not cure the problem; the converged output still fails the test, so the error is in the algorithm's target distribution, not in incomplete iteration.
Reading between the lines
- A natural extension, not pursued in the paper, is to use the same test with several functions $h$ and update counts $r$ to measure how far the claimed sampler's output is from the G-Wishart, for example in total variation, and how that distance scales with graph size, $\delta$, and $D$.
- Because the test needs only a detailed-balance kernel, it could serve as a routine acceptance diagnostic for approximate samplers in any field, with the p-value providing a checkable certificate of correctness at a chosen sample size.
- Repairing the direct G-Wishart sampler will probably require constructing a valid joint update rather than matching conditional distributions, since the positivity condition that would justify the conditional-specification route is genuinely unavailable on the space of positive definite matrices.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a Monte Carlo hypothesis test for checking whether a claimed exact sampler really samples from a stated target distribution. The test initializes a detailed-balance Metropolis--Hastings chain with a sample from the claimed sampler, runs a fixed number of updates, and uses the resulting exchangeability of initial and final states under the null hypothesis to build a permutation-based p-value. The method is applied to the Lenkoski (2013) G-Wishart sampler, with an exact sampler for decomposable graphs as a control. The reported p-values reject the claimed sampler for the non-decomposable graphs and for a small decomposable graph when the sample size is increased, while the exact-sampler control shows no rejection.
Significance. The proposed test is a clean and correct formalization of an idea from Besag and Clifford (1989): the exchangeability argument in Section 2 and the validity of the permutation p-value in Section 3 are sound, and the method needs only a detailed-balance kernel rather than an alternative exact sampler. This is a useful diagnostic tool for samplers that lack proofs. The application is practically important because the Lenkoski (2013) sampler is widely used inside exchange algorithms for Gaussian graphical models, and the paper includes a sensible control experiment with an exact sampler for decomposable graphs. The exploratory analysis is honest about the need to use fresh data after choosing the test statistic, and the two-author independent implementation effort is a strength. The main weakness is that the empirical rejection falsifies the conjunction of algorithm and implementation, so the strong wording of the central claim in the abstract and Section 6 exceeds what the evidence supports.
major comments (3)
- [Abstract and Section 6] The conclusion that the Lenkoski (2013) algorithm 'is in fact not sampling from the specified distribution' is stronger than the test can support. The null hypothesis in (12) is explicitly about 'the proposed algorithm and the associated implementation', so a rejection falsifies the conjunction of algorithm and implementation, not the algorithm alone. Section 6 itself acknowledges this ambiguity. The two independent implementations reduce, but do not eliminate, the risk of a shared misreading of the algorithm, particularly since no code is released. The abstract and closing remarks should either be rephrased as findings about the algorithm as implemented and tested, or the authors should add a proof that the fixed-point map in (27) cannot be an exact G-Wishart sampler. As written, the central claim overstates the evidence.
- [Section 4.4] The theoretical critique of Lenkoski (2013) is not developed enough to support the paper's strong conclusion. The assertion that the family of positive definite matrices fails the positivity condition for the Hammersley--Clifford theorem is stated in a single sentence without a formal argument or citation, and even if correct it does not by itself prove that the output of the fixed-point iteration has the wrong joint distribution. Section 4.4 therefore only motivates the empirical test; the central claim rests entirely on the empirical rejection discussed above.
- [Sections 5.1 and 5.3] No code or data are provided, so the two independent implementations cannot be audited. Because the main conclusion is an empirical falsification of a published algorithm, the reproducibility burden is high. The algorithmic description is also underspecified in places: the stopping rule 'two consecutive iterations agree to machine precision' with a 10,000-iteration cap is not a rigorous convergence criterion for a nonlinear fixed-point iteration, and the handling of the reported oscillation cases is vague. Small differences in these implementation choices could plausibly change the distribution of the generated matrices. The authors should release code and provide a precise algorithmic specification, including the fixed-point solver and the treatment of oscillating cases.
minor comments (5)
- [Introduction] Page 2 contains a duplicated word: 'generated samples are from from some distribution' should read 'generated samples are from some distribution'.
- [Section 5.3] The text 'q = 999 9999' is a typo; the reported minimum p-value of 10^-6 corresponds to q = 999,999 or q = 9,999,999, and the spacing should be corrected.
- [Figure 4] In the right column, the axis labels for the first two rows read 'lnptrpQqq' even though the text says these panels show ln|Q|; the labels should be corrected to match the description.
- [References] The reference list contains a few spelling errors: 'Mariott' should be 'Marriott', 'Clarenden Press' should be 'Clarendon Press', and 'Candés' should be 'Candès'.
- [Section 4.4] The claim that the positive-definite support violates the positivity condition would benefit from a precise statement or reference, since it is central to the criticism of Lenkoski's argument but is currently only asserted.
Circularity Check
No significant circularity: the hypothesis-test construction is self-contained, the test statistic is evaluated on fresh simulated data, and the central claim rests on an empirical rejection plus an explicitly acknowledged implementation caveat rather than on any fitted input or self-citation.
full rationale
The paper's central claim is that the Lenkoski (2013) direct sampler is not producing samples from the specified G-Wishart distribution. The claim is supported by the MCMC hypothesis test of Section 3, whose validity is established directly: under H0, initializing a detailed-balance Metropolis-Hastings kernel with a claimed exact sample produces exchangeable initial and final states (Proposition 1 and the discussion around equations (5)-(11)), so the permutation-based p-value in (16)-(17) is valid by construction. This is a genuine derivation rather than an assumption imported from the authors' prior work. The test statistic h(Q)=ln|Q| and H(T) in (28)-(29) are chosen after exploratory analysis in Section 5.2, but the paper explicitly re-simulates fresh data for the formal tests: 'As we choose the h(.) and H(.) functions based on our observations in Section 5.2, we of course can not use the simulated values we explored in that section in our hypothesis tests. So we simulate new Q_i^(0) and Q_i^(r) values for the hypotheses tests'. Thus the rejection is not forced by construction or by fitting the test statistic to the tested sample. The theoretical critique in Section 4.4 identifies genuine gaps in Lenkoski's argument (the independence claim and the positivity condition for the Hammersley-Clifford theorem), but it does not by itself assert the conclusion; it motivates the empirical test. There are no fitted parameters renamed as predictions, no uniqueness theorem imported from the authors, and no self-citation chain carrying the argument. The main limitation, acknowledged in Section 6, is that rejection of H0 as stated in (12) falsifies the conjunction of the algorithm and its implementation, so a shared bug in the two independent implementations could in principle explain the small p-values. This is an honest caveat about empirical identifiability, not a circular step: the test would also reject a genuinely biased algorithm, and the conclusion is not equivalent to the test's input by definition. Because the derivation chain is self-contained against an independent exact sampler for decomposable graphs and the p-value validity argument is derived from first principles, the circularity score is 0.
Assumptions & free parameters
assumptions (5)
- standard math Detailed balance implies exchangeability of X0 and Xr for a reversible chain started at stationarity.
- domain assumption For any maximal clique C of a G-Wishart graph, the Schur complement Q*_C,C is Wishart and independent of the remaining entries.
- domain assumption The recursive sampler in Section 4.3 exactly generates G-Wishart draws for decomposable graphs.
- standard math The Hammersley-Clifford theorem cannot justify Lenkoski's sampler because the positivity condition fails on the positive definite cone.
- domain assumption The fixed-point iteration for Lenkoski's equation system converges to the intended solution for the tested graphs.
Cite this review
Pith. "Pith review of An MCMC hypothesis test to check a claimed sampler: applied to a claimed sampler for the G-Wishart distribution." pith.science (2026). https://pith.science/paper/NFA4FSJH
@misc{pith2026250524400,
author = {Pith},
title = {Pith review of: An MCMC hypothesis test to check a claimed sampler: applied to a claimed sampler for the G-Wishart distribution},
year = {2026},
howpublished = {\url{https://pith.science/paper/NFA4FSJH}},
note = {Machine review of arXiv:2505.24400}
}
abstract
Suppose we have a distribution of interest, with density $p(x),x\in {\cal X}$ say, and an algorithm claimed to generate samples from $p(x)$. Moreover, assume we have available a Metropolis--Hastings transition kernel fulfilling detail balance with respect to $p(x)$. In such a situation we formulate a hypothesis test where $H_0$ is that the claimed sampler really generates correct samples from $p(x)$. We use that if initialising the Metropolis--Hastings algorithm with a sample generated by the claimed sampler and run the chain for a fixed number of updates, the initial and final states are exchangeable if $H_0$ is true. Combining this idea with the permutation strategy we define a natural test statistic and a valid p-value. Our motivation for considering the hypothesis test situation is a proposed sampler in the literature, claimed to generate samples from G-Wishart distribution. As no proper proof for the validity of this sampler seems to be available, we are exactly in the hypothesis test situation discussed above. We therefore apply the defined hypothesis test to the claimed sampler. For comparison we also apply the hypothesis test to a known exact sampler for a subset of G-Wishart distributions. The obtained p-values clearly show that the sampler claimed to be able to generate samples from any G-Wishart distribution is in fact not sampling from the specified distribution. In contrast, and as one should expect, the p-values obtained when using the known exact algorithm does not indicate any problems.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Atay-Kayis, A. and Massam, H. (2005). A Monte Carlo method for comput- ing the marginal likelihood in nondecomposable Gaussian graphical models, Biometrika 92: 317–335
work page 2005
-
[2]
Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems, Journal of the Royal Statistical Society, Series B36: 192–225
work page 1974
-
[3]
Besag, J. and Clifford, P. (1989). Generalized Monte Carlo significance tests, Biometrika 76: 633–642
work page 1989
-
[4]
Brook, D. (1964). On the distinction between the conditional probability and the joint probability approaches in the specification of nearest-neighbour systems, Biometrika 51: 481–483
work page 1964
-
[5]
Colombi, A., Argiento, R., Paci, L. and Pini, A. (2023). Learning block structured graphs in Gaussian graphical models,Journal of Computational and Graphical Statistics 33: 152–165
work page 2023
-
[6]
Cowell, R. G., Dawid, A. P., Lauritzen, S. L. and Spiegelhalter, D. J. (2007). Probabilistic Networks and Expert Systems, Exact Computational Methods for Bayesian Networks, Springer, London
work page 2007
-
[7]
Ernst, M. (2004). Permutation methods: A basis for exact inference,Statistical Science 19: 676–685. 23
work page 2004
-
[8]
Efficientsampling of Gaussian graphical models using conditional Bayes factors,Stat 3: 326–336
Hinne, M., Lenkoski, A., Heskes, T.andvanGerven, M.(2014). Efficientsampling of Gaussian graphical models using conditional Bayes factors,Stat 3: 326–336
work page 2014
Show all 19 references
- [9]
-
[10]
and West, M
Jones, B., Carvalho, C., Dobra, A., Hans, C., Carter, C. and West, M. (2005). Experiments in stochastic computation for high-dimensional graphical models, Statistical Science20: 388–400
2005
-
[11]
(1996).Graphical Models, Clarenden Press, Oxford
Lauritzen, S. (1996).Graphical Models, Clarenden Press, Oxford
1996
-
[12]
Lenkoski, A. (2013). A direct sampler for G-Wishart variates,Stat 2: 119–128
2013
-
[13]
Mariott, F. H. C. (1979). Monte Carlo tests: how many simulations?,Applied Statistics 28: 75–77
1979
-
[14]
Meyn, S. P. and Tweedie, R. L. (2009).Markov chains and stochastic stability, 2nd edn, Cambridge University Press. Møller, J., Pettitt, A., Reeves, R. and Berthelsen, K. (2006). An efficient Markov chain Monte Carlo method for distributions with intractable normalising con- st...
2009
-
[15]
(2007).Advances in Markov chain Monte Carlo methods, PhD thesis, Gatsby Computational Neuroscience Unit, University College London
Murray, I. (2007).Advances in Markov chain Monte Carlo methods, PhD thesis, Gatsby Computational Neuroscience Unit, University College London
2007
-
[16]
and MacKay, D
Murray, I., Ghahramani, Z. and MacKay, D. (2006). MCMC for doubly- intractable distributions,Proceedings of the Twenty-Second Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-06), AUAI Press, Ar- lington, Virginia, pp. 359–366
2006
-
[17]
F., Candés, E
Ramdas, A., Barber, R. F., Candés, E. J. and Tibshirani, R. J. (2023). Permu- tation tests using arbitrary permutation distributions,Sankhya A: The Indian Journal of Statistics85: 1156–1177
2023
-
[18]
Roverato, A. (2002). Hyper inverse Wishart distribution for non-decomposable graphs and its application to Bayesian inference for Gaussian graphical models, Scandinavian Journal of Statistics29: 391–411
2002
-
[19]
Tierney, L. (1998). A note on Metropolis–Hastings kernels for general state spaces, The Annals of Applied Probability8: 1–9. 24 van den Boom, W., Beskos, A. and de Iorio, M. (2022). The G-Wishart weighted proposal algorithm: Efficient posterior computation for Gaussian graphic...
1998
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.