Pith. sign in

REVIEW 3 major objections 4 minor 28 references

PolyStan: PolyChord nested sampling and Bayesian evidences for Stan models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PolyStan gives Stan users black-box nested sampling for Bayes factors and hard posteriors.

desk verdict A genuinely useful Stan–PolyChord integration whose central evidence-computation claim is undercut by a false statement about `_lupdf` normalization that must be fixed before publication. read the letter →

arxiv 2505.17620 v1 pith:EE2AMVBW submitted 2025-05-23 stat.CO physics.data-an

Andrew Fowlie This is my paper
classification stat.COphysics.data-an MSC 62F1565C05
keywords nestedsamplingmarginallikelihoodBayesfactorStanPolyChordBridgemodelselectionmultimodalposterior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PolyStan is a new inference engine that connects Stan models to the PolyChord nested sampling algorithm through BridgeStan, giving Stan users a black-box way to compute Bayesian evidences and sample from multimodal or degenerate posteriors. The paper's central claim is that this interface works as a cmdstan-style command-line tool: a user writes a Stan model with parameters on the unit hypercube, transforms them to physical parameters, and leaves only likelihood contributions in the model block, and PolyStan returns an evidence estimate plus posterior samples. The motivation is that marginal likelihoods, needed for Bayes factors and model selection, are hard to compute, and nested sampling handles problems where Hamiltonian Monte Carlo fails. If the claim holds, Stan users gain a practical route to evidence-based model comparison that was previously out of reach.

What carries the argument

The evidence integral and its nested-sampling reduction: $Z = \int L(\theta)\pi(\theta)\,d\theta = \int_0^1 L(X)\,dX$, where $X(L_\star)$ is the prior volume with $L \ge L_\star$. PolyChord implements this by maintaining $n_\mathrm{live}$ points, evicting the minimum-likelihood point, and using slice sampling along random directions to replace it from the constrained prior, with whitening and clustering for degenerate and multimodal targets. PolyStan's specific contribution is the bridge: parameters are declared on the unit hypercube, transformed to physical parameters via inverse CDF transforms, and the model block is restricted to likelihood contributions, so the prior lives in the transform and the evidence is computed with a normalized likelihood.

What would settle it

For a model whose evidence is known analytically, such as a univariate normal with known prior, compare PolyStan's reported $\log Z$ and uncertainty to the exact value; a deviation beyond the reported uncertainty would show the interface is not computing the claimed evidence.

Watch

Extended reading notes

Core claim

PolyStan couples Stan to PolyChord via BridgeStan, treating the target function as log-likelihood and the unit hypercube as the prior space. Nested sampling rewrites the evidence integral $Z = \int L(\theta)\pi(\theta)\,d\theta$ as a one-dimensional integral $\int_0^1 L(X)\,dX$ over the prior mass $X$, and PolyChord evolves a set of live points upward in likelihood by replacing the worst point with a slice-sampled draw from the constrained prior. The paper demonstrates on examples such as the eggbox, Rastrigin, Rosenbrock, shell, and slab-and-spike problems that PolyStan yields evidence estimates consistent with bridge sampling when HMC works, and that it produces sensible evidences where HMC and bridge sampling fail.

Load-bearing premise

The user must put all prior information into the inverse transforms from the unit hypercube and keep prior statements out of the model block; PolyStan cannot enforce this because it cannot read the user's intent, and an error silently changes the evidence by a constant factor.

Editorial extensions

If this is right

  • Stan users can compute Bayes factors for model comparison directly from the command line, without writing custom nested-sampling code.
  • Multimodal and degenerate targets that produce divergent transitions in HMC become amenable to black-box sampling through a familiar cmdstan-like workflow.
  • The insertion-index p-value provides a built-in diagnostic for whether the nested-sampling run faithfully sampled from the constrained prior.
  • Because likelihood normalization is preserved, evidence estimates from PolyStan are on a consistent scale across models, unlike default Stan which drops constant terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to generate the inverse-transform priors automatically from Stan's parameter declarations, which would remove the main user error source.
  • The same BridgeStan coupling could drive other nested-sampling implementations, such as those with dynamic live points or plateau-aware evidence resummation, giving Stan users a broader algorithm zoo.
  • The plateau issue noted in the GLMM example suggests a diagnostic extension: automatically detect zero-likelihood plateaus and warn the user, rather than reporting a p-value of zero and requiring external tools like anesthetic.
  • PolyStan's CLI could be wrapped by higher-level Bayesian model-comparison workflows to automate pairwise Bayes factors across a set of candidate models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces PolyStan, a C++/command-line interface that couples Stan models to the PolyChord nested sampling algorithm via BridgeStan. The user writes a Stan model with unit-hypercube parameters and inverse-transform priors, and PolyStan returns evidence estimates, posterior samples, diagnostics such as insertion-index p-values, and JSON/ArviZ-compatible output. The paper demonstrates the package on toy and benchmark examples, comparing evidence estimates with bridgesampling applied to HMC chains.

Significance. If the software works as described, it would fill a real gap: Stan currently lacks a built-in nested sampling engine, so users who need evidences for multimodal or degenerate posteriors must export to other tools or write custom code. PolyStan leverages two mature external packages, PolyChord and BridgeStan, so the main novel component is the integration layer rather than the sampling algorithm. The paper provides reproducible examples, openly documents limitations, and uses independent external components for the core algorithm, which is a strength. However, the central evidence-calculation claim depends on correct normalization of the Stan target and on benchmark validation; both need attention before the paper can be accepted.

major comments (3)
  1. [Section 5.2, Eq. (3)] The claim that `lupdf`/`lupmf` functions "increment the target function by a normalized contribution" and that specifying `propto=false` via BridgeStan retains normalization constants is incorrect. In the Stan language, `_lupdf` and `_lupmf` suffixes denote log-densities with normalizing constants removed by construction, independently of the `propto` template parameter. For example, `std_normal_lupdf(x)` omits the constant `-0.5*log(2*pi)`, and for distributions such as gamma or inverse-gamma the omitted constant depends on the parameters, so it changes both the posterior and the evidence integral in Eq. (3). Because PolyStan reads the target via BridgeStan as-is, a model that follows the paper's own Section 5.2 example will produce a biased log-evidence with no diagnostic warning. This is not a user-intent ambiguity but a documented instruction that is wrong, and it directly affects the central promise of the package. The documentation and any examples using `_lupdf`/`_lupmf` must be corrected, and a test should verify that distribution statements and `_lpdf`/`_lpmf` forms yield the same evidence while `_lupdf` forms do not.
  2. [Section 6, Eqs. (18)-(25), (29)-(34)] The benchmark examples do not provide reference evidence values for the problems where HMC fails. For eggbox, Rastrigin, Himmelblau, shell, and slab & spike, the paper reports PolyStan and bridgesampling estimates, but when bridgesampling is deemed faulty there is no independent ground truth against which to judge PolyStan's log Z. Self-consistency with PolyChord and the insertion-index diagnostic is not a verification of the evidence integral. The authors should add exact or highly accurate reference values, for example by deterministic quadrature in low dimensions or by an independent nested-sampling implementation, for at least the eggbox, Himmelblau, and Rastrigin cases, or otherwise restrict the claims to illustrative rather than validated.
  3. [Section 6, GLMM paragraph, Eq. (42)] The GLMM example reports an insertion-index p-value of zero, which the paper attributes to a likelihood plateau, and notes that plateaus "may spoil evidence estimates." Despite this, the paper presents the PolyStan and bridgesampling Bayes factors as agreement (B01=1.3±0.3 vs 1.271±0.006). A failed insertion-index diagnostic means the run did not faithfully sample from the constrained prior, so the agreement is not a reliable validation of evidence accuracy. The example should either be re-analyzed with plateau-aware resummation, as the paper itself suggests with anesthetic, or removed/reclassified as a known-failure illustration.
minor comments (4)
  1. [Section 2.2] The final sentence after the sample output says "We see the estimate logZ = −6.20258 ± 0.0408675", but the screen output printed immediately above shows log(Z) = -6.21093 ± 0.04053; the JSON excerpt later shows yet another value. This inconsistency should be corrected.
  2. [Section 4.2] There is a typo: "posteror" should be "posterior".
  3. [Section 1] The introduction contains a typo "annd" in "discrete parameters annd multi-modal", and the text near Eq. (2) says "relative plausibly" instead of "relative plausibility".
  4. [Section 5.1] The abstract and introduction describe PolyStan as a "black-box" method, but the requirement that the model block contain only likelihood contributions and that priors be specified via inverse transforms from the unit hypercube means existing Stan models cannot be run as-is. This is documented in Section 5.1 but should be stated more prominently, since it is central to obtaining correct evidence values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PolyStan's evidence estimates are benchmarked against independent external tools, and its self-citations are parameter-free diagnostics rather than fitted inputs.

full rationale

PolyStan is an interface paper: it couples Stan models to PolyChord via BridgeStan and compares the resulting evidence estimates against independent implementations (bridgesampling with HMC, and anesthetic). The evidence integral in Eq. (3) is implemented by the standard nested-sampling sum in Eq. (6); no fitted parameter is renamed as a prediction. The prior/likelihood split in Section 5.1 is an explicit user obligation, openly stated to be unenforceable, not an input that is later presented as an output. The normalization discussion in Section 5.2 concerns whether constants are retained in the target; whatever its correctness, it is a usage/documentation claim about the Stan language, not a circular step in which a result is equivalent to its own input. The only self-referential elements are the insertion-index diagnostic (Fowlie, Handley, and Su 2020) and the plateau caveat (Fowlie, Handley, and Su 2021); these are cited as parameter-free or descriptive checks rather than as fitted parameters or as the source of the benchmark evidences. Because the benchmark evidence values are produced by the independent PolyChord/BridgeStan stack and compared to external tools, the central demonstration does not reduce to the paper's own assumptions or definitions.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard nested-sampling theory plus several user-dependent modeling assumptions. The algorithm configuration parameters (nlive, num_repeats, precision) are user choices and could affect results. No new physical or mathematical entities are introduced.

free parameters (3)
  • nlive = default 500
    Number of live points in nested sampling; chosen by hand, not fitted to data; controls accuracy vs runtime.
  • num_repeats = default 5 per dimension
    Slice sampling steps in PolyChord; user-tuned; affects decorrelation of new live points.
  • precision_criterion (epsilon) = default 0.001
    Stopping rule for evidence truncation; user-specified; smaller values give more accurate evidence.
assumptions (5)
  • standard math Nested sampling volume estimate X_i = exp(-i/nlive) from geometric compression
    Standard approximation in nested sampling, from Skilling 2004; the paper uses it in eq. (8)-(9).
  • domain assumption PolyChord slice sampling yields independent samples from the constrained prior
    Central to PolyChord's correctness; the insertion-index test is used to check it, but it is not guaranteed.
  • domain assumption Model block contains only likelihood contributions, not priors
    Section 5.1 explicitly requires this and states it cannot be enforced.
  • domain assumption The likelihood function has no plateaus that spoil evidence estimates
    The GLMM example shows a plateau at L=0, causing insertion p-value 0; the paper notes plateaus may spoil evidence (Section 6).
  • domain assumption Target function is evaluated with propto=false so normalization constants are retained
    PolyStan uses BridgeStan to access the target without dropping constants; necessary for correct evidence normalization (Section 5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of PolyStan: PolyChord nested sampling and Bayesian evidences for Stan models." pith.science (2026). https://pith.science/paper/EE2AMVBW

@misc{pith2026250517620,
  author       = {Pith},
  title        = {Pith review of: PolyStan: PolyChord nested sampling and Bayesian evidences for Stan models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EE2AMVBW}},
  note         = {Machine review of arXiv:2505.17620}
}
read the original abstract

Sampling from multi-modal distributions and estimating marginal likelihoods, also known as evidences and normalizing constants, are well-known challenges in statistical computation. They can be overcome by nested sampling, which evolves a set of live points through a sequence of distributions upwards in likelihood. We introduce PolyStan -- a nested sampling inference engine for Stan. PolyStan provides a Stan interface to the PolyChord nested sampling algorithm using bridgestan. PolyStan introduces a new user-base to nested sampling algorithms and provides a black-box method for sampling from challenging distributions and computing marginal likelihoods. We demonstrate the robustness of nested sampling on several degenerate and multi-modal problems, comparing it to bridge sampling and Hamiltonian Monte Carlo.

Figures

Figures reproduced from arXiv: 2505.17620 by the authors.

Figure 1
Figure 1. Posterior pdf for theta in the bernoulli.stan model, found by PolyStan and plotted with ArviZ (Kumar et al. 2019) and matplotlib (Hunter 2007). "metadata": "The evidence is log-normally distributed", "log evidence": -6.202493190765381, "error log evidence": 0.040782634168863299 }, "neval": { "metadata": "Total number of likelihood evaluations", "neval": 86408 } } These show our evidence estimate, effective sample si… view at source ↗
Figure 2
Figure 2. Posterior distribution of a discrete change-point parameter in the coal mining disasters [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 9 canonical work pages

  1. [1]

    Nested sampling for parameter inference in systems biology: application to an exemplar circadian model

    Aitken S, Akman OE (2013). Nested sampling for parameter inference in systems biology: application to an exemplar circadian model. BMC Syst. Biol., 7 (1), 72. doi:10.1186/1752-0509-7-72

    work page doi:10.1186/1752-0509-7-72 2013

  2. [2]

    Ashton G, et al. (2022). Nested sampling for physical scientists . Nature, 2. doi:10.1038/s43586-022-00121-x. 2205.15570

    arXiv 2022

  3. [3]

    A Conceptual Introduction to Hamiltonian Monte Carlo

    Betancourt M (2017). A Conceptual Introduction to Hamiltonian Monte Carlo . 1701.02434

    arXiv 2017

  4. [4]

    Nested sampling methods

    Buchner J (2021). Nested sampling methods . Stat. Surv., 17. doi:10.1214/23-ss144. 2101.09675

    arXiv 2021

  5. [5]

    The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters

    Dickey JM (1971). The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters . Ann. Math. Stat., 42 (1), 204 -- 223. doi:10.1214/aoms/1177693507

    arXiv 1971

  6. [6]

    Nested sampling cross-checks using order statistics

    Fowlie A, Handley WJ, Su L (2020). Nested sampling cross-checks using order statistics . Mon. Not. Roy. Astron. Soc., 497 (4), 5256--5263. doi:10.1093/mnras/staa2345. 2006.03371

    work page Pith review arXiv 2020

  7. [7]

    Nested sampling with plateaus

    Fowlie A, Handley WJ, Su L (2021). Nested sampling with plateaus . Mon. Not. Roy. Astron. Soc., 503 (1), 1199--1205. doi:10.1093/mnras/stab590. 2010.13884

    work page Pith review arXiv 2021

  8. [8]

    bridgesampling : An R Package for Estimating Normalizing Constants

    Gronau QF, Singmann H, Wagenmakers EJ (2020). bridgesampling : An R Package for Estimating Normalizing Constants. J. Stat. Softw., 92 (10). doi:10.18637/jss.v092.i10

    doi:10.18637/jss.v092.i10 2020

Show all 28 references
  1. [9]

    anesthetic : nested sampling visualisation

    Handley WJ (2019). anesthetic : nested sampling visualisation. J. Open Source Softw., 4 (37), 1414. doi:10.21105/joss.01414

    2019 doi

  2. [10]

    PolyChord : nested sampling for cosmology

    Handley WJ, Hobson MP, Lasenby AN (2015 a ). PolyChord : nested sampling for cosmology . Mon. Not. Roy. Astron. Soc., 450 (1), L61--L65. doi:10.1093/mnrasl/slv047. 1502.01856

    2015 arXiv

  3. [11]

    polychord : next-generation nested sampling

    Handley WJ, Hobson MP, Lasenby AN (2015 b ). polychord : next-generation nested sampling . Mon. Not. Roy. Astron. Soc., 453 (4), 4385--4399. doi:10.1093/mnras/stv1911. 1506.00171

    2015 arXiv

  4. [12]

    Matplotlib : A 2D graphics environment

    Hunter JD (2007). Matplotlib : A 2D graphics environment. Computing in Science & Engineering, 9 (3), 90--95. doi:10.1109/MCSE.2007.55

    2007 doi

  5. [13]

    Experimental analysis of an early life-history stage: Selection on size of hatchling turtles

    Janzen FJ, Tucker JK, Paukstis GL (2000). Experimental analysis of an early life-history stage: Selection on size of hatchling turtles. Ecology, 81 (8), 2290–2304. ISSN 0012-9658. doi:10.1890/0012-9658(2000)081[2290:eaoael]2.0.co;2

    2000 doi

  6. [14]

    A note on the intervals between coal-mining disasters

    Jarrett RG (1979). A note on the intervals between coal-mining disasters. Biometrika, 66 (1), 191--193

    1979

  7. [15]

    The Theory of Probability

    Jeffreys H (1939). The Theory of Probability . Oxford University Press. ISBN 978-0-19-850368-2

    1939

  8. [16]

    Bayes Factors

    Kass RE, Raftery AE (1995). Bayes Factors. J. Am. Stat. Assoc., 90 (430), 773--795. doi:10.1080/01621459.1995.10476572

    1995

  9. [17]

    ArviZ a unified library for exploratory analysis of Bayesian models in Python

    Kumar R, Carroll C, Hartikainen A, Martin O (2019). ArviZ a unified library for exploratory analysis of Bayesian models in Python . J. Open Source Softw., 4 (33), 1143. doi:10.21105/joss.01143

    2019 doi

  10. [18]

    The Kolmogorov-Smirnov Test for Goodness of Fit

    Massey FJ (1951). The Kolmogorov-Smirnov Test for Goodness of Fit. J. Am. Stat. Assoc., 46 (253), 68–78. doi:10.1080/01621459.1951.10500769

    1951

  11. [19]

    Simulating ratios of normalizing constants via a simple identity: a theoretical exploration

    Meng XL, Wong WH (1996). Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Stat. Sin., pp. 831--860. ://www.jstor.org/stable/24306045

    1996

  12. [20]

    Slice sampling

    Neal RM (2003). Slice sampling. Ann. Stat., 31 (3). doi:10.1214/aos/1056562461. physics/0009028

    2003

  13. [21]

    Handbook of Markov Chain Monte Carlo

    Neal RM (2011). Handbook of Markov Chain Monte Carlo . Chapman and Hall/CRC. doi:10.1201/b10905. 1206.1901

    2011 arXiv

  14. [22]

    Sampling from high-dimensional, multimodal distributions using automatically tuned, tempered Hamiltonian Monte Carlo

    Park J (2021). Sampling from high-dimensional, multimodal distributions using automatically tuned, tempered Hamiltonian Monte Carlo. 2111.06871

    2021 arXiv

  15. [23]

    BridgeStan : Efficient in-memory access to the methods of a Stan model

    Roualdes EA, Ward B, Carpenter B, Seyboldt A, Axen SD (2023). BridgeStan : Efficient in-memory access to the methods of a Stan model. J. Open Source Softw., 8 (87), 5236. doi:10.21105/joss.05236

    2023 doi

  16. [24]

    CLI11 : Command line parser for C++11

    Schreiner H, et al. CLI11 : Command line parser for C++11 . ://github.com/CLIUtils/CLI11

  17. [25]

    Nested Sampling

    Skilling J (2004). Nested Sampling. In AIP Conference Proceedings, volume 735, p. 395–405. AIP. doi:10.1063/1.1835238

    2004 doi

  18. [26]

    Nested sampling for general Bayesian computation

    Skilling J (2006). Nested sampling for general Bayesian computation . Bayesian Anal., 1 (4), 833 -- 859. doi:10.1214/06-BA127

    2006 doi

  19. [27]

    Stan Modeling Language User's Guide and Reference Manual

    Stan Development Team (2024). Stan Modeling Language User's Guide and Reference Manual . ://mc-stan.org

    2024

  20. [28]

    Rank-Normalization, Folding, and Localization: An Improved R for Assessing Convergence of MCMC

    Vehtari A, Gelman A, Simpson D, Carpenter B, Bürkner PC (2021). Rank-Normalization, Folding, and Localization: An Improved R for Assessing Convergence of MCMC . Bayesian Anal., 16 (2). doi:10.1214/20-ba1221. 1903.08008

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.