REVIEW 4 major objections 4 minor 36 references
Probabilistic Programming with Sufficient Statistics for faster Bayesian Computation
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Precomputing sufficient statistics makes each MCMC likelihood evaluation independent of the number of observations, cutting runtime dramatically in common Bayesian models.
desk verdict A practical, honest demonstration that sufficient statistics make Stan much faster for several standard models; the benchmarks need diagnostics and a fairer mixed-effects comparison, but the core claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sufficient-statistics reparameterisation of an exponential-family likelihood: if $f(y;\theta)=h(y)\exp\{\eta(\theta)^\top T(y)-A(\theta)\}$, then the joint log-likelihood is $\eta(\theta)^\top s(y)-nA(\theta)$ plus a data-only constant, with $s(y)=\sum_i T(y_i)$. Precomputing $s(y)$ in the transformed data block of Stan moves the $O(n)$ sum out of the MCMC loop; for linear regression the summaries are $S_{xx}=X^\top X$, $S_{yx}=y^\top X$, and $S_{yy}=y^\top y$, for mixed effects models they include per-group counts and sums, and for factor models the summary is the sample covariance $S=y^\top y/n$. An optional Woodbury identity computes the precision matrix $\Omega=(\Lambda\Lambda^\top+\mathrm{diag}(\psi))^{-1}$ in $O(d^3)$ rather than $O(p^3)$ operations, which matters when the number of factors $d$ is much smaller than the observation dimension $p$.
What would settle it
Run the same four implementations with convergence diagnostics reported per unit time: compute effective sample size per second and require all chains to pass $\hat{R}<1.01$. If the sufficient-statistics implementations need substantially more draws to reach the same effective sample size, the wall-clock speedups shrink or disappear; this can be checked with the supplied Stan code on the paper's simulated data.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that for any model whose likelihood is an exponential family, the log-likelihood for $n$ iid observations can be written as $\eta(\theta)^\top s(y) - nA(\theta)$, where $s(y)=\sum_i T(y_i)$ is the vector of sufficient statistics. Since $s(y)$ depends only on the data, it can be computed once in Stan's transformed data block. After that, each evaluation of the likelihood costs the same regardless of $n$, because all $n$-dependent work is absorbed into the precomputed summaries. The paper supplies Stan programs that do this for linear regression with Student-$t$ and Cauchy priors, linear mixed effects models, and factor models (using the sample covariance matrix $S=Y^\top Y/n$ and, in one variant, a Woodbury decomposition of the precision matrix), and shows the sampled posteriors match those of brms, rstanarm, and a vectorized implementation while the runtime either stays flat or grows far more slowly with $n$. For Poisson regression, where the term $\sum_i \exp(x_i^\top\beta)$ cannot be precomputed, the paper still reports gains from precomputing $X^\top y$ and vectorizing the remaining sum.
Load-bearing premise
The speedups are measured as wall-clock time for a fixed number of posterior draws, and the comparison assumes all implementations converge and mix at comparable rates per draw; if the sufficient-statistics chains need effectively more iterations to reach the same accuracy—or if the differing prior on $\sigma_u$ in the rstanarm mixed-effects model biases that baseline—the reported gains would change.
Editorial extensions
If this is right
- For Gaussian linear regression with non-conjugate priors, time to produce 5000 post-warm-up draws stays essentially constant as $n$ rises from 100 to 10,000, while brms, rstanarm, and vectorised code all slow down considerably.
- For mixed effects models, precomputed per-group summaries make the sufficient-statistics implementation scale more slowly with $n$ and keep it faster than brms and rstanarm as the number of groups $J$ varies.
- For factor models, replacing the raw data by the sample covariance matrix $Y^\top Y/n$ speeds up sampling, and adding the Woodbury decomposition yields further gains that grow with observation dimension $p$.
- Even when only partial sufficient statistics exist, as in Poisson regression, the paper reports that the implementation using $X^\top y$ plus vectorised $\sum_i \exp(x_i^\top\beta)$ is the fastest of the four compared methods.
- If such implementations were folded into brms and rstanarm, users would get these speedups without changing how they write models.
Reading between the lines
- My inference: the same precomputation applies to any other exponential-family likelihood with non-conjugate or hierarchical priors—logistic, gamma, negative binomial, and multinomial regression—so the demonstrated speedups should generalise beyond the three examples.
- My inference: the one-time cost of forming $X^\top X$ and $X^\top y$ scales like $O(np^2)$ in dense form, so the method's advantage is largest when $n$ is large and $p$ is moderate; for very high $p$, that precomputation, not the MCMC likelihood evaluation, could become the bottleneck.
- My inference: a natural stress test is to compare effective sample size per second, not wall-clock time per number of draws, because the sufficient-statistics reformulation changes the HMC target's gradient and could in principle alter mixing; if mixing per draw is worse, part of the reported speedup would be offset.
- My inference: the partial-sufficient-statistics idea in Poisson regression suggests a general recipe for GLMs where only an additive nonlinear term resists summarisation—precompute what can be summarised and vectorise the rest—which could extend to other nonlinear link functions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes that, when a Stan model's likelihood can be written in terms of precomputed sufficient statistics, each MCMC likelihood evaluation becomes independent of the number of observations n, and that this yields large practical speedups over existing Stan implementations. The authors demonstrate the idea on Gaussian linear regression with non-conjugate priors, linear mixed effects models, factor analysis models, and a Poisson regression model where only partial sufficient statistics are available. They compare their implementations against vectorised Stan code, brms, and rstanarm, measuring wall-clock time for a fixed number of posterior draws. Code for all implementations is provided in the appendix and online.
Significance. If the central claim holds, the paper offers a genuinely useful and easy-to-adopt practical improvement for Stan users: the sufficient-statistic reformulations are simple, the code is provided, and the potential savings grow with n. The strengths of the paper are that the Stan code is complete and reproducible, the comparisons use standard external baselines, and the algebraic likelihoods in the code appear correct. However, the headline conclusion that the method is 'faster Bayesian computation' is currently supported only by wall-clock timings for fixed draw counts, without evidence that the compared chains achieve comparable accuracy per draw. Because of this, the practical significance is conditional on additional convergence and mixing diagnostics.
major comments (4)
- [Sections 3.1-3.4, Figures 1-4] The central empirical claim is measured by wall-clock time to produce a fixed number of posterior draws, not by time to a given estimation accuracy. No effective sample size, R-hat, or Monte Carlo standard error is reported for any timing run, so the reported speedups assume that all implementations mix at comparable rates per draw. This assumption is especially insecure in the mixed-effects example: the vectorised baseline in Listing A.5 uses the centered parameterization z_1 ~ N(0, sd_1), while the sufficient-statistics implementation in Listing A.7 uses the non-centered parameterization r_1_1 = sd_1 * z_1 with z_1 ~ N(0,1). These parameterizations have different HMC geometry and can require substantially different numbers of effective samples. Please report ESS and convergence diagnostics for every implementation and scenario, and ideally compare time to a target ESS or to a target Monte Carlo error rather than time for a fixed draw count.
- [Section 3.1, Listings A.2-A.4] The priors used by the baselines are not identical to the priors stated in the text. Section 3.1 states that σ ~ t3(0, 3.7) for all models, but the brms code in Listing A.3 uses σ ~ t3(0, 3.2), while the vectorised and sufficient-statistics code use σ ~ t3(0, 3.7). This means the brms baseline targets a slightly different posterior, and the posterior overlay in Figure A.1 cannot establish that the four implementations sample from the same distribution for σ. Please either use the same prior in all implementations or explicitly document the discrepancy and assess whether it affects the timing comparisons.
- [Section 3.2, Listings A.5-A.7] Section 3.2 states that the common prior is σ, σ_u ~ t3(0, 3.7), but Listings A.5, A.6, and A.7 all use t3(0, 2.5) for both σ and sd_1. The paper acknowledges that rstanarm uses a different prior on σ_u, but the code-to-text mismatch for the other baselines is unexplained. Since the mixed-effects comparison also changes the parameterization of the random effects between the vectorised and sufficient-statistics implementations, the per-draw speedup cannot be interpreted without first establishing that the targets and the mixing behavior are comparable.
- [Sections 3.2 and 3.3] The displayed sufficient-statistic likelihood equations contain errors that make them difficult to verify against the code. In Section 3.2, the expression contains the term '- n⊤1 / 2 log(2πσ^2)', which appears to be a typographical corruption of a factor involving n (or the vector of group sizes); it should be written as -(sum_j n_j)/2 * log(2πσ^2). In Section 3.3, the factor-model log-likelihood is written as n/2 (-d log(2π) + |Ω| - tr(SΩ)); this should be -(n p/2) log(2π) + (n/2) log |Ω| - (n/2) tr(SΩ), i.e., the dimension is p, not d, and the determinant term is missing the logarithm. The Stan code in Listings A.9 and A.10 contains the correct expression, but the displayed equations need to be corrected so that the mathematical claim matches the implementations.
minor comments (4)
- [Section 2.1] In the displayed expression for the log-likelihood, the sufficient statistic is written as s(θ), but it should be s(y), since it is a function of the data only.
- [Section 3.4, Listing A.13] The Poisson regression data block declares vector[N] Y, although the response is a count variable. Using array[N] int Y would match the data type and avoid an implicit real conversion; the subsequent sufficient-statistic algebra is correct either way.
- [Throughout] There are several typographical issues: the author name 'Bürkner' appears as 'Burkner' or 'B¨ urkner' in the text and references, the Poisson distribution is typeset as 'P oi' in Sections 3.4 and A.4, and the phrase 'rstanarm' is sometimes split across a line break. These are cosmetic but should be fixed.
- [Figure A.3] The caption says the figure overlays posteriors from 'different Stan implementations', but the legend shows only suff-stat, Wood, and vect; please make the caption consistent with the three implementations actually compared for the factor model.
Circularity Check
No significant circularity: the sufficient-statistics likelihood is an exact algebraic rewrite and all speedups are measured against external Stan-based baselines.
full rationale
The paper's central claim is that precomputing sufficient statistics from exponential-family likelihoods makes MCMC likelihood evaluations independent of n, yielding speedups over existing Stan implementations. This is not circular. The sufficient-statistics representation is derived directly from the exponential-family form, ℓ(y;θ) = η(θ)ᵀs(y) − nA(θ) plus a θ-free constant, which is an exact identity rather than a fitted or predicted quantity. The speed comparisons in Figures 1–4 are empirical benchmarks against brms, rstanarm, and vectorised Stan code, all external baselines; no parameter is fitted to a subset of the timing data and then 'predicted' on the rest. The paper also acknowledges, rather than hides, the one implementation mismatch (rstanarm's prior on σu differs from the others in the mixed-effects example), and it verifies posterior agreement with overlay plots. The only substantive concerns—whether fixed-draw-count wall-clock times reflect mixing efficiency, and whether the brms linear-regression prior specification matches the stated t3(0, 3.7)—are correctness and benchmarking-fairness issues, not circularity. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The mathematical content is self-contained and the empirical claims are externally grounded.
Assumptions & free parameters
assumptions (4)
- standard math The exponential family likelihood can be rewritten using sufficient statistics, so terms not depending on parameters can be dropped.
- domain assumption Stan/NUTS with default tuning produces samples from the target posterior for both the original and sufficient-statistics implementations.
- standard math The multivariate normal likelihood with factor covariance can be marginalized and written in terms of S = Y^T Y / n.
- standard math The Woodbury identity applies to compute the precision matrix of the factor model in O(p d^2) time.
Cite this review
Pith. "Pith review of Probabilistic Programming with Sufficient Statistics for faster Bayesian Computation." pith.science (2026). https://pith.science/paper/O7QPMSQ6
@misc{pith2026250204990,
author = {Pith},
title = {Pith review of: Probabilistic Programming with Sufficient Statistics for faster Bayesian Computation},
year = {2026},
howpublished = {\url{https://pith.science/paper/O7QPMSQ6}},
note = {Machine review of arXiv:2502.04990}
}
read the original abstract
Probabilistic programming methods have revolutionised Bayesian inference, making it easier than ever for practitioners to perform Markov-chain-Monte-Carlo sampling from non-conjugate posterior distributions. Here we focus on Stan, arguably the most used probabilistic programming tool for Bayesian inference (Carpenter et al., 2017), and its interface with R via the brms (Burkner, 2017) and rstanarm (Goodrich et al., 2024) packages. Although easy to implement, these tools can become computationally prohibitive when applied to datasets with many observations or models with numerous parameters. While the use of sufficient statistics is well-established in theory, it has been surprisingly overlooked in state-of-the-art Stan software. We show that when the likelihood can be written in terms of sufficient statistics, considerable computational improvements can be made to current implementations. We demonstrate how this approach provides accurate inference at a fraction of the time than state-of-the-art implementations for Gaussian linear regression models with non-conjugate priors, hierarchical random effects models, and factor analysis models. Our results also show that moderate computational gains can be achieved even in models where the likelihood can only be partially written in terms of sufficient statistics.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....
-
[3]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....
-
[4]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...
-
[5]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....
-
[6]
author Abril-Pla, O. , author Andreani, V. , author Carroll, C. , author Dong, L. , author Fonnesbeck, C.J. , author Kochurov, M. , author Kumar, R. , author Lao, J. , author Luhmann, C.C. , author Martin, O.A. , et al., year 2023 . title Pymc: a modern, and comprehensive probabilistic programming framework in python . journal PeerJ Computer Science volum...
work page 2023
-
[7]
author Berger, J.O. , year 1985 . title Statistical decision theory and Bayesian analysis . Springer series in statistics. edition 2nd ed. ed., publisher Springer , address New York, N.Y
work page 1985
-
[8]
author Berger, J.O. , author Berry, D.A. , year 1988 . title Statistical analysis and the illusion of objectivity . journal American scientist volume 76 , pages 159--165
work page 1988
Show all 36 references
-
[9]
, author Carlin, B.P
author Berry, S.M. , author Carlin, B.P. , author Lee, J.J. , author Muller, P. , year 2010 . title Bayesian Adaptive Methods for Clinical Trials . publisher CRC Press
2010
-
[10]
, year 2017
author B \"u rkner, P.C. , year 2017 . title brms: An r package for bayesian multilevel models using stan . journal Journal of statistical software volume 80 , pages 1--28
2017
-
[11]
, author Gelman, A
author Carpenter, B. , author Gelman, A. , author Hoffman, M.D. , author Lee, D. , author Goodrich, B. , author Betancourt, M. , author Brubaker, M.A. , author Guo, J. , author Li, P. , author Riddell, A. , year 2017 . title Stan: A probabilistic programming language . journal...
2017
-
[12]
, year 1935
author Darmois, G. , year 1935 . title Sur les lois de probabilit \'e a estimation exhaustive . journal CR Acad. Sci. Paris volume 260 , pages 85
1935
-
[13]
, author Ylvisaker, D
author Diaconis, P. , author Ylvisaker, D. , year 1979 . title Conjugate priors for exponential families . journal The Annals of statistics , pages 269--281
1979
-
[14]
, author Kennedy, A.D
author Duane, S. , author Kennedy, A.D. , author Pendleton, B.J. , author Roweth, D. , year 1987 . title Hybrid M onte C arlo . journal Physics Letters B volume 195 , pages 216--222
1987
-
[15]
, year 2015
author Farouni, R. , year 2015 . title Fitting a Bayesian Factor Analysis Model in Stan . https://rfarouni.github.io/assets/projects/BayesianFactorAnalysis/BayesianFactorAnalysis.html
2015
-
[16]
, author Smith, A.F
author Gelfand, A.E. , author Smith, A.F. , year 1990 . title Sampling-based approaches to calculating marginal densities . journal Journal of the American Statistical Association volume 85 , pages 398--409
1990
-
[17]
, author Carlin, J.B
author Gelman, A. , author Carlin, J.B. , author Stern, H.S. , author Dunson, D.B. , author Vehtari, A. , author Rubin, D.B. , year 2013 . title Bayesian Data Analysis . publisher CRC Press
2013
-
[18]
, author Hill, J
author Gelman, A. , author Hill, J. , year 2006 . title Data Analysis Using Regression and Multilevel/Hierarchical Models . Analytical Methods for Social Research, publisher Cambridge University Press
2006
-
[19]
, author Jakulin, A
author Gelman, A. , author Jakulin, A. , author Pittau, M.G. , author Su, Y. , year 2008 . title A default prior distribution for logistic and other regression models . journal Annals of Applied Statistics volume 2 , pages 1360--1383
2008
-
[20]
, author Hinton, G.E
author Ghahramani, Z. , author Hinton, G.E. , et al., year 1996 . title The EM algorithm for mixtures of factor analyzers . type Technical Report . Technical Report CRG-TR-96-1, University of Toronto
1996
-
[21]
, year 2006
author Goldstein, M. , year 2006 . title Subjective bayesian analysis: Principles and practice . journal Bayesian Analysis volume 1 , pages 403--420
2006
-
[22]
, author Gabry, J
author Goodrich, B. , author Gabry, J. , author Ali, I. , author Brilleman, S. , year 2024 . title rstanarm: Bayesian applied regression modeling via Stan . https://mc-stan.org/rstanarm/. note r package version 2.32.1
2024
-
[23]
, author Gabry, J
author Guo, J. , author Gabry, J. , author Goodrich, B. , author Weber, S. , year 2020 . title Package ‘rstan’ . journal URL https://cran. r―project. org/web/packages/rstan/(2020)
2020
-
[24]
, year 1970
author Hastings, W.K. , year 1970 . title Monte carlo sampling methods using markov chains and their applications
1970
-
[25]
, author Gelman, A
author Hoffman, M.D. , author Gelman, A. , year 2014 . title The N o- U - T urn sampler: adaptively setting path lengths in H amiltonian M onte C arlo. journal Journal of Machine Learning Research volume 15 , pages 1593--1623
2014
-
[26]
, year 1936
author Koopman, B.O. , year 1936 . title On distributions admitting a sufficient statistic . journal Transactions of the American Mathematical society volume 39 , pages 399--409
1936
-
[27]
, year 2018
author McElreath, R. , year 2018 . title Statistical rethinking: A Bayesian course with examples in R and Stan . publisher Chapman and Hall/CRC
2018
-
[28]
, year 2024
author Mersmann, O. , year 2024 . title microbenchmark: Accurate Timing Functions . https://github.com/joshuaulrich/microbenchmark. note r package version 1.5.0
2024
-
[29]
, author Rosenbluth, A.W
author Metropolis, N. , author Rosenbluth, A.W. , author Rosenbluth, M.N. , author Teller, A.H. , author Teller, E. , year 1953 . title Equation of state calculations by fast computing machines . journal The journal of chemical physics volume 21 , pages 1087--1092
1953
-
[30]
, et al., year 2011
author Neal, R.M. , et al., year 2011 . title MCMC using H amiltonian dynamics . journal Handbook of Markov chain Monte Carlo volume 2 , pages 2
2011
-
[31]
, author Mahoney, D.W
author Oberg, A.L. , author Mahoney, D.W. , year 2007 . title Linear mixed effects models . journal Topics in biostatistics , pages 213--234
2007
-
[32]
, author Pradhan, N
author Phan, D. , author Pradhan, N. , author Jankowiak, M. , year 2019 . title Composable effects for flexible and accelerated probabilistic programming in numpyro . journal arXiv volume 1912.11554 , pages 1--10
2019 arXiv
-
[33]
, year 1936
author Pitman, E.J.G. , year 1936 . title Sufficient statistics and intrinsic accuracy , in: booktitle Mathematical Proceedings of the cambridge Philosophical society , organization Cambridge University Press . pp. pages 567--579
1936
-
[34]
title Stan functions reference
author Stan Development Team , year 2024 . title Stan functions reference . https://mc-stan.org/docs/2\_23/functions-reference/index.html
2024
-
[35]
, author Turek, D
author de Valpine, P. , author Turek, D. , author Paciorek, C.J. , author Anderson-Bergman, C. , author Lang, D.T. , author Bodik, R. , year 2017 . title Programming with models: writing statistical algorithms for general model structures with nimble . journal Journal of Compu...
2017
-
[36]
, author Roberts, G
author Zanella, G. , author Roberts, G. , year 2021 . title Multilevel linear models, gibbs samplers and multigrid decompositions (with discussion) . journal Bayesian Analysis volume 16 , pages 1309--1391
2021
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.