REVIEW 4 major objections 5 minor 31 references
Posterior Mean Matching: Generative Modeling through Online Bayesian Inference
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Posterior means, iterated, become a generative sampler.
desk verdict Genuinely new method class with competitive results, but the consistency theorem is about the ideal process, not the learned sampler; fix the theorem conditions and be honest about the approximation gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the online posterior mean $\mu_t$ of a conjugate Bayesian model, updated by the closed-form rule $\mu_{t+1} = f_t(\mu_t, y_{t+1})$ as noisy observations arrive. Its consistency — $\mu_t \to x^\ast$ almost surely under the augmented target — is what turns Bayesian inference into generation: $\mu_t$ is effectively a sample from $p^\ast(x)$. The approximating distribution $q_\varphi$ replaces the unknown next observation with a draw from $\pi_\alpha(\cdot \mid g_\varphi(\mu_{t-1},t))$, where $g_\varphi$ is a neural network, so the KL objective collapses to matching $g_\varphi$ against the true $x$; for the Normal-Normal model this yields the weighted squared error $\alpha_s\|x - g_\varphi(\mu_{s-1},s)\|_2^2$.
What would settle it
Train the Normal-Normal PMM on a low-dimensional target with known density (for example a mixture of Gaussians), push the network to convergence, and run Algorithm 1 for a very large number of steps; if the empirical distribution of the output $\mu_t$ does not approach the target as the number of steps grows, then the consistency guarantee does not survive the neural approximation and the central reduction fails. A cleaner version compares PMM's samples against direct sampling from the exact posterior-mean joint distribution on the same target.
Extended reading notes
Core claim
The central claim is that, in an augmented Bayesian model with a conjugate prior $\pi(x)$ and a noisy observation model $\pi_{\alpha_s}(y|x)$, the posterior mean $\mu_t = \mathbb{E}_\pi[x \mid y_{1:t}]$ converges almost surely to the hidden true observation $x^\ast \sim p^\ast(x)$ as $t\to\infty$; Theorem 1 proves this for the Normal-Normal model, with analogues for the other pairs. It follows that sampling from the target $p^\ast(x)$ reduces to sampling from the joint distribution $p(\mu_1,\ldots,\mu_t)$ of posterior-mean trajectories and reporting $\mu_t$. Since that joint distribution is intractable, PMM approximates it with a parametric transition $q_\varphi$ whose one step draws a noisy observation around $g_\varphi(\mu_{t-1},t)$ and applies the exact online update; the network is trained by minimizing the KL divergence in Equation (9). The paper works out closed-form objectives for the Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical cases and validates the approach on image and text benchmarks.
Load-bearing premise
The learned network must be close enough to the exact Bayesian update that the consistency guarantee carries over; the paper gives no bound on how close is close enough.
Editorial extensions
If this is right
- Any conjugate pair whose posterior mean is consistent gives a working PMM, so the paper supplies a single template for real-valued, count, and discrete targets; the demonstrated pairs are Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical.
- The Normal-Normal PMM converges in the continuum limit to a stochastic differential equation of the form $d\mu(t) = \frac{f(t)(x-\mu(t))}{b+\int_0^t f(\tau)d\tau}dt + \sqrt{\frac{f(t)}{b+\int_0^t f(\tau)d\tau}}dW_t$, so off-the-shelf SDE solvers apply and the sampling cost can be traded against quality.
- The Gamma-Poisson PMM converges to an SDE driven by a Cox process with random base measure $x\,dt$, $x\sim p^\ast(x)$, a departure from Brownian-motion generative models and the first such limit for a Poisson-likelihood generative model.
- The reported results put PMM near diffusion models on images (FID 2.18 on CIFAR-10 for the Normal-Normal model) and ahead of non-autoregressive diffusion baselines on text8 (1.29 BPC), narrowing the gap to autoregressive language models.
Reading between the lines
- The consistency theorems govern the ideal posterior mean, not the fitted network; a practical test would isolate how much sample quality degrades as network capacity or training length shrinks on a target with known density.
- The same outer algorithm could be instantiated with other conjugate pairs in the exponential family, for example the InverseGamma-Gamma pair sketched in the appendix, to target positive continuous data or variance-like quantities without modifying the training loop.
- Because the PMM objective is a reweighted denoising loss, established diffusion-model machinery such as learned noise schedules, guidance, and distillation could plausibly be transferred to PMM, though the paper does not explore this.
- The Cox-process SDE for the Gamma-Poisson model suggests that PMM may be a natural fit for sparse integer data and for algorithms that natively produce counts without an explicit discretization or rounding step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Posterior Mean Matching (PMM), a generative modeling framework that uses conjugate Bayesian models to define an online denoising process. A data point x is corrupted by a sequence of noisy observations y_1:t; the posterior mean sequence mu_1:t of a conjugate Bayesian model is consistent, mu_t -> x, so the authors argue that sampling from the target distribution reduces to sampling from the joint distribution of posterior means. They approximate this joint distribution with a neural-network-parameterized transition q_phi, giving training objectives for Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical models. They further derive continuous-time limits connecting Normal-Normal PMM to an SDE and Gamma-Poisson PMM to a Cox-process-driven jump SDE. The paper reports competitive image generation results (CIFAR-10 FID 2.18) and language modeling results (text8 BPC 1.29, OpenWebText generative perplexity 42.58).
Significance. If the theoretical framing were fully established, PMM would be an appealing unification of Bayesian inference and generative modeling: it provides closed-form objectives for multiple data modalities, an explicit connection to diffusion SDEs, and a conceptually simple sampling algorithm. The paper is strong in its derivational detail: the PMM objectives are derived step by step, the hyperparameter schedules are specified, and the experiments cover several architectures and datasets. The empirical results are competitive with current diffusion-based models on images and with non-autoregressive diffusion language models on text. However, the central claim that Algorithm 1 samples from the target distribution is not proven, and several consistency theorems are stated more strongly than their proofs support. These issues are load-bearing because they concern the correctness guarantee that distinguishes PMM from a purely heuristic denoising scheme.
major comments (4)
- [§2, Eqs. (9)-(11), Algorithm 1] The central correctness claim that 'sampling from the target p*(x) reduces to sampling mu_t from p(mu_1:t)' is not established for the learned process. The ideal transitions have conditional variance b_s^2 alpha_s^{-1} given x, but the marginal transition p(mu_s | mu_{s-1}) has additional variance b_s^2 Var[x | mu_{s-1}] when x is integrated out. The variational transition q_phi in Eqs. (10)-(11) has variance b_s^2 alpha_s^{-1}, so even with the optimal g_phi(mu) = E[x | mu], the KL divergence between the true transition and q_phi is positive for any non-degenerate posterior. The PMM objective (20) matches only conditional means and does not control the terminal marginal q_phi(mu_T). Theorem 1 concerns the ideal Bayesian process, not the Markov chain in Algorithm 1; without a bound on the propagation of this variational error, the paper's theoretical guarantee does not cover the sampling procedure.
- [§3.1, Theorem 1 and Appendix A.2.2, Theorem A.1] The statement of Theorem 1 assumes only that alpha_t is positive, increasing, and lim_{t->infinity} alpha_t = infinity, but the proof relies on the stronger condition sum_{s=1}^t alpha_s = Omega(t^{1+eta}). For example, alpha_t = log t satisfies the stated assumption but not the proof's condition, since sum_{s=1}^t log s ~ t log t = o(t^{1+eta}) for every eta > 0. The theorem as stated is therefore unproven; the stronger summability condition should appear in the main statement, or a genuine proof under only lim alpha_t = infinity is needed.
- [§3.2, Appendix A.3.2 (Theorem A.3)] The Dirichlet-Categorical consistency proof establishes only convergence in probability: P(mu_t != x) -> 0 by the union bound. Section 2 defines consistency as almost sure convergence and states that all PMM models considered are consistent. The claim for this model is stronger than what is proven. Either prove almost sure convergence (e.g., by Borel-Cantelli with a summable bound on the per-time failure probability) or state the weaker mode of convergence.
- [§4, Theorem 3 and Appendix A.4.4] The Gamma-Poisson SDE limit is presented as a theorem, but the proof is a sketch. It asserts convergence of the discrete updates to a Merton jump process without a convergence argument, replaces the partial sums of Poisson variables by a non-homogeneous Poisson process N(t), and applies the Poisson Ito formula to N(t) while N(t) is actually a Cox process with random base measure x dt. The proof does not justify why the random nature of the base measure is compatible with the deterministic view of N(t) used in the Ito calculation. At minimum, the hypotheses under which the discrete process converges in distribution to the SDE, and the sense of convergence, should be stated.
minor comments (5)
- [§3.3, Eq. (32)] The factor alpha_s multiplying y_s in the stated Gamma-Poisson update appears inconsistent with the posterior mean derivation in Appendix A.4.1, Eq. (128), where the coefficient of y_s is 1. Please correct or clarify.
- [Appendix A.4.2, Theorem A.4] The statement 'lim_{t->infinity} sum_{s=1}^t alpha_s = O(t^{1+eta}) -> infinity' mixes asymptotic notations incorrectly; the proof requires a lower bound sum_{s=1}^t alpha_s = Omega(t^{1+eta}), not an upper bound.
- [§2, paragraph before Eq. (9)] The phrase 'variational inference variational inference' is duplicated; remove the repetition.
- [§5.1, Table 1] The table reports FID scores with various superscript symbols but the caption does not define these symbols; make the caption self-contained so that the reader can tell which rows use higher-order solvers or data augmentation.
- [Appendix A.2.3, Eqs. (76)-(77)] The reweighted loss is a modification of the derived PMM objective, and the main text's Section 5 does not clearly state that the reported experiments use the reweighted version; this should be disclosed in the main text.
Circularity Check
No circularity: the PMM derivation is self-contained; the posterior-mean limit theorems are proved in the appendix and the training objective is a genuine variational divergence.
full rationale
The paper's derivation chain is self-contained and does not reduce to its own inputs. Theorem 1 and its appendix version prove consistency of the ideal posterior-mean sequence directly from the stated conjugate model and the growth condition on alpha_t; the conclusion mu_t -> x* a.s. is a limit theorem about the Bayesian update, not a fitted quantity, so it is not a 'prediction' of the trained system. The PMM objective (9) is an explicit KL divergence between the true posterior-mean process and the variational process q_phi, and Equations (20), (31), and (73)-(77) follow by algebra from the conjugate update rules and the reparametrization trick; the reweighted loss is disclosed as an implementation choice corresponding to a learning-rate adjustment rather than a hidden fit. Theorems 2 and 3 are continuum limits of the stated discrete update rules, derived in the appendix, so they are mathematical consequences rather than circular predictions. The only self-citations (Salazar 2024; Beltran-Velez et al. 2024) appear as examples of tabular-data applications and are not load-bearing for any central claim. A skeptical reader is right that Theorem 1 does not by itself establish that the learned chain of Algorithm 1 has terminal marginal equal to p*: the variational transition (10)-(11) omits posterior variance, so the guarantee does not fully transfer to the neural sampler. That is a correctness or approximation gap, not a circularity, because the theorem is neither defined in terms of the algorithm's output nor fitted to it. The empirical evaluation is also measured against external baselines and standard FID, BPC, and perplexity benchmarks, so no self-referential validation scheme is present.
Assumptions & free parameters
free parameters (5)
- Normal-Normal prior precision beta =
2
- Gamma-Poisson prior parameters =
alpha=0.1, gamma=2 (Table 5)
- Noise schedule f(t) for images =
f(t)=13/250 * e^(13t)
- Unmasking schedule f_c(t) for text =
staircase or time-dependent variants in Table 5
- Number of posterior updates (NFE) =
3000-5000 for images, varied for text
assumptions (5)
- domain assumption The dataset consists of i.i.d. samples from the target distribution p*(x).
- domain assumption The augmented Bayesian model has a closed-form posterior mean and is amenable to online updates.
- ad hoc to paper The posterior mean is consistent, i.e., mu_t -> x* almost surely, under the chosen noise schedule.
- domain assumption The neural network g_phi can represent and learn the conditional expectation E[x | mu_{t-1}, t].
- standard math The Poisson sum sum_s Pois(x alpha_s) converges to a non-homogeneous Poisson process in the continuum limit.
invented entities (2)
-
Cox process with random base measure x dt
-
Augmented target distribution p(x, y_1:t)
Cite this review
Pith. "Pith review of Posterior Mean Matching: Generative Modeling through Online Bayesian Inference." pith.science (2026). https://pith.science/paper/4MZY2MXB
@misc{pith2026241213286,
author = {Pith},
title = {Pith review of: Posterior Mean Matching: Generative Modeling through Online Bayesian Inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/4MZY2MXB}},
note = {Machine review of arXiv:2412.13286}
}
read the original abstract
This paper introduces posterior mean matching (PMM), a new method for generative modeling that is grounded in Bayesian inference. PMM uses conjugate pairs of distributions to model complex data of various modalities like images and text, offering a flexible alternative to existing methods like diffusion models. PMM models iteratively refine noisy approximations of the target distribution using updates from online Bayesian inference. PMM is flexible because its mechanics are based on general Bayesian models. We demonstrate this flexibility by developing specialized examples: a generative PMM model of real-valued data using the Normal-Normal model, a generative PMM model of count data using a Gamma-Poisson model, and a generative PMM model of discrete data using a Dirichlet-Categorical model. For the Normal-Normal PMM model, we establish a direct connection to diffusion models by showing that its continuous-time formulation converges to a stochastic differential equation (SDE). Additionally, for the Gamma-Poisson PMM, we derive a novel SDE driven by a Cox process, which is a significant departure from traditional Brownian motion-based generative models. PMMs achieve performance that is competitive with generative models for language modeling and image generation.
Figures
Reference graph
Works this paper leans on
-
[1]
D., Ho, J., Tarlow, D., and van den Berg, R
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. (2023). Structured denoising diffusion models in discrete state-spaces
work page 2023
-
[2]
A., Nazaret, A., Kucukelbir, A., and Blei, D
Beltran-Velez, N., Grande, A. A., Nazaret, A., Kucukelbir, A., and Blei, D. (2024). Treeffuser: Probabilistic predictions via conditional diffusions with gradient-boosted trees
work page 2024
-
[3]
Chen, T. and Zhou, M. (2023). Learning to jump: Thinning and thickening latent counts for generative modeling. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 5367--5382. PMLR
work page 2023
-
[4]
Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conference on Computer Vision and Pattern Recognition
work page 2020
-
[5]
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis
work page 2021
-
[6]
Dinh, L., Krueger, D., and Bengio, Y. (2014). Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516
arXiv 2014
-
[7]
and Cohen, V
Gokaslan, A. and Cohen, V. (2019). Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus
2019
-
[8]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. NeurIPS , 27
work page 2014
Show all 31 references
-
[9]
K., Atkinson, T., and Gomez, F
Graves, A., Srivastava, R. K., Atkinson, T., and Gomez, F. (2024). Bayesian flow networks
2024
-
[10]
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 30
2017
-
[11]
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS , 33:6840--6851
2020
-
[12]
Karras, T., Aittala, M., Aila, T., and Laine, S. (2022). Elucidating the design space of diffusion-based generative models. In NeurIPS
2022
-
[13]
Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks
2019
-
[14]
Kingma, D. P. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114
2013 arXiv
-
[15]
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images
2009
-
[16]
Lou, A., Meng, C., and Ermon, S. (2024). Discrete diffusion modeling by estimating the ratios of the data distribution
2024
-
[17]
MacKay, D. J. (2003). Information Theory, Inference, and Learning Algorithms . Cambridge University Press, Cambridge, UK
2003
-
[18]
Mahoney, M. (2011). text8: About the test data. http://mattmahoney.net/dc/textdata.html
2011
-
[19]
and Xie, S
Peebles, W. and Xie, S. (2022). Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748
2022 arXiv
-
[20]
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9
2019
-
[21]
and Mohamed, S
Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International Conference on Machine Learning , pages 1530--1538. PMLR
2015
-
[22]
J., Mohamed, S., and Wierstra, D
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , pages 1278--1286. PMLR
2014
-
[23]
S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024a). Simple and effective masked diffusion language models
2024
-
[24]
S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024b). Simple and effective masked diffusion language models
2024
-
[25]
Salazar, S. (2024). Vart: Variational regression trees. Advances in Neural Information Processing Systems , 36
2024
-
[26]
E., Fox, Z
Santos, J. E., Fox, Z. R., Lubbers, N., and Lin, Y. T. (2023). Blackout diffusion: generative diffusion models in discrete-state spaces. In International Conference on Machine Learning , pages 9034--9059. PMLR
2023
-
[27]
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. (2024). Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems
2024
-
[28]
P., Kumar, A., Ermon, S., and Poole, B
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456
2020 arXiv
-
[29]
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. (2016). Conditional image generation with pixelcnn decoders. NeuIPS , 29
2016
-
[30]
Vaswani, A. (2017). Attention is all you need. NeurIPS
2017
-
[31]
L., Juergens, D., Bennett, N
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., Wicky, B. I. M., Hanikel, N., Pellock, S. J., Courbet, A., Sheffler, W., Wang, J., Venkatesh, P., Sappington, I., Torres, S. V., Lauko,...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.