Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Posterior Mean Matching: Generative Modeling through Online Bayesian Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Posterior means, iterated, become a generative sampler.

desk verdict Genuinely new method class with competitive results, but the consistency theorem is about the ideal process, not the learned sampler; fix the theorem conditions and be honest about the approximation gap. read the letter →

arxiv 2412.13286 v2 pith:4MZY2MXB submitted 2024-12-17 cs.LG cs.AIstat.ML

classification cs.LGcs.AIstat.ML
keywords posteriormeanmatchinggenerativemodelingonlineBayesianinferenceconjugatepriorsdiffusionmodelsstochasticdifferentialequationsCoxprocessDirichlet-Categoricalmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes posterior mean matching (PMM), a generative modeling method that turns online Bayesian inference into a sampler. The recipe is to draw noisy versions of a training sample, run them through a conjugate Bayesian model, and treat the sequence of posterior means as the generation path; a consistency theorem says the final mean recovers the original sample almost surely. Because the argument works for any conjugate pair, swapping the pair yields models for real-valued data, count data, and discrete text without changing the outer algorithm. The paper also derives the continuous-time limits: the Normal-Normal PMM converges to a diffusion-like SDE, the Gamma-Poisson PMM to a jump SDE driven by a Cox process, and reports image and text generation competitive with diffusion baselines.

What carries the argument

The load-bearing object is the online posterior mean $\mu_t$ of a conjugate Bayesian model, updated by the closed-form rule $\mu_{t+1} = f_t(\mu_t, y_{t+1})$ as noisy observations arrive. Its consistency — $\mu_t \to x^\ast$ almost surely under the augmented target — is what turns Bayesian inference into generation: $\mu_t$ is effectively a sample from $p^\ast(x)$. The approximating distribution $q_\varphi$ replaces the unknown next observation with a draw from $\pi_\alpha(\cdot \mid g_\varphi(\mu_{t-1},t))$, where $g_\varphi$ is a neural network, so the KL objective collapses to matching $g_\varphi$ against the true $x$; for the Normal-Normal model this yields the weighted squared error $\alpha_s\|x - g_\varphi(\mu_{s-1},s)\|_2^2$.

What would settle it

Train the Normal-Normal PMM on a low-dimensional target with known density (for example a mixture of Gaussians), push the network to convergence, and run Algorithm 1 for a very large number of steps; if the empirical distribution of the output $\mu_t$ does not approach the target as the number of steps grows, then the consistency guarantee does not survive the neural approximation and the central reduction fails. A cleaner version compares PMM's samples against direct sampling from the exact posterior-mean joint distribution on the same target.

Watch

Extended reading notes

Core claim

The central claim is that, in an augmented Bayesian model with a conjugate prior $\pi(x)$ and a noisy observation model $\pi_{\alpha_s}(y|x)$, the posterior mean $\mu_t = \mathbb{E}_\pi[x \mid y_{1:t}]$ converges almost surely to the hidden true observation $x^\ast \sim p^\ast(x)$ as $t\to\infty$; Theorem 1 proves this for the Normal-Normal model, with analogues for the other pairs. It follows that sampling from the target $p^\ast(x)$ reduces to sampling from the joint distribution $p(\mu_1,\ldots,\mu_t)$ of posterior-mean trajectories and reporting $\mu_t$. Since that joint distribution is intractable, PMM approximates it with a parametric transition $q_\varphi$ whose one step draws a noisy observation around $g_\varphi(\mu_{t-1},t)$ and applies the exact online update; the network is trained by minimizing the KL divergence in Equation (9). The paper works out closed-form objectives for the Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical cases and validates the approach on image and text benchmarks.

Load-bearing premise

The learned network must be close enough to the exact Bayesian update that the consistency guarantee carries over; the paper gives no bound on how close is close enough.

Editorial extensions

If this is right

  • Any conjugate pair whose posterior mean is consistent gives a working PMM, so the paper supplies a single template for real-valued, count, and discrete targets; the demonstrated pairs are Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical.
  • The Normal-Normal PMM converges in the continuum limit to a stochastic differential equation of the form $d\mu(t) = \frac{f(t)(x-\mu(t))}{b+\int_0^t f(\tau)d\tau}dt + \sqrt{\frac{f(t)}{b+\int_0^t f(\tau)d\tau}}dW_t$, so off-the-shelf SDE solvers apply and the sampling cost can be traded against quality.
  • The Gamma-Poisson PMM converges to an SDE driven by a Cox process with random base measure $x\,dt$, $x\sim p^\ast(x)$, a departure from Brownian-motion generative models and the first such limit for a Poisson-likelihood generative model.
  • The reported results put PMM near diffusion models on images (FID 2.18 on CIFAR-10 for the Normal-Normal model) and ahead of non-autoregressive diffusion baselines on text8 (1.29 BPC), narrowing the gap to autoregressive language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistency theorems govern the ideal posterior mean, not the fitted network; a practical test would isolate how much sample quality degrades as network capacity or training length shrinks on a target with known density.
  • The same outer algorithm could be instantiated with other conjugate pairs in the exponential family, for example the InverseGamma-Gamma pair sketched in the appendix, to target positive continuous data or variance-like quantities without modifying the training loop.
  • Because the PMM objective is a reweighted denoising loss, established diffusion-model machinery such as learned noise schedules, guidance, and distillation could plausibly be transferred to PMM, though the paper does not explore this.
  • The Cox-process SDE for the Gamma-Poisson model suggests that PMM may be a natural fit for sparse integer data and for algorithms that natively produce counts without an explicit discretization or rounding step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Posterior Mean Matching (PMM), a generative modeling framework that uses conjugate Bayesian models to define an online denoising process. A data point x is corrupted by a sequence of noisy observations y_1:t; the posterior mean sequence mu_1:t of a conjugate Bayesian model is consistent, mu_t -> x, so the authors argue that sampling from the target distribution reduces to sampling from the joint distribution of posterior means. They approximate this joint distribution with a neural-network-parameterized transition q_phi, giving training objectives for Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical models. They further derive continuous-time limits connecting Normal-Normal PMM to an SDE and Gamma-Poisson PMM to a Cox-process-driven jump SDE. The paper reports competitive image generation results (CIFAR-10 FID 2.18) and language modeling results (text8 BPC 1.29, OpenWebText generative perplexity 42.58).

Significance. If the theoretical framing were fully established, PMM would be an appealing unification of Bayesian inference and generative modeling: it provides closed-form objectives for multiple data modalities, an explicit connection to diffusion SDEs, and a conceptually simple sampling algorithm. The paper is strong in its derivational detail: the PMM objectives are derived step by step, the hyperparameter schedules are specified, and the experiments cover several architectures and datasets. The empirical results are competitive with current diffusion-based models on images and with non-autoregressive diffusion language models on text. However, the central claim that Algorithm 1 samples from the target distribution is not proven, and several consistency theorems are stated more strongly than their proofs support. These issues are load-bearing because they concern the correctness guarantee that distinguishes PMM from a purely heuristic denoising scheme.

major comments (4)
  1. [§2, Eqs. (9)-(11), Algorithm 1] The central correctness claim that 'sampling from the target p*(x) reduces to sampling mu_t from p(mu_1:t)' is not established for the learned process. The ideal transitions have conditional variance b_s^2 alpha_s^{-1} given x, but the marginal transition p(mu_s | mu_{s-1}) has additional variance b_s^2 Var[x | mu_{s-1}] when x is integrated out. The variational transition q_phi in Eqs. (10)-(11) has variance b_s^2 alpha_s^{-1}, so even with the optimal g_phi(mu) = E[x | mu], the KL divergence between the true transition and q_phi is positive for any non-degenerate posterior. The PMM objective (20) matches only conditional means and does not control the terminal marginal q_phi(mu_T). Theorem 1 concerns the ideal Bayesian process, not the Markov chain in Algorithm 1; without a bound on the propagation of this variational error, the paper's theoretical guarantee does not cover the sampling procedure.
  2. [§3.1, Theorem 1 and Appendix A.2.2, Theorem A.1] The statement of Theorem 1 assumes only that alpha_t is positive, increasing, and lim_{t->infinity} alpha_t = infinity, but the proof relies on the stronger condition sum_{s=1}^t alpha_s = Omega(t^{1+eta}). For example, alpha_t = log t satisfies the stated assumption but not the proof's condition, since sum_{s=1}^t log s ~ t log t = o(t^{1+eta}) for every eta > 0. The theorem as stated is therefore unproven; the stronger summability condition should appear in the main statement, or a genuine proof under only lim alpha_t = infinity is needed.
  3. [§3.2, Appendix A.3.2 (Theorem A.3)] The Dirichlet-Categorical consistency proof establishes only convergence in probability: P(mu_t != x) -> 0 by the union bound. Section 2 defines consistency as almost sure convergence and states that all PMM models considered are consistent. The claim for this model is stronger than what is proven. Either prove almost sure convergence (e.g., by Borel-Cantelli with a summable bound on the per-time failure probability) or state the weaker mode of convergence.
  4. [§4, Theorem 3 and Appendix A.4.4] The Gamma-Poisson SDE limit is presented as a theorem, but the proof is a sketch. It asserts convergence of the discrete updates to a Merton jump process without a convergence argument, replaces the partial sums of Poisson variables by a non-homogeneous Poisson process N(t), and applies the Poisson Ito formula to N(t) while N(t) is actually a Cox process with random base measure x dt. The proof does not justify why the random nature of the base measure is compatible with the deterministic view of N(t) used in the Ito calculation. At minimum, the hypotheses under which the discrete process converges in distribution to the SDE, and the sense of convergence, should be stated.
minor comments (5)
  1. [§3.3, Eq. (32)] The factor alpha_s multiplying y_s in the stated Gamma-Poisson update appears inconsistent with the posterior mean derivation in Appendix A.4.1, Eq. (128), where the coefficient of y_s is 1. Please correct or clarify.
  2. [Appendix A.4.2, Theorem A.4] The statement 'lim_{t->infinity} sum_{s=1}^t alpha_s = O(t^{1+eta}) -> infinity' mixes asymptotic notations incorrectly; the proof requires a lower bound sum_{s=1}^t alpha_s = Omega(t^{1+eta}), not an upper bound.
  3. [§2, paragraph before Eq. (9)] The phrase 'variational inference variational inference' is duplicated; remove the repetition.
  4. [§5.1, Table 1] The table reports FID scores with various superscript symbols but the caption does not define these symbols; make the caption self-contained so that the reader can tell which rows use higher-order solvers or data augmentation.
  5. [Appendix A.2.3, Eqs. (76)-(77)] The reweighted loss is a modification of the derived PMM objective, and the main text's Section 5 does not clearly state that the reported experiments use the reweighted version; this should be disclosed in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the PMM derivation is self-contained; the posterior-mean limit theorems are proved in the appendix and the training objective is a genuine variational divergence.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. Theorem 1 and its appendix version prove consistency of the ideal posterior-mean sequence directly from the stated conjugate model and the growth condition on alpha_t; the conclusion mu_t -> x* a.s. is a limit theorem about the Bayesian update, not a fitted quantity, so it is not a 'prediction' of the trained system. The PMM objective (9) is an explicit KL divergence between the true posterior-mean process and the variational process q_phi, and Equations (20), (31), and (73)-(77) follow by algebra from the conjugate update rules and the reparametrization trick; the reweighted loss is disclosed as an implementation choice corresponding to a learning-rate adjustment rather than a hidden fit. Theorems 2 and 3 are continuum limits of the stated discrete update rules, derived in the appendix, so they are mathematical consequences rather than circular predictions. The only self-citations (Salazar 2024; Beltran-Velez et al. 2024) appear as examples of tabular-data applications and are not load-bearing for any central claim. A skeptical reader is right that Theorem 1 does not by itself establish that the learned chain of Algorithm 1 has terminal marginal equal to p*: the variational transition (10)-(11) omits posterior variance, so the guarantee does not fully transfer to the neural sampler. That is a correctness or approximation gap, not a circularity, because the theorem is neither defined in terms of the algorithm's output nor fitted to it. The empirical evaluation is also measured against external baselines and standard FID, BPC, and perplexity benchmarks, so no self-referential validation scheme is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claims rest on conjugate pairs with closed-form posterior means, consistency of the posterior mean under a fast-growing noise schedule, and the assumption that the neural network approximates the Bayesian update well. The hand-selected schedules and prior parameters are not fitted to minimize the reported metrics, but they do affect performance.

free parameters (5)
  • Normal-Normal prior precision beta = 2
    Hand-selected; controls prior tightness in the Normal-Normal model and affects posterior mean dynamics.
  • Gamma-Poisson prior parameters = alpha=0.1, gamma=2 (Table 5)
    Hand-selected; the Gamma prior shape and rate affect the posterior mean update.
  • Noise schedule f(t) for images = f(t)=13/250 * e^(13t)
    Hand-selected exponential schedule; defines the alpha_t sequence and controls noise corruption over time.
  • Unmasking schedule f_c(t) for text = staircase or time-dependent variants in Table 5
    Hand-selected schedules for token unmasking; they affect training dynamics and sample quality.
  • Number of posterior updates (NFE) = 3000-5000 for images, varied for text
    Chosen as a compute/quality trade-off; the paper reports FID as a function of NFE.
assumptions (5)
  • domain assumption The dataset consists of i.i.d. samples from the target distribution p*(x).
    Stated at the start of Section 2: 'we assume that we are given a dataset {x_i} of i.i.d. samples from the target distribution p*(x)'.
  • domain assumption The augmented Bayesian model has a closed-form posterior mean and is amenable to online updates.
    Required at the start of Section 2; the paper only considers conjugate pairs that satisfy these properties.
  • ad hoc to paper The posterior mean is consistent, i.e., mu_t -> x* almost surely, under the chosen noise schedule.
    Proven in the appendix under sum alpha_s = Omega(t^(1+eta)); the main text states a weaker condition. This is load-bearing for the claim that mu_t is a sample from p*.
  • domain assumption The neural network g_phi can represent and learn the conditional expectation E[x | mu_{t-1}, t].
    Standard capacity and optimization assumption for all neural generative models; no theoretical guarantee is provided.
  • standard math The Poisson sum sum_s Pois(x alpha_s) converges to a non-homogeneous Poisson process in the continuum limit.
    Used in the Gamma-Poisson SDE derivation (Appendix A.4.4); relies on standard Poisson process convergence, stated without full proof.
invented entities (2)
  • Cox process with random base measure x dt
    purpose: The driving noise in the Gamma-Poisson SDE, replacing Brownian motion in the generative process.
    A stochastic process defined by the paper's model; no independent empirical handle is provided beyond the SDE derivation itself.
  • Augmented target distribution p(x, y_1:t)
    purpose: Bridges the target p*(x) and the Bayesian model, enabling posterior mean sampling.
    A mathematical construction; not a physical entity with independent evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Posterior Mean Matching: Generative Modeling through Online Bayesian Inference." pith.science (2026). https://pith.science/paper/4MZY2MXB

@misc{pith2026241213286,
  author       = {Pith},
  title        = {Pith review of: Posterior Mean Matching: Generative Modeling through Online Bayesian Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4MZY2MXB}},
  note         = {Machine review of arXiv:2412.13286}
}
read the original abstract

This paper introduces posterior mean matching (PMM), a new method for generative modeling that is grounded in Bayesian inference. PMM uses conjugate pairs of distributions to model complex data of various modalities like images and text, offering a flexible alternative to existing methods like diffusion models. PMM models iteratively refine noisy approximations of the target distribution using updates from online Bayesian inference. PMM is flexible because its mechanics are based on general Bayesian models. We demonstrate this flexibility by developing specialized examples: a generative PMM model of real-valued data using the Normal-Normal model, a generative PMM model of count data using a Gamma-Poisson model, and a generative PMM model of discrete data using a Dirichlet-Categorical model. For the Normal-Normal PMM model, we establish a direct connection to diffusion models by showing that its continuous-time formulation converges to a stochastic differential equation (SDE). Additionally, for the Gamma-Poisson PMM, we derive a novel SDE driven by a Cox process, which is a significant departure from traditional Brownian motion-based generative models. PMMs achieve performance that is competitive with generative models for language modeling and image generation.

Figures

Figures reproduced from arXiv: 2412.13286 by the authors.

Figure 1
Figure 1. Diagram of the online Bayesian inference [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Convergence of the posterior mean trajec [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of sample generations for the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    D., Ho, J., Tarlow, D., and van den Berg, R

    Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. (2023). Structured denoising diffusion models in discrete state-spaces

  2. [2]

    A., Nazaret, A., Kucukelbir, A., and Blei, D

    Beltran-Velez, N., Grande, A. A., Nazaret, A., Kucukelbir, A., and Blei, D. (2024). Treeffuser: Probabilistic predictions via conditional diffusions with gradient-boosted trees

  3. [3]

    and Zhou, M

    Chen, T. and Zhou, M. (2023). Learning to jump: Thinning and thickening latent counts for generative modeling. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 5367--5382. PMLR

  4. [4]

    Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conference on Computer Vision and Pattern Recognition

  5. [5]

    and Nichol, A

    Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis

  6. [6]

    Dinh, L., Krueger, D., and Bengio, Y. (2014). Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516

  7. [7]

    and Cohen, V

    Gokaslan, A. and Cohen, V. (2019). Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus

  8. [8]

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. NeurIPS , 27

Show all 31 references
  1. [9]

    K., Atkinson, T., and Gomez, F

    Graves, A., Srivastava, R. K., Atkinson, T., and Gomez, F. (2024). Bayesian flow networks

  2. [10]

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 30

  3. [11]

    Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS , 33:6840--6851

  4. [12]

    Karras, T., Aittala, M., Aila, T., and Laine, S. (2022). Elucidating the design space of diffusion-based generative models. In NeurIPS

  5. [13]

    Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks

  6. [14]

    Kingma, D. P. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  7. [15]

    Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images

  8. [16]

    Lou, A., Meng, C., and Ermon, S. (2024). Discrete diffusion modeling by estimating the ratios of the data distribution

  9. [17]

    MacKay, D. J. (2003). Information Theory, Inference, and Learning Algorithms . Cambridge University Press, Cambridge, UK

  10. [18]

    Mahoney, M. (2011). text8: About the test data. http://mattmahoney.net/dc/textdata.html

  11. [19]

    and Xie, S

    Peebles, W. and Xie, S. (2022). Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748

  12. [20]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9

  13. [21]

    and Mohamed, S

    Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International Conference on Machine Learning , pages 1530--1538. PMLR

  14. [22]

    J., Mohamed, S., and Wierstra, D

    Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , pages 1278--1286. PMLR

  15. [23]

    S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J

    Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024a). Simple and effective masked diffusion language models

  16. [24]

    S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J

    Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024b). Simple and effective masked diffusion language models

  17. [25]

    Salazar, S. (2024). Vart: Variational regression trees. Advances in Neural Information Processing Systems , 36

  18. [26]

    E., Fox, Z

    Santos, J. E., Fox, Z. R., Lubbers, N., and Lin, Y. T. (2023). Blackout diffusion: generative diffusion models in discrete-state spaces. In International Conference on Machine Learning , pages 9034--9059. PMLR

  19. [27]

    Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. (2024). Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems

  20. [28]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456

  21. [29]

    Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. (2016). Conditional image generation with pixelcnn decoders. NeuIPS , 29

  22. [30]

    Vaswani, A. (2017). Attention is all you need. NeurIPS

  23. [31]

    L., Juergens, D., Bennett, N

    Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., Wicky, B. I. M., Hanikel, N., Pellock, S. J., Courbet, A., Sheffler, W., Wang, J., Venkatesh, P., Sappington, I., Torres, S. V., Lauko,...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.