{"id":"393f3ab2-42d8-4bd2-a618-acdc14f58ae1","arxiv_id":"2412.13286","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A generative model learns to refine noisy observations via Bayesian posterior updates, matching diffusion-model performance and yielding a Cox-process SDE for count data.","lead":"This paper introduces posterior mean matching, a generative modeling method that uses online Bayesian inference with conjugate distribution pairs to create new images, text, and count data. It offers a Bayesian alternative to diffusion models with competitive performance on benchmarks, plus a new mathematical connection to jump-process SDEs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 proves forward consistency of the ideal posterior mean, but Algorithm 1's learned chain is never shown to inherit that guarantee; even with an optimal g_phi the variational transition omits posterior variance, so the terminal distribution q_phi(mu_T) is not proven to equal p*.","rationale":"The reader's weakest assumption was that g_phi is a sufficiently accurate approximation of the conditional expectation, with no error bounds. I agree that approximation error is unquantified, but I see a more structural issue: Theorem 1 concerns the forward process in which y_s is drawn from the true x*, while Algorithm 1 is a different Markov chain driven by g_phi. The paper's theoretical bridge from consistency to a sampling scheme is incomplete. This is a genuine load-bearing concern for the central claim, and it is testable with an oracle simulation that removes network error entirely. The concern does not, however, invalidate the empirical results or the framework; the right response is to require an additional theoretical argument (or a negative oracle result) before treating the sampling reduction as established. The paper has other weaknesses the reader noted: the main-text Theorem 1 assumes lim alpha_t = infinity while the appendix proof (Theorem A.1) requires sum alpha_s = Omega(t^{1+eta}), and the Cox-process SDE derivation is heuristic. These are secondary but reinforce conditional acceptance. Independently, the paper reports competitive FID and BPC numbers but provides no released code or checkpoints, which limits verification. On balance, the reader's CONDITIONAL verdict remains appropriate; nothing in this stress-test changes that recommendation, though it sharpens the condition: the authors should either prove a stationarity/contraction result for Algorithm 1 or provide an oracle experiment showing the variance misspecification is harmless.","tokens_in":30264,"tokens_out":12673,"duration_ms":139795,"concrete_test":"Oracle test in low dimension: take p* to be a 2-D mixture of Gaussians on which E[x|mu_{s-1}] and Var[x|mu_{s-1}] are computable in closed form. Simulate the Normal-Normal version of Algorithm 1 with g_s(mu) = E[x|mu_{s-1}=mu] (the exact minimizer of Eq. 20) instead of a learned network, using the paper's exponential noise schedule and T=3000. Compare the empirical distribution of mu_T over many rollouts to p* and to the ideal p(mu_T) using Wasserstein distance and the covariance of mu_T. If q_phi(mu_T) converges to p* as the time grid is refined, the variance misspecification is asymptotically harmless; if it does not, the Section 2 reduction is incomplete and the paper needs an explicit stationarity or contraction argument for Algorithm 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 2's inference that 'sampling from the target reduces to sampling mu_t from p(mu_1:t)' and that Algorithm 1, which replaces the unknown x* with g_phi, inherits this guarantee. Theorem 1 only proves consistency of the ideal posterior mean sequence when each y_s is generated from a fixed x* ~ p*. It says nothing about the Markov chain defined by equations (10)-(11). Even in an infinite-capacity, zero-approximation-error limit, the variational family q_phi is misspecified. For the Normal-Normal model, the true transition is p(mu_s | mu_{s-1}) = E_{x|mu_{s-1}} N(mu_s; a_s mu_{s-1} + b_s x, b_s^2 alpha_s^{-1}), where the expectation is over the posterior of x given mu_{s-1}; the learned transition q_phi is N(mu_s; a_s mu_{s-1} + b_s g_phi(mu_{s-1}), b_s^2 alpha_s^{-1}). The optimal g_phi(mu) = E[x|mu] matches the conditional mean, but q's variance omits b_s^2 Var[x|mu], so KL(p||q_phi) > 0 for any non-degenerate posterior. The PMM objective (20) only matches conditional means; it does not enforce that q-rollouts have terminal marginal q_phi(mu_T) equal to p(mu_T) or p*. Thus the central correctness claim is not established by the provided theorems, a gap distinct from and logically prior to finite-network approximation error.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Posterior Mean Matching (PMM), a generative modeling framework that uses conjugate Bayesian models to define an online denoising process. A data point x is corrupted by a sequence of noisy observations y_1:t; the posterior mean sequence mu_1:t of a conjugate Bayesian model is consistent, mu_t -> x, so the authors argue that sampling from the target distribution reduces to sampling from the joint distribution of posterior means. They approximate this joint distribution with a neural-network-parameterized transition q_phi, giving training objectives for Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical models. They further derive continuous-time limits connecting Normal-Normal PMM to an SDE and Gamma-Poisson PMM to a Cox-process-driven jump SDE. The paper reports competitive image generation results (CIFAR-10 FID 2.18) and language modeling results (text8 BPC 1.29, OpenWebText generative perplexity 42.58).","tokens_in":30704,"tokens_out":6025,"duration_ms":54588,"significance":"If the theoretical framing were fully established, PMM would be an appealing unification of Bayesian inference and generative modeling: it provides closed-form objectives for multiple data modalities, an explicit connection to diffusion SDEs, and a conceptually simple sampling algorithm. The paper is strong in its derivational detail: the PMM objectives are derived step by step, the hyperparameter schedules are specified, and the experiments cover several architectures and datasets. The empirical results are competitive with current diffusion-based models on images and with non-autoregressive diffusion language models on text. However, the central claim that Algorithm 1 samples from the target distribution is not proven, and several consistency theorems are stated more strongly than their proofs support. These issues are load-bearing because they concern the correctness guarantee that distinguishes PMM from a purely heuristic denoising scheme.","major_comments":[{"comment":"The central correctness claim that 'sampling from the target p*(x) reduces to sampling mu_t from p(mu_1:t)' is not established for the learned process. The ideal transitions have conditional variance b_s^2 alpha_s^{-1} given x, but the marginal transition p(mu_s | mu_{s-1}) has additional variance b_s^2 Var[x | mu_{s-1}] when x is integrated out. The variational transition q_phi in Eqs. (10)-(11) has variance b_s^2 alpha_s^{-1}, so even with the optimal g_phi(mu) = E[x | mu], the KL divergence between the true transition and q_phi is positive for any non-degenerate posterior. The PMM objective (20) matches only conditional means and does not control the terminal marginal q_phi(mu_T). Theorem 1 concerns the ideal Bayesian process, not the Markov chain in Algorithm 1; without a bound on the propagation of this variational error, the paper's theoretical guarantee does not cover the sampling procedure.","section":"§2, Eqs. (9)-(11), Algorithm 1"},{"comment":"The statement of Theorem 1 assumes only that alpha_t is positive, increasing, and lim_{t->infinity} alpha_t = infinity, but the proof relies on the stronger condition sum_{s=1}^t alpha_s = Omega(t^{1+eta}). For example, alpha_t = log t satisfies the stated assumption but not the proof's condition, since sum_{s=1}^t log s ~ t log t = o(t^{1+eta}) for every eta > 0. The theorem as stated is therefore unproven; the stronger summability condition should appear in the main statement, or a genuine proof under only lim alpha_t = infinity is needed.","section":"§3.1, Theorem 1 and Appendix A.2.2, Theorem A.1"},{"comment":"The Dirichlet-Categorical consistency proof establishes only convergence in probability: P(mu_t != x) -> 0 by the union bound. Section 2 defines consistency as almost sure convergence and states that all PMM models considered are consistent. The claim for this model is stronger than what is proven. Either prove almost sure convergence (e.g., by Borel-Cantelli with a summable bound on the per-time failure probability) or state the weaker mode of convergence.","section":"§3.2, Appendix A.3.2 (Theorem A.3)"},{"comment":"The Gamma-Poisson SDE limit is presented as a theorem, but the proof is a sketch. It asserts convergence of the discrete updates to a Merton jump process without a convergence argument, replaces the partial sums of Poisson variables by a non-homogeneous Poisson process N(t), and applies the Poisson Ito formula to N(t) while N(t) is actually a Cox process with random base measure x dt. The proof does not justify why the random nature of the base measure is compatible with the deterministic view of N(t) used in the Ito calculation. At minimum, the hypotheses under which the discrete process converges in distribution to the SDE, and the sense of convergence, should be stated.","section":"§4, Theorem 3 and Appendix A.4.4"}],"minor_comments":[{"comment":"The factor alpha_s multiplying y_s in the stated Gamma-Poisson update appears inconsistent with the posterior mean derivation in Appendix A.4.1, Eq. (128), where the coefficient of y_s is 1. Please correct or clarify.","section":"§3.3, Eq. (32)"},{"comment":"The statement 'lim_{t->infinity} sum_{s=1}^t alpha_s = O(t^{1+eta}) -> infinity' mixes asymptotic notations incorrectly; the proof requires a lower bound sum_{s=1}^t alpha_s = Omega(t^{1+eta}), not an upper bound.","section":"Appendix A.4.2, Theorem A.4"},{"comment":"The phrase 'variational inference variational inference' is duplicated; remove the repetition.","section":"§2, paragraph before Eq. (9)"},{"comment":"The table reports FID scores with various superscript symbols but the caption does not define these symbols; make the caption self-contained so that the reader can tell which rows use higher-order solvers or data augmentation.","section":"§5.1, Table 1"},{"comment":"The reweighted loss is a modification of the derived PMM objective, and the main text's Section 5 does not clearly state that the reported experiments use the reweighted version; this should be disclosed in the main text.","section":"Appendix A.2.3, Eqs. (76)-(77)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a real new method class, and the empirical work is credible, but the paper's theoretical framing is stronger than what is actually proven. The stress-test note is right: Theorem 1 shows the ideal posterior mean converges, but Algorithm 1's learned chain is never shown to inherit that guarantee, and even with a perfect g_phi the variational transition omits the posterior variance term, so KL(p||q_phi) > 0. That is a genuine gap, not a nitpick. It is a common gap in generative modeling—most diffusion papers don't prove the learned sampler exactly samples the target either—but the paper should be explicit that the consistency theorem applies to the ideal process, not to the fitted sampler.\n\nWhat is new: the PMM framework itself, extended to Normal-Normal, Dirichlet-Categorical, and Gamma-Poisson, with a Cox-process SDE in the Poisson case. The Normal-Normal variant is close to BFN, and the paper acknowledges that. The Dirichlet-Categorical non-informative prior update and the continuous-time objective are genuinely useful, and the Inverse-Gamma extension in the appendix is a nice bonus. The results—FID 2.18 on CIFAR-10, BPC 1.29 on text8—are competitive with non-autoregressive baselines, and the NFE trade-off table is informative.\n\nSoft spots, in proportion:\n\n- Theorem 1 in the main text says lim alpha_t -> infinity, but the appendix proof needs sum alpha_s = Omega(t^{1+eta}). That's a fixable mismatch, but it should be reconciled.\n- The Gamma-Poisson SDE (Theorem 3) proof is a sketch. The step where the counting process becomes a Cox process with random base measure is hand-waved. This is fine as an informal derivation, but it should be labeled as such.\n- The Dirichlet-Categorical consistency is only in probability, not a.s. Minor.\n- No code or checkpoints released, and no error bars on the headline numbers. The generative perplexity comparison uses a different tokenizer and a smaller model, which the paper discloses in a footnote, but it still weakens the comparison.\n\nCitation pattern is honest; BFN is credited and compared to. No fitted constants are folded into the theory, so the circular-burden concern is not real.\n\nBottom line: this is a solid, publishable paper needing revision, not rejection. The framework is coherent, the experiments are competitive, and the theoretical gaps are identifiable and fixable. A serious referee should engage with it, but should push for the theorem conditions to be stated accurately and for the sampler's approximation error to be discussed honestly.\n\nI'd send it to review, and I'd cite it if I worked in this area.","headline":"Genuinely new method class with competitive results, but the consistency theorem is about the ideal process, not the learned sampler; fix the theorem conditions and be honest about the approximation gap.","tokens_in":31213,"tokens_out":3507,"would_cite":true,"duration_ms":32896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Posterior means, iterated, become a generative sampler.","keywords":["posterior mean matching","generative modeling","online Bayesian inference","conjugate priors","diffusion models","stochastic differential equations","Cox process","Dirichlet-Categorical model"],"falsifier":"Train the Normal-Normal PMM on a low-dimensional target with known density (for example a mixture of Gaussians), push the network to convergence, and run Algorithm 1 for a very large number of steps; if the empirical distribution of the output $\\mu_t$ does not approach the target as the number of steps grows, then the consistency guarantee does not survive the neural approximation and the central reduction fails. A cleaner version compares PMM's samples against direct sampling from the exact posterior-mean joint distribution on the same target.","tokens_in":30080,"feed_emoji":"🧠","tokens_out":7746,"duration_ms":65133,"temperature":0.7,"pith_summary":"This paper proposes posterior mean matching (PMM), a generative modeling method that turns online Bayesian inference into a sampler. The recipe is to draw noisy versions of a training sample, run them through a conjugate Bayesian model, and treat the sequence of posterior means as the generation path; a consistency theorem says the final mean recovers the original sample almost surely. Because the argument works for any conjugate pair, swapping the pair yields models for real-valued data, count data, and discrete text without changing the outer algorithm. The paper also derives the continuous-time limits: the Normal-Normal PMM converges to a diffusion-like SDE, the Gamma-Poisson PMM to a jump SDE driven by a Cox process, and reports image and text generation competitive with diffusion baselines.","feed_headline":"Posterior means, iterated, become a generative sampler","feed_subtitle":"A conjugate model's posterior mean provably converges to the target sample, rivaling diffusion models on images and text.","key_machinery":"The load-bearing object is the online posterior mean $\\mu_t$ of a conjugate Bayesian model, updated by the closed-form rule $\\mu_{t+1} = f_t(\\mu_t, y_{t+1})$ as noisy observations arrive. Its consistency — $\\mu_t \\to x^\\ast$ almost surely under the augmented target — is what turns Bayesian inference into generation: $\\mu_t$ is effectively a sample from $p^\\ast(x)$. The approximating distribution $q_\\varphi$ replaces the unknown next observation with a draw from $\\pi_\\alpha(\\cdot \\mid g_\\varphi(\\mu_{t-1},t))$, where $g_\\varphi$ is a neural network, so the KL objective collapses to matching $g_\\varphi$ against the true $x$; for the Normal-Normal model this yields the weighted squared error $\\alpha_s\\|x - g_\\varphi(\\mu_{s-1},s)\\|_2^2$.","core_discovery":"The central claim is that, in an augmented Bayesian model with a conjugate prior $\\pi(x)$ and a noisy observation model $\\pi_{\\alpha_s}(y|x)$, the posterior mean $\\mu_t = \\mathbb{E}_\\pi[x \\mid y_{1:t}]$ converges almost surely to the hidden true observation $x^\\ast \\sim p^\\ast(x)$ as $t\\to\\infty$; Theorem 1 proves this for the Normal-Normal model, with analogues for the other pairs. It follows that sampling from the target $p^\\ast(x)$ reduces to sampling from the joint distribution $p(\\mu_1,\\ldots,\\mu_t)$ of posterior-mean trajectories and reporting $\\mu_t$. Since that joint distribution is intractable, PMM approximates it with a parametric transition $q_\\varphi$ whose one step draws a noisy observation around $g_\\varphi(\\mu_{t-1},t)$ and applies the exact online update; the network is trained by minimizing the KL divergence in Equation (9). The paper works out closed-form objectives for the Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical cases and validates the approach on image and text benchmarks.","pith_inferences":["The consistency theorems govern the ideal posterior mean, not the fitted network; a practical test would isolate how much sample quality degrades as network capacity or training length shrinks on a target with known density.","The same outer algorithm could be instantiated with other conjugate pairs in the exponential family, for example the InverseGamma-Gamma pair sketched in the appendix, to target positive continuous data or variance-like quantities without modifying the training loop.","Because the PMM objective is a reweighted denoising loss, established diffusion-model machinery such as learned noise schedules, guidance, and distillation could plausibly be transferred to PMM, though the paper does not explore this.","The Cox-process SDE for the Gamma-Poisson model suggests that PMM may be a natural fit for sparse integer data and for algorithms that natively produce counts without an explicit discretization or rounding step."],"forward_implications":["Any conjugate pair whose posterior mean is consistent gives a working PMM, so the paper supplies a single template for real-valued, count, and discrete targets; the demonstrated pairs are Normal-Normal, Gamma-Poisson, and Dirichlet-Categorical.","The Normal-Normal PMM converges in the continuum limit to a stochastic differential equation of the form $d\\mu(t) = \\frac{f(t)(x-\\mu(t))}{b+\\int_0^t f(\\tau)d\\tau}dt + \\sqrt{\\frac{f(t)}{b+\\int_0^t f(\\tau)d\\tau}}dW_t$, so off-the-shelf SDE solvers apply and the sampling cost can be traded against quality.","The Gamma-Poisson PMM converges to an SDE driven by a Cox process with random base measure $x\\,dt$, $x\\sim p^\\ast(x)$, a departure from Brownian-motion generative models and the first such limit for a Poisson-likelihood generative model.","The reported results put PMM near diffusion models on images (FID 2.18 on CIFAR-10 for the Normal-Normal model) and ahead of non-autoregressive diffusion baselines on text8 (1.29 BPC), narrowing the gap to autoregressive language models."],"supporting_citations":[{"why":"Closest Bayesian generative baseline; PMM is positioned against Bayesian flow nets and compared with it on text.","marker":"Graves et al., 2024"},{"why":"Defines the DDPM denoising formulation whose objectives and FID results PMM's Normal-Normal model is set against.","marker":"Ho et al., 2020"},{"why":"Provides the score-SDE framework that the PMM continuous-time limits are formally connected to and contrasted with.","marker":"Song et al., 2020"},{"why":"Supplies the DDPM++ architecture, FID evaluation pipeline, and baseline numbers used in image experiments.","marker":"Karras et al., 2022"},{"why":"Provides the UNet implementation used for all image PMM models.","marker":"Dhariwal and Nichol, 2021"},{"why":"Establishes the SEDD discrete-diffusion baseline and the language-model evaluation protocol for text.","marker":"Lou et al., 2024"},{"why":"Defines the MDLM masked-diffusion baseline used for generative perplexity comparisons.","marker":"Sahoo et al., 2024a"},{"why":"Supplies the simplified masked-diffusion baseline for discrete data that the text results are compared with.","marker":"Shi et al., 2024"}],"fun_headline_variants":["Iterated posterior means rival diffusion on images and text","Bayesian online updates generate images and text","Posterior mean matching turns inference into generation","Conjugate posterior means converge to a sampler","PMM: sampling by iterating Bayesian posterior means"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned network must be close enough to the exact Bayesian update that the consistency guarantee carries over; the paper gives no bound on how close is close enough.","fun_headline_variants_meta":{"raw":{"variants":["Iterated posterior means rival diffusion on images and text","Bayesian online updates generate images and text","Posterior mean matching turns inference into generation","Conjugate posterior means converge to a sampler","PMM: sampling by iterating Bayesian posterior means"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2632,"prompt_tokens":982,"completion_tokens":1650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1580}},"tokens_in":598,"tokens_out":1650,"duration_ms":14462,"temperature":1.0,"reasoning_tokens":1580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:16:59.609349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the Normal-Normal PMM on a low-dimensional target with known density (for example a mixture of Gaussians), push the network to convergence, and run Algorithm 1 for a very large number of steps; if the empirical distribution of the output $\\mu_t$ does not approach the target as the number of steps grows, then the consistency guarantee does not survive the neural approximation and the central reduction fails. A cleaner version compares PMM's samples against direct sampling from the exact posterior-mean joint distribution on the same target.","supporting_citations":[{"cited_title":"K., Atkinson, T., and Gomez, F","cited_arxiv_id":null,"evidence_quote":"Closest Bayesian generative baseline; PMM is positioned against Bayesian flow nets and compared with it on text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DDPM++ architecture, FID evaluation pipeline, and baseline numbers used in image experiments."},{"cited_title":"and Nichol, A","cited_arxiv_id":null,"evidence_quote":"Provides the UNet implementation used for all image PMM models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the SEDD discrete-diffusion baseline and the language-model evaluation protocol for text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the simplified masked-diffusion baseline for discrete data that the text results are compared with."}],"review_version":1}