{"id":"24cae55b-a9d6-40de-b5de-5118739d19d1","arxiv_id":"2412.11743","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A likelihood-free Bayesian RL method trains deep generative environment models with prequential scoring rules, samples via SMC, and learns policies with expected Thompson sampling.","lead":"This paper combines likelihood-free Bayesian inference with deep learning models of an environment, then uses many posterior samples to choose actions in reinforcement learning. It adds a theory showing the approximate posterior concentrates, and tests the approach on pendulum and hopper control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BvM rests on stationarity/mixing assumptions that the adaptive greedy-policy ETS procedure does not guarantee; under-sampled actions can break the action-decomposed ULLN.","rationale":"Good-faith summary: the paper proposes likelihood-free Thompson sampling based on prequential scoring-rule posteriors, with a claimed BvM theorem and an ETS value-error bound. The empirical chain-task, pendulum, and hopper results provide supporting evidence, but the central theoretical claims are only as strong as the assumptions under which the ULLN and BvM are proved. I agree with the reader's weakest-assumption identification: A2/A3 are not checked and are not automatically satisfied in the adaptive episodic setting. My reading sharpens this concern: because Algorithm 1 uses deterministic greedy policies, exploration can vanish, so the action-decomposed ULLN in Lemma 2 is especially fragile; there may be actions with bounded counts, in which case A2/A3 fail structurally. The paper's own closing remark that neural-network posteriors can be multimodal is a further admission that smoothness/unimodality assumptions A8-A10 are not met in the misspecified experiments; this is consistent with a conditional rather than an unconditional accept. I am not calling for rejection: the method is plausible and the empirical results, though limited, support the practical claims. The concern is that the headline theory is not yet proven for the algorithm as run, so the reader's CONDITIONAL verdict is appropriate and no verdict change is needed from this pass.","tokens_in":26277,"tokens_out":6256,"duration_ms":62065,"concrete_test":"Run the chain-task setup (Section 2.1) with the true M0 known and execute Algorithm 1 exactly: posterior updates after each episode, deterministic arg max policy updates, no added exploration, for both n=1 and n>1 posterior samples. For K=100, 500, and 1000 episodes with fixed episode length tau, track per-action counts |T_a|, estimate alpha-mixing coefficients of each action-conditioned state subsequence, and compute the ULLN quantity sup_{theta in grid} |PS_T(theta)/T - PS*(theta)| using the prequential energy score and the known true model. If some |T_a| stays bounded as T grows, or if the sup-norm fails to vanish across repeated seeds, Assumptions A2/A3 are violated and Lemma 2/Theorem 5 cannot cover the adaptive ETS algorithm as run. If instead all action counts grow linearly and the sup-norm vanishes, the concern is refuted in this controlled setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumptions A2/A3 in Section 3.2 are the load-bearing conditions for Lemma 2, and hence for Corollary 3, Lemma 4 and the BvM Theorem 5. They are stated about the data-generating process under the agent's policy, but Algorithm 1 updates the policy every episode from accumulated data and then plays greedily (mu_{k+1}(s)=arg max_a Q^{theta(k)}_mu). For such an adaptive, deterministic policy sequence, A2 and A3 are not automatic consequences of M0's Markov property, as the paper suggests: the action sequence depends on the whole history, and conditioning on the random times T_a={t<=T:a_t=a} can destroy mixing. More sharply, if the policy converges to a deterministic one, |T_a| may fail to grow for suboptimal actions, so the Cesaro average in A2 has no non-degenerate limit and the per-action ULLN in the proof of Lemma 2 cannot be applied. The paper provides no verification of A2/A3 on the chain, pendulum, or hopper experiments, so Theorem 5 is not established for the adaptive episodic setting. Theorem 6 inherits this gap because its zeta_l(k,n) terms are claimed to shrink via Theorem 5; Appendix B additionally assumes without proof that greedy policies under Q^{theta*} and Q^{theta(k)} coincide for large k and n, which is a stability property that does not follow from BvM alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generalized Bayesian framework for model-based reinforcement learning in which the environment is a parametric Markov decision process with an intractable likelihood, modeled by a deep generative network. A prequential scoring rule posterior proportional to the prior times exp(-PS_T(theta)) is introduced, sampled via sequential Monte Carlo with an adjusted stochastic gradient Riemannian Langevin kernel. The authors prove a uniform law of large numbers, consistency of the prequential score minimizer, and a Bernstein-von Mises theorem under assumptions A1-A10, then derive an error bound for expected Thompson sampling in Theorem 6. Experiments on a chain task, inverted pendulum (well-specified and misspecified), and Hopper show sample-efficiency gains over model-free baselines, with the continuous-action Hopper extension explicitly lacking theoretical guarantees.","tokens_in":26549,"tokens_out":10960,"duration_ms":103417,"significance":"If fully established, the framework would provide a useful likelihood-free posterior inference recipe for simulator-based MDPs and a multi-sample Thompson sampling scheme with a formal value-function error bound. The paper is transparent that the continuous-action extension and the misspecified deep generative models fall outside the stated theoretical conditions. The main weaknesses are that the central asymptotic results are stated for an adaptively updated, greedy policy while the stationarity and mixing assumptions are neither verified nor automatic, and that the ETS rate argument in Appendix B contains a gap. These issues are substantive but appear addressable by restricting the theoretical claims to settings where A2/A3 hold or by proving them under an explicit exploration scheme.","major_comments":[{"comment":"The ULLN in Lemma 2 is the basis for Corollary 3, Lemma 4, and Theorem 5, and it relies on Assumptions A2/A3 for the process generated under the agent's policy. In Algorithm 1, however, the policy is updated after every episode from accumulated data and is then played greedily, so the action at time t depends on the whole history. For such an adaptive policy sequence, the Cesaro limit in A2 need not exist, the sets T_a may fail to grow for actions that become suboptimal once the policy converges, and conditioning on the event A_t=a can destroy the mixing property even when the environment M0 is Markovian. The statement following Lemma 2 that these properties hold automatically for the Markovian model M0 is therefore too strong. No verification of A2/A3 is supplied for the chain, pendulum, or Hopper experiments, so Theorem 5 is not established for the episodic adaptive setting that the algorithm actually implements.","section":"Section 3.2, Assumptions A2/A3 and Lemma 2"},{"comment":"The proof of Theorem 6 defines Y_ki = ||theta_ki - hat_theta_k|| + ||hat_theta_k - theta*|| and claims that (1/n) sum_{i=1}^n |Y_ki| = O_p((n k tau)^{-1/2}). This does not follow from the BvM result: each Y_ki is nonnegative and of stochastic order (k tau)^{-1/2}, so averaging n such terms does not reduce the order below the mean unless the terms are centered at zero, which they are not. Thus the claimed sqrt(n) improvement of ETS over classical TS is not proven by this argument. In addition, the assertion that the policy-iteration difference terms ||Q^{theta*}_{mu*_j} - Q^{theta*}_{mu_j}|| and ||Q^{theta*}_{mu*_{j+1}} - Q^{theta*}_{mu_{j+1}}|| vanish 'from the consistency of the posterior mean' requires that the greedy policies under Q^{theta*} and Q^{theta(k)} eventually coincide; this stability property does not follow from the BvM theorem alone and is not proved in the appendix.","section":"Appendix B, Eq. (26) and following paragraph"},{"comment":"The text states that Theorem 5 and Theorem 6 'assume an unimodal posterior for model parameters'. Global unimodality is not among assumptions A1-A10, and a Bernstein-von Mises theorem is a local result; if unimodality is intended as an additional condition, it must be stated where the theorems are proved. More importantly, the misspecified inverted-pendulum experiment and the Hopper experiment use generative neural networks whose posteriors the authors themselves describe as possibly multimodal, so the demonstrations involving deep generative models are not covered by the conditions under which Theorem 5 and Theorem 6 are claimed. This limitation should be made explicit in the theorem statements or in the theory section.","section":"Section 5.1, misspecified paragraph; Theorem 5 assumptions"},{"comment":"The proof of Theorem 5 is a sketch that imports Theorems 4 and 6 of Miller (2021). This is acceptable only if the imported hypotheses are verified. In particular, Assumption A8 requires PS''_T(theta*) to converge to a positive definite H*, and Assumption A10 requires a uniform separation of the empirical prequential score outside shrinking neighborhoods of hat_theta_T; these are substantive conditions on a simulator-based loss and are neither derived from the assumed generative model class nor checked in the experiments. The statement that q_T converges 'almost surely' also needs a precise formulation, since q_T is the density of a random probability measure and the total-variation convergence should be phrased as almost sure convergence of that random measure under M0, matching the formulation in Miller (2021).","section":"Appendix A.5, Theorem 5 proof"}],"minor_comments":[{"comment":"The notation '1/|T_a| G^a_t converges weakly' is ambiguous; the intended statement is that the Cesaro average (1/|T_a|) sum_{t in T_a} G^a_t converges weakly to G^a, and the condition |T_a| -> infinity should be stated explicitly.","section":"Section 3.2, Assumption A2"},{"comment":"There is a typo in 'pospteerior', and in the normalizing-constant formula exp(-PS_T(theta_T)) the symbol theta_T appears to be a typo for hat_theta_T.","section":"Theorem 5 statement"},{"comment":"In Eq. (19) the notation gPS(theta) is missing the subscript T, and the bound epsilon T / 2 should be justified with the T-normalization made explicit when it is used in the following display.","section":"Appendix A.3, Eq. (19)"},{"comment":"The zeroth-order gradient formula appears to contain a typo: the second bracket should be [U(theta + mu z_i) - U(theta)] rather than [U(theta + mu z_i) - f(theta)].","section":"Appendix C.2"},{"comment":"The Lipschitz constant K'_mu = max_{(s,a) in S x A} K_mu(s,a) is used without a uniform bound over the policies generated by policy iteration; a uniform bound over the relevant policy class is needed for the recursion to be meaningful.","section":"Appendix B"},{"comment":"The reference to Pacchiardi et al. (2024a) contains the typo 'Journal of Mahcine Learning Research', and the Hopper experiment would benefit from reporting the total number of episodes and the number of seeds used beyond the moving-average plot in Fig. 6.","section":"References and Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core relies heavily on Pacchiardi et al. (2024a,b) and Miller (2021), with the incremental contribution being the application to RL and the ETS policy. The adaptive-policy gap in Assumptions A2/A3 and the rate claim in Appendix B are substantial enough that I would ask for a careful revision before considering publication. If the authors prefer to retain the empirical claims, they should either verify A2/A3 for an explicit exploration strategy or state the theorems for fixed or stationary policies and present the episodic adaptive algorithm as a heuristic extension. I see no circularity issue; the imports are properly credited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my read of the generalized Bayesian deep RL paper. The core idea is nice: use prequential scoring rule posteriors to do likelihood-free Thompson sampling for deep generative dynamics models, and average Q-functions over multiple posterior samples (ETS) instead of one. The action-decomposed prequential score for MDPs is a sensible extension, and the ETS idea with the approximate policy-iteration bound is a genuine contribution, even if the bound is loose. The chain experiment convincingly shows that more posterior samples reduce regret, and the pendulum and hopper results support sample-efficiency gains over model-free baselines. So there is real substance here.\n\nThe soft spots are in the theory. The BvM theorem (Theorem 5) rests on assumptions A2/A3 — asymptotic stationarity and mixing of the state-action process under the agent's policy. The paper says these 'automatically hold' for Markovian M0, but that's not right: Algorithm 1 updates the policy every episode from accumulated data and then plays greedily. That makes the action process history-dependent, and the visited-state distribution need not stabilize or mix. If the policy converges to a deterministic one, some actions get sampled only finitely often, so the per-action Cesaro averages in A2 have no non-degenerate limit. The paper never verifies A2/A3 for the experiments, so the BvM statement isn't established for the adaptive episodic setting. The authors do flag that unimodality fails for their neural simulators, which is honest, but it means Theorem 5 isn't doing the work for the main application.\n\nTheorem 6 inherits this gap: the zeta_l(k,n) terms are claimed to shrink via Theorem 5, and Appendix B assumes without proof that greedy policies under Q^{theta*} and Q^{theta(k)} coincide for large k and n. That is a stability property, not a consequence of BvM. The sqrt(n) improvement claim from averaging is also not demonstrated rigorously in Appendix B. So the theory is conditional on assumptions that are stated but not delivered.\n\nAll that said, this is not a desk-reject. The framework is well-motivated, the empirical study is informative, and the gaps are localized. A serious referee could push the authors to either verify or replace A2/A3 for their algorithm, and to tighten the policy-iteration argument. Lack of code is a reproducibility issue but not fatal for a methods paper.\n\nRecommendation: send to peer review, with a request for major revision. I'd bring it to a reading group if someone is working on likelihood-free RL; I wouldn't cite it in my own work until the theory is fixed.","headline":"Promising likelihood-free Thompson sampling framework, but the BvM and ETS bounds rest on stationarity/mixing assumptions that the adaptive greedy-policy algorithm doesn't guarantee.","tokens_in":27136,"tokens_out":2183,"would_cite":false,"duration_ms":29239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that Thompson sampling works for model-based RL with deep generative dynamics and no likelihood, via a prequential scoring-rule posterior that provably concentrates and turns Gaussian, plus a bound for expected…","keywords":["generalized Bayesian inference","prequential scoring rule","likelihood-free reinforcement learning","expected Thompson sampling","sequential Monte Carlo","deep generative models","Bernstein-von Mises theorem","Markov decision process"],"falsifier":"Run a known parametric MDP with the prequential energy-score posterior while updating the policy every episode, and check whether the empirical distribution of states visited under each action converges as episodes accumulate and whether $\\sqrt{T}(\\theta-\\hat\\theta_T)$ is approximately Gaussian around the true parameter; if the visited-state distribution keeps drifting under a changing policy, the mixing and stationarity assumptions fail and the Bernstein–von Mises conclusion should not hold.","tokens_in":26032,"feed_emoji":"🤖","tokens_out":8766,"duration_ms":74856,"temperature":0.7,"pith_summary":"Bayesian reinforcement learning normally needs a tractable likelihood for the environment's dynamics, which rules out the deep generative models used for complex tasks. This paper proposes a generalized posterior for a parametric MDP based on the prequential (predictive-sequential) scoring rule, replacing the unknown one-step transition likelihood with a strictly proper scoring rule that can be estimated purely from simulations. The central theoretical contribution is a Bernstein–von Mises theorem for this posterior: under stationarity, mixing, and regularity conditions, the posterior concentrates at the expected-score minimizer and the scaled parameter error becomes Gaussian. The paper also introduces expected Thompson sampling (ETS), which averages Q-functions over many posterior samples rather than one, and proves a bound showing its value-approximation error shrinks as posterior samples and episodes increase. If these results hold, likelihood-free Bayesian model-based RL becomes principled for deep generative dynamics, with empirical support on inverted-pendulum and Hopper tasks.","feed_headline":"Prequential scores put Thompson sampling on a provable footing","feed_subtitle":"No likelihood needed: the posterior provably tightens around the true dynamics, and ETS learns faster.","key_machinery":"The load-bearing object is the prequential scoring-rule posterior, formed by exponentiating the negative cumulative one-step-ahead score instead of the log-likelihood; with the log score it reduces exactly to the usual likelihood posterior. The paper uses the energy score, a strictly proper scoring rule that can be estimated unbiasedly from simulator draws, and decomposes the prequential score by action to obtain a uniform law of large numbers under mixing assumptions. Posterior sampling uses sequential Monte Carlo with an adjusted stochastic-gradient Riemannian Langevin dynamics kernel, which handles the unbiased but noisy score gradients and scales to neural-network parameter spaces. Expected Thompson sampling then replaces the single posterior draw of classical Thompson sampling with an average of Q-functions over $n$ posterior samples, reducing estimation variance and, via the Bernstein–von Mises theorem, making the Q-approximation error contract.","core_discovery":"On its own terms, the paper's claim is that the prequential scoring rule posterior $$\\pi_{\\mathrm{PS}}(\\$\\theta$\\,|\\,h_T) \\propto \\pi(\\$\\theta$)\\exp\\{-\\mathrm{PS}_T(\\$\\theta$)\\},$$ with $\\mathrm{PS}_T(\\theta)=\\sum_{t=1}^{T} S(P_\\theta(\\cdot|s_t,a_t),s_{t+1})$ for a strictly proper scoring rule $S$, is a sound replacement for the intractable likelihood posterior of an MDP. Theorem 5 states that, under assumptions A1–A10, the minimizer $\\hat\\theta_T$ of the prequential score is consistent for $\\theta^*$, the posterior concentrates at $\\theta^*$, and the density of $\\sqrt{T}(\\theta-\\hat\\theta_T)$ converges in total variation to $\\mathcal{N}(0,H_*^{-1})$; $H_*$ plays the role of the Fisher information matrix. Theorem 6 bounds the ETS policy-iteration error by $\\gamma^j\\|Q^*-Q^{\\theta^{(k)}}_{\\mu_1}\\|_\\infty + \\sum_{l=1}^j \\gamma^{j-l+1}\\zeta_l(k,n)$, where $\\zeta_l(k,n)$ vanishes as episodes $k$ and posterior samples $n$ grow. The paper further shows empirically that ETS with a prequential posterior learns a balancing policy faster than model-free LSPI on a well-specified and a misspecified inverted pendulum, and that ETS-integrated REINFORCE outperforms model-free REINFORCE on the Hopper benchmark.","pith_inferences":["Reading from the paper's assumptions, the mixing and asymptotic-stationarity conditions (A2–A3) are not automatic when policies are updated every episode from accumulated data, and they are not checked in the pendulum and Hopper experiments; the Bernstein–von Mises guarantee should be read as holding for a stabilized or sufficiently slowly adapting policy unless verified.","When the generative model is misspecified, $\\theta^*$ is the best one-step-ahead predictive proxy rather than the true dynamics; ETS will then explore around a proxy that may be poor for long-horizon planning, so a fair comparison would need to measure regret against the true environment rather than only against the fitted model.","A testable extension is to compute the effective sample size of the SMC particles and compare regret across the number of posterior samples $n$; if the posterior is multimodal or poorly mixed, increasing $n$ may not reduce the ETS error as Theorem 6 predicts.","The continuous-action ETS extension with REINFORCE is empirical; an actor-critic variant using the same prequential posterior could make the sample-efficiency gain available to high-dimensional action spaces."],"forward_implications":["A generative neural network for environment dynamics can be used for principled Bayesian exploration without a likelihood, since only forward simulation and score gradients are needed.","As interaction length grows, the generalized posterior behaves like a Gaussian centered at the prequential-score minimizer, so posterior uncertainty quantification and Thompson-style exploration inherit familiar asymptotic guarantees.","Averaging Q-functions over $n$ posterior samples cuts the posterior-sampling component of the value-approximation error by a factor of about $\\sqrt{n}$ relative to single-sample Thompson sampling.","In the well-specified discrete-action case, ETS policy iteration contracts toward the optimal Q-function as episodes and samples grow.","In the misspecified and continuous-action cases studied, ETS improves sample efficiency over model-free baselines, though without the paper's theoretical guarantees."],"supporting_citations":[{"why":"Introduces prequential scoring rules for training generative forecasting models and supplies the construction of the prequential score posterior used here.","marker":"Pacchiardi et al. (2024a)"},{"why":"Establishes the scoring-rule posterior and its contraction and robustness properties, and gives the unbiased gradient estimation used inside SMC.","marker":"Pacchiardi et al. (2024b)"},{"why":"Provides the general framework of coherent generalized Bayesian updating that motivates defining a posterior from a loss instead of a likelihood.","marker":"Bissiri et al. (2016)"},{"why":"Supplies the sequential Monte Carlo sampler machinery used to move particles between episodic posteriors.","marker":"Del Moral et al. (2006)"},{"why":"Gives the generalized-posterior asymptotic normality theorems from which the paper's Bernstein–von Mises result is derived.","marker":"Miller (2021)"},{"why":"Defines strictly proper scoring rules and the energy score, the scoring rule used in the prequential posterior.","marker":"Gneiting and Raftery (2007)"},{"why":"Provides the uniform law of large numbers for dependent and heterogeneous data used to prove Lemma 2.","marker":"Pötscher and Prucha (1989)"},{"why":"Provides the prequential consistency theorem used to prove Lemma 4.","marker":"Skouras (1998)"},{"why":"The ABC-based Thompson sampling baseline that this work extends to high-dimensional generative models.","marker":"Dimitrakakis and Tziortziotis (2013)"}],"fun_headline_variants":["Prequential scores tighten posteriors in model-based RL","Expected Thompson sampling provably beats single-sample TS","Generalized Bayesian deep RL needs no likelihood, only scores","SMC-based prequential posterior for scalable deep RL","ETS learns faster: provable advantage over Thompson sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, under the agent's actual adaptive policy, the sequence of visited states becomes stationary and mixing fast enough with bounded moments; if the policy keeps changing as data accumulate, that is not automatic and is never verified in the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Prequential scores tighten posteriors in model-based RL","Expected Thompson sampling provably beats single-sample TS","Generalized Bayesian deep RL needs no likelihood, only scores","SMC-based prequential posterior for scalable deep RL","ETS learns faster: provable advantage over Thompson sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1891,"prompt_tokens":1134,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":679}},"tokens_in":750,"tokens_out":757,"duration_ms":7991,"temperature":1.0,"reasoning_tokens":679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:37:32.822382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a known parametric MDP with the prequential energy-score posterior while updating the policy every episode, and check whether the empirical distribution of states visited under each action converges as episodes accumulate and whether $\\sqrt{T}(\\theta-\\hat\\theta_T)$ is approximately Gaussian around the true parameter; if the visited-state distribution keeps drifting under a changing policy, the mixing and stationarity assumptions fail and the Bernstein–von Mises conclusion should not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the generalized-posterior asymptotic normality theorems from which the paper's Bernstein–von Mises result is derived."}],"review_version":1}