{"id":"7ec6c129-2422-4af9-89cc-a13c8d734002","arxiv_id":"1908.09381","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A tutorial survey re-deriving known PGM and VI methods in deep RL, with incorrect key derivations and overstated novelty.","lead":"This paper is a tutorial survey that re-derives existing probabilistic graphical model and variational inference methods in deep reinforcement learning and organizes them into a taxonomy. It is unlikely to change practice because several key derivations contain errors and the claimed novelty over prior tutorials is overstated.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (36), the central recursion behind the control-as-inference section, silently drops the action prior p(a|s), so the claimed 'detailed derivation' is not valid as displayed.","rationale":"Good-faith reading: this is a tutorial and survey, not an original method, so its value depends on correct, self-contained derivations. The reader's reject verdict is justified, but my strongest concern is not the exponential-tilting premise itself (a standard import from Levine's tutorial) nor Eq. (1), whose algebra can be made consistent with the usual ELBO definitions. The decisive issue is a dropped factor in the central recursion. The paper's own text invites this check: it claims Eq. (36) is derived in detail because [13] only states the conclusion. However, the displayed derivation omits p(a_{t+1}|s_{t+1}) in an intermediate factorization. Without that factor, the mass of the future action is never accounted for, and the recursion holds only after restoring the missing integration over a_{t+1}. This is not a cosmetic typo: Eq. (35) defines β with an explicit action-prior factor, and Eq. (40) is advertised as the derivation of the soft Bellman backup. A tutorial containing a false equality in its flagship derivation cannot be trusted as complementary material. Therefore I agree with the reject verdict and recommend no change; the paper requires a corrected derivation, a consistently defined β, and an explicit treatment of the uniform-prior assumption (discrete versus improper continuous) before it can serve its stated purpose.","tokens_in":14221,"tokens_out":10885,"duration_ms":99279,"concrete_test":"Evaluate Eq. (36) on a 2-state, 2-action tabular MDP with a non-uniform action prior p(a|s), fixed transitions and rewards, and horizon T=2. Compute p(O_{t:T}=1|s_t,a_t) by exact enumeration of the graph and compare with the printed recursion. The enumeration will match only after replacing the factor p(s_{t+1}|s_t,a_t) by p(s_{t+1}|s_t,a_t)p(a_{t+1}|s_{t+1}) and integrating over a_{t+1}. If the printed equality is used literally, the values differ, confirming that the derivation is not reliable as a tutorial.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that it supplies reliable detailed derivations complementing the original references. The derivation with the most weight is the backward-message recursion, Eq. (36), because it connects the posterior p(a_t|s_t,O_{t:T}=1) to the soft Bellman backup used for all later soft-Q results. In the displayed chain, the term p(O_{t+1:T},s_{t+1},a_{t+1}|s_t,a_t) is factorized as p(O_{t+1:T}|s_{t+1},a_{t+1}) p(s_{t+1}|s_t,a_t), omitting the action density p(a_{t+1}|s_{t+1}). The graph in Fig. 7 includes this factor, and the β defined in Eq. (35) explicitly includes p(a_t|s_t). The final line can be recovered only if the omitted a_{t+1} integral is restored and β(s_{t+1}) is understood to contain p(a_{t+1}|s_{t+1}). As printed, an intermediate equality in Eq. (36) is false. Since the text advertises this recursion as a derivation 'not available in [13]', the invalid factorization directly undermines the central pedagogical claim. I do not rely on the Eq. (1) criticism; that identity is algebraically salvageable with the standard ELBO definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a tutorial/survey aiming to connect probabilistic graphical models (PGMs) and variational inference to deep reinforcement learning. It reviews RL basics, provides PGM representations for MDPs and POMDPs, discusses undirected-graph policy models (e.g., restricted Boltzmann machines), develops control-as-inference with an auxiliary optimality variable, derives soft-Q/energy-based policies, and covers variational approaches to environment modeling (VIME and VaST). The stated contribution is to supply detailed derivations for equations that the authors claim are not available in the original references, and to organize these methods into a taxonomy.","tokens_in":14463,"tokens_out":11084,"duration_ms":104681,"significance":"If the derivations were reliable, the paper could serve as a useful complementary tutorial for readers of Levine's control-as-inference tutorial and for the papers on VIME, VaST, and soft Q-learning. The attempt to recast recent deep RL methods as PGMs and to prove auxiliary formulas (e.g., the soft-policy improvement identity) is a legitimate pedagogical contribution. However, the value of the paper rests entirely on the correctness of those derivations, and at present several load-bearing equations are incorrect or malformed as displayed. The paper contains no machine-checked proofs or reproducible code, so the reader must rely on the displayed mathematics.","major_comments":[{"comment":"The backward-message recursion is the paper's central advertised derivation, but the displayed chain contains a false equality. In passing from the joint conditional p(O_{t+1:T}, s_{t+1}, a_{t+1} | s_t, a_t) to the product p(O_{t+1:T}|s_{t+1},a_{t+1}) p(s_{t+1}|s_t,a_t), the factor p(a_{t+1}|s_{t+1}) is silently dropped. This factor is present in Figure 7 and in the definition of beta in Eq. (35). The final line of Eq. (36) can be recovered only if the a_{t+1} integral is retained and beta(s_{t+1}) is understood to include p(a_{t+1}|s_{t+1}); as printed, an intermediate equality is false. Because this recursion is offered as the derivation 'not available in [13]', the error directly undermines the paper's central pedagogical claim.","section":"III-B2, Eq. (36)"},{"comment":"The ELBO identity, which the paper presents as a contribution, is garbled. The chain includes terms such as -D_KL(q_phi(z|x) || p(x,z)) and H_q(p) that are not defined, and the expression '-F(phi,theta)+H_q(p)-H(q)' is not a transparent consequence of the previous line. The standard identity log p(x) = ELBO(phi,theta) + D_KL(q_phi(z|x) || p(z|x)) should be written out with the usual definitions of the ELBO and of the free energy. As a foundational section of a tutorial, the present display will mislead readers.","section":"I-B, Eq. (1)"},{"comment":"The trajectory likelihood in Eq. (34) omits the action prior p(a_t|s_t) that appears in Eq. (35) and in Figure 7. If p(a_t|s_t) is intended to be uniform, that should be stated explicitly; otherwise the 'evidence' is inconsistent with the posterior computation that follows. Since the equivalence between control and inference rests on this likelihood, the omission is load-bearing.","section":"III-B1, Eq. (34)"},{"comment":"The exponential-tilting likelihood p(O_t=1|s_t,a_t)=exp(r(s_t,a_t)) and the identification of the optimal policy with the posterior p(a_t|s_t,O_{t:T}=1) are imported from [13] without derivation. In a paper that advertises itself as a 'self-inclusive tutorial', this is a missing derivation; at minimum, the assumptions (bounded, negative rewards; uniform action prior) should be stated before Eq. (34) is used.","section":"III-B, text before Eq. (35)"}],"minor_comments":[{"comment":"In Eq. (7), the conditioning should be S_t=s, not 'forall S_t=s'; after Eq. (8), the line 'Aact_{t+i} ~ pi(a|S_{t+i+1})' should read 'S_{t+i}'.","section":"II-A, Eq. (7) and text after Eq. (8)"},{"comment":"The displayed line involving the log-partition function is malformed; it should read log[ exp(Q(s,a)) / (int exp(Q(s,a')) da') ] - log int exp(Q(s,a')) da' in the intermediate line, before the final D_KL expression. As printed, the line does not parse.","section":"III-B5, Eq. (48)"},{"comment":"The table caption and column meanings are unclear: the 'var' column mixes 'na', conditional densities, and unspecified symbols, and the 'P' and 'standalone' columns are not explained in the text.","section":"Table I"},{"comment":"The factorized variational posterior q(S_{0:T}|O_{0:T}) = prod_t q_phi(S_t|O_{t-k:t}) is stated without discussing the edge cases at t<k or the treatment of the final time step; this should be clarified for a tutorial.","section":"III-C2, Eq. (54)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for publication in its current form. The central derivation errors are local and correctable, so I recommend major revision rather than rejection. The editor may also wish to check the citation pattern: several self-citations (e.g., [3], [9], [14], [22], [25]) appear in places where a standard reference would be more appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a survey/tutorial that promises detailed derivations to complement Levine's control-as-inference tutorial, but the central derivations contain errors. I wouldn't hand it to a student in the current form.\n\nWhat's actually here: the paper organizes a taxonomy of PGM/VI methods in deep RL, gives PGM diagrams for VIME and VaST that aren't in the original papers, and tries to fill in derivation gaps. The coverage is broad and the writing is reasonably accessible. If you need a one-stop pointer to the literature, the reference list is fine.\n\nThe soft spots are where it matters. Equation (36), the backward-message recursion, factorizes p(O_{t+1:T},s_{t+1},a_{t+1}|s_t,a_t) as p(O_{t+1:T}|s_{t+1},a_{t+1}) p(s_{t+1}|s_t,a_t), dropping the action prior p(a_{t+1}|s_{t+1}). The graph in Figure 7 includes that factor, and the recursion only closes if you integrate over a_{t+1} with that prior. As printed, the intermediate equality is false. Equation (1) is also a garbled ELBO decomposition, with a malformed KL term. And the paper repeatedly claims derivations are 'not available in [13]' when Levine's tutorial actually has them; that claim should be corrected or softened.\n\nThese are not minor typos: the paper's stated contribution is reliable detailed derivations, and the two most important derivations in the control-as-inference section are wrong as displayed. The VIME derivation (Eqs. 49–52) looks fine to me, and the VaST ELBO is standard.\n\nWho's this for? Someone who wants a quick map of the area and is willing to check every equation against the primary sources. It's not trustworthy as a standalone tutorial.\n\nRecommendation: I would not send this to peer review in its current form. If the authors fix Equation (36), fix Equation (1), and temper the novelty claims, it could be a useful supplementary note. As is, I'd desk-reject.","headline":"A survey with a useful taxonomy but unreliable central derivations; useful as a pointer, not as a teaching text.","tokens_in":15025,"tokens_out":6008,"would_cite":false,"duration_ms":50237,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This tutorial derives deep reinforcement learning from probabilistic graphical models and variational inference.","keywords":["probabilistic graphical models","variational inference","deep reinforcement learning","control as inference","maximum entropy reinforcement learning","evidence lower bound","Bellman equation","exploration"],"falsifier":"Take a small stochastic MDP with all rewards shifted negative, compute the posterior policy $p(a_t|s_t,O_{t:T}=1)$ from the backward recursion, and compare its induced value function with the true optimal value function found by value iteration; a divergence at any state would falsify the claimed equivalence.","tokens_in":13991,"feed_emoji":"🤖","tokens_out":7665,"duration_ms":65431,"temperature":0.7,"pith_summary":"This tutorial survey sets out to make deep reinforcement learning readable as a branch of probabilistic inference rather than a collection of heuristics. It supplies graphical models for standard RL concepts and for several recent DRL methods, then derives equations the original papers left unproved: the backward-message recursion for the optimal-policy posterior, the risk-seeking Bellman backup that follows from it, and the ELBO objectives behind maximum-entropy and exploration methods. The intended payoff is a single taxonomy in which value functions, policies, and learned environment models are all inference quantities on graphs. A reader of the tutorial should be able to reproduce the derivations instead of taking them on faith.","feed_headline":"Tutorial rewrites deep RL as inference on probabilistic graphs","feed_subtitle":"Step-by-step derivations tie value functions, policies, and exploration to variational inference","key_machinery":"The central mechanism is an auxiliary optimality variable $O_t$ with likelihood $p(O_t=1|s_t,a_t)=\\exp(r(s_t,a_t))$ on shifted rewards, paired with the backward message $\\beta(a_t,s_t)=p(O_{t:T}=1|a_t,s_t)p(a_t|s_t)$. The paper derives a recursion for $\\beta$ and shows that $\\log\\beta(a_t,s_t)$ is a value-style quantity whose backup takes a softmax over next states, which is what produces the risk-seeking behavior and, under a variational approximation, the maximum-entropy family of objectives. This mechanism is what carries the directed-graph branch of the taxonomy and links it to both value-based RL and exploration bonuses.","core_discovery":"The paper's central claim is that the variety of deep reinforcement learning algorithms can be organized into a small number of graphical-model moves, and that the critical equations of each family can be derived in full. In the undirected-graph branch, the negative free energy of a Restricted Boltzmann Machine stands in for the state-action value function and its conditional distribution over actions serves as the policy. In the directed-graph branch, an auxiliary binary optimality variable $O_t$ with likelihood $p(O_t=1|s_t,a_t)=\\exp(r(s_t,a_t))$ turns the optimal policy into the posterior $p(a_t|s_t,O_{t:T}=1)$, and the paper derives the backward-message recursion that computes this posterior and shows its logarithm is a risk-seeking Bellman backup. In the environment-model branch, variational inference targets either the transition distribution or the latent-state posterior, yielding intrinsic rewards and VAE-style ELBO objectives. Across all branches, the paper's contribution is the derivations and the unified picture that connects them.","pith_inferences":["If the taxonomy is right, a natural testable extension would be to derive new algorithms by swapping graph structure, such as adding memory or hierarchical variables, and reading off the resulting Bellman or ELBO update.","The exponential-tilting assumption suggests a family of alternative optimality likelihoods, for example tempered or bounded reward transforms, that would interpolate between risk-seeking and standard expected-return backups; the paper does not explore this.","The unified derivations imply that differences among algorithm families may hinge more on which inference approximation is chosen than on the underlying objective, an explicit comparison the paper leaves to future work."],"forward_implications":["Readers can reproduce derivations that previous papers left as exercises, making the tutorial a self-contained bridge for beginners and researchers entering the area.","The message-passing recursion for the optimal-policy posterior yields a specific Bellman-style backup that is risk-seeking, so the taxonomy predicts when exact control-as-inference will diverge from standard RL.","Through the variational approximation, maximum-entropy objectives such as reward-plus-entropy arise naturally rather than as heuristics.","Environment-model variational methods, including intrinsic rewards from conditional mutual information and latent-state tabulation via a VAE-like ELBO, fit into the same inference framework and can be combined with base RL algorithms."],"supporting_citations":[{"why":"Supplies the control-as-inference equivalence and the risk-seeking backup observation that this paper re-derives in detail.","marker":"[13]"},{"why":"Introduces the Restricted Boltzmann Machine value-function-and-policy construction that anchors the undirected-graph branch.","marker":"[27]"},{"why":"Presents deep energy-based policies and a soft Bellman update whose key inequality this paper proves.","marker":"[28]"},{"why":"Proposes variational information maximizing exploration, the source of the intrinsic-reward derivation.","marker":"[4]"},{"why":"Introduces variational state tabulation and the VAE-style ELBO that the paper formalizes with an explicit graphical model.","marker":"[5]"},{"why":"Establishes the maximum-entropy actor-critic family that the variational policy objective connects to.","marker":"[29]"},{"why":"Provides the variational autoencoder and reparameterization machinery used in the latent-state branch.","marker":"[31]"}],"fun_headline_variants":["Graphical models unify deep RL's algorithm zoo","Deep RL as inference: a full derivation tutorial","One framework to derive all deep RL methods","Probabilistic graphs explain deep RL's design space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that optimality can be encoded as $p(O_t=1|s_t,a_t)=\\exp(r(s_t,a_t))$ with all rewards negative, so that the optimal policy is exactly the posterior over actions given that all future steps are optimal; if that equivalence fails, the directed-graph derivation does not describe optimal control.","fun_headline_variants_meta":{"raw":{"variants":["Graphical models unify deep RL's algorithm zoo","Deep RL as inference: a full derivation tutorial","One framework to derive all deep RL methods","Probabilistic graphs explain deep RL's design space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1295,"prompt_tokens":804,"completion_tokens":491,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":420,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":420,"tokens_out":491,"duration_ms":5185,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:13:07.460427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small stochastic MDP with all rewards shifted negative, compute the posterior policy $p(a_t|s_t,O_{t:T}=1)$ from the backward recursion, and compare its induced value function with the true optimal value function found by value iteration; a divergence at any state would falsify the claimed equivalence.","supporting_citations":[{"cited_title":"Reinforcement learning with factored states and actions,","cited_arxiv_id":null,"evidence_quote":"Introduces the Restricted Boltzmann Machine value-function-and-policy construction that anchors the undirected-graph branch."},{"cited_title":"Reinforcement learning with deep energy-based policies,","cited_arxiv_id":null,"evidence_quote":"Presents deep energy-based policies and a soft Bellman update whose key inequality this paper proves."},{"cited_title":"Vime: Variational information maximizing exploration,","cited_arxiv_id":null,"evidence_quote":"Proposes variational information maximizing exploration, the source of the intrinsic-reward derivation."},{"cited_title":"Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation","cited_arxiv_id":"1802.04325","evidence_quote":"Introduces variational state tabulation and the VAE-style ELBO that the paper formalizes with an explicit graphical model."}],"review_version":1}