Pith. sign in

REVIEW 4 major objections 4 minor 32 references

Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This tutorial derives deep reinforcement learning from probabilistic graphical models and variational inference.

desk verdict A survey with a useful taxonomy but unreliable central derivations; useful as a pointer, not as a teaching text. read the letter →

arxiv 1908.09381 v5 pith:LYIPUVHP submitted 2019-08-25 cs.LG cs.AIstat.ML

Xudong Sun , Bernd Bischl This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords probabilisticgraphicalmodelsvariationalinferencedeepreinforcementlearningcontrolasmaximumentropyevidencelowerboundBellmanequationexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This tutorial survey sets out to make deep reinforcement learning readable as a branch of probabilistic inference rather than a collection of heuristics. It supplies graphical models for standard RL concepts and for several recent DRL methods, then derives equations the original papers left unproved: the backward-message recursion for the optimal-policy posterior, the risk-seeking Bellman backup that follows from it, and the ELBO objectives behind maximum-entropy and exploration methods. The intended payoff is a single taxonomy in which value functions, policies, and learned environment models are all inference quantities on graphs. A reader of the tutorial should be able to reproduce the derivations instead of taking them on faith.

What carries the argument

The central mechanism is an auxiliary optimality variable $O_t$ with likelihood $p(O_t=1|s_t,a_t)=\exp(r(s_t,a_t))$ on shifted rewards, paired with the backward message $\beta(a_t,s_t)=p(O_{t:T}=1|a_t,s_t)p(a_t|s_t)$. The paper derives a recursion for $\beta$ and shows that $\log\beta(a_t,s_t)$ is a value-style quantity whose backup takes a softmax over next states, which is what produces the risk-seeking behavior and, under a variational approximation, the maximum-entropy family of objectives. This mechanism is what carries the directed-graph branch of the taxonomy and links it to both value-based RL and exploration bonuses.

What would settle it

Take a small stochastic MDP with all rewards shifted negative, compute the posterior policy $p(a_t|s_t,O_{t:T}=1)$ from the backward recursion, and compare its induced value function with the true optimal value function found by value iteration; a divergence at any state would falsify the claimed equivalence.

Watch

Extended reading notes

Core claim

The paper's central claim is that the variety of deep reinforcement learning algorithms can be organized into a small number of graphical-model moves, and that the critical equations of each family can be derived in full. In the undirected-graph branch, the negative free energy of a Restricted Boltzmann Machine stands in for the state-action value function and its conditional distribution over actions serves as the policy. In the directed-graph branch, an auxiliary binary optimality variable $O_t$ with likelihood $p(O_t=1|s_t,a_t)=\exp(r(s_t,a_t))$ turns the optimal policy into the posterior $p(a_t|s_t,O_{t:T}=1)$, and the paper derives the backward-message recursion that computes this posterior and shows its logarithm is a risk-seeking Bellman backup. In the environment-model branch, variational inference targets either the transition distribution or the latent-state posterior, yielding intrinsic rewards and VAE-style ELBO objectives. Across all branches, the paper's contribution is the derivations and the unified picture that connects them.

Load-bearing premise

The load-bearing premise is that optimality can be encoded as $p(O_t=1|s_t,a_t)=\exp(r(s_t,a_t))$ with all rewards negative, so that the optimal policy is exactly the posterior over actions given that all future steps are optimal; if that equivalence fails, the directed-graph derivation does not describe optimal control.

Editorial extensions

If this is right

  • Readers can reproduce derivations that previous papers left as exercises, making the tutorial a self-contained bridge for beginners and researchers entering the area.
  • The message-passing recursion for the optimal-policy posterior yields a specific Bellman-style backup that is risk-seeking, so the taxonomy predicts when exact control-as-inference will diverge from standard RL.
  • Through the variational approximation, maximum-entropy objectives such as reward-plus-entropy arise naturally rather than as heuristics.
  • Environment-model variational methods, including intrinsic rewards from conditional mutual information and latent-state tabulation via a VAE-like ELBO, fit into the same inference framework and can be combined with base RL algorithms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the taxonomy is right, a natural testable extension would be to derive new algorithms by swapping graph structure, such as adding memory or hierarchical variables, and reading off the resulting Bellman or ELBO update.
  • The exponential-tilting assumption suggests a family of alternative optimality likelihoods, for example tempered or bounded reward transforms, that would interpolate between risk-seeking and standard expected-return backups; the paper does not explore this.
  • The unified derivations imply that differences among algorithm families may hinge more on which inference approximation is chosen than on the underlying objective, an explicit comparison the paper leaves to future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper is a tutorial/survey aiming to connect probabilistic graphical models (PGMs) and variational inference to deep reinforcement learning. It reviews RL basics, provides PGM representations for MDPs and POMDPs, discusses undirected-graph policy models (e.g., restricted Boltzmann machines), develops control-as-inference with an auxiliary optimality variable, derives soft-Q/energy-based policies, and covers variational approaches to environment modeling (VIME and VaST). The stated contribution is to supply detailed derivations for equations that the authors claim are not available in the original references, and to organize these methods into a taxonomy.

Significance. If the derivations were reliable, the paper could serve as a useful complementary tutorial for readers of Levine's control-as-inference tutorial and for the papers on VIME, VaST, and soft Q-learning. The attempt to recast recent deep RL methods as PGMs and to prove auxiliary formulas (e.g., the soft-policy improvement identity) is a legitimate pedagogical contribution. However, the value of the paper rests entirely on the correctness of those derivations, and at present several load-bearing equations are incorrect or malformed as displayed. The paper contains no machine-checked proofs or reproducible code, so the reader must rely on the displayed mathematics.

major comments (4)
  1. [III-B2, Eq. (36)] The backward-message recursion is the paper's central advertised derivation, but the displayed chain contains a false equality. In passing from the joint conditional p(O_{t+1:T}, s_{t+1}, a_{t+1} | s_t, a_t) to the product p(O_{t+1:T}|s_{t+1},a_{t+1}) p(s_{t+1}|s_t,a_t), the factor p(a_{t+1}|s_{t+1}) is silently dropped. This factor is present in Figure 7 and in the definition of beta in Eq. (35). The final line of Eq. (36) can be recovered only if the a_{t+1} integral is retained and beta(s_{t+1}) is understood to include p(a_{t+1}|s_{t+1}); as printed, an intermediate equality is false. Because this recursion is offered as the derivation 'not available in [13]', the error directly undermines the paper's central pedagogical claim.
  2. [I-B, Eq. (1)] The ELBO identity, which the paper presents as a contribution, is garbled. The chain includes terms such as -D_KL(q_phi(z|x) || p(x,z)) and H_q(p) that are not defined, and the expression '-F(phi,theta)+H_q(p)-H(q)' is not a transparent consequence of the previous line. The standard identity log p(x) = ELBO(phi,theta) + D_KL(q_phi(z|x) || p(z|x)) should be written out with the usual definitions of the ELBO and of the free energy. As a foundational section of a tutorial, the present display will mislead readers.
  3. [III-B1, Eq. (34)] The trajectory likelihood in Eq. (34) omits the action prior p(a_t|s_t) that appears in Eq. (35) and in Figure 7. If p(a_t|s_t) is intended to be uniform, that should be stated explicitly; otherwise the 'evidence' is inconsistent with the posterior computation that follows. Since the equivalence between control and inference rests on this likelihood, the omission is load-bearing.
  4. [III-B, text before Eq. (35)] The exponential-tilting likelihood p(O_t=1|s_t,a_t)=exp(r(s_t,a_t)) and the identification of the optimal policy with the posterior p(a_t|s_t,O_{t:T}=1) are imported from [13] without derivation. In a paper that advertises itself as a 'self-inclusive tutorial', this is a missing derivation; at minimum, the assumptions (bounded, negative rewards; uniform action prior) should be stated before Eq. (34) is used.
minor comments (4)
  1. [II-A, Eq. (7) and text after Eq. (8)] In Eq. (7), the conditioning should be S_t=s, not 'forall S_t=s'; after Eq. (8), the line 'Aact_{t+i} ~ pi(a|S_{t+i+1})' should read 'S_{t+i}'.
  2. [III-B5, Eq. (48)] The displayed line involving the log-partition function is malformed; it should read log[ exp(Q(s,a)) / (int exp(Q(s,a')) da') ] - log int exp(Q(s,a')) da' in the intermediate line, before the final D_KL expression. As printed, the line does not parse.
  3. [Table I] The table caption and column meanings are unclear: the 'var' column mixes 'na', conditional densities, and unspecified symbols, and the 'P' and 'standalone' columns are not explained in the text.
  4. [III-C2, Eq. (54)] The factorized variational posterior q(S_{0:T}|O_{0:T}) = prod_t q_phi(S_t|O_{t-k:t}) is stated without discussing the edge cases at t<k or the treatment of the final time step; this should be clarified for a tutorial.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the tutorial's derivations are self-contained expansions of standard definitions, and the self-citations are peripheral framing rather than load-bearing evidence.

full rationale

This is a tutorial and survey, not an empirical paper: it makes no predictions and fits no parameters, so the classic circularity patterns (fitted input called prediction, defined quantity used to predict itself) do not apply. The central derivation chain in Section III-B imports the control-as-inference construction p(O_t=1|s_t,a_t)=exp(r(s_t,a_t)) from Levine's external tutorial [13] and from the deep energy-based policy paper [28]; this is an acknowledged modeling assumption rather than a conclusion derived from the paper's own inputs, and it is not circular because it does not presuppose the soft Bellman result it is used to obtain. The paper's own derivations, such as the policy-gradient identity in Equations (25)-(29), the ELBO decomposition in Equations (43)-(45), and the VIME conditional-mutual-information identity in Equations (49)-(50), expand standard definitions and do not reduce to their own conclusions by construction. The paper does cite several works by its own authors ([3], [9], [14], [22], [25]), but these are descriptive examples from the survey portions (AutoML, variational resampling, multi-goal RL, recommender systems, Bayesian optimization) and none of them carries the weight of the variational-inference or graphical-model derivations. There is no invoked uniqueness theorem, no ansatz hidden behind a self-citation, and no renamed empirical pattern presented as a new unification. A separate mathematical concern exists: the displayed recursion in Equation (36) appears to omit the action factor p(a_{t+1}|s_{t+1}) in an intermediate factorization; that is a correctness issue in the exposition, not a circularity, because it does not make the conclusion equivalent to the premise. Overall, the paper is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fit in this tutorial; all quantities come from the cited methods. The key imported modelling choice is the optimality variable in Section III-B, listed as an axiom since it is assumed rather than introduced by this paper.

assumptions (4)
  • standard math Standard probability theory and graphical model semantics, including d-separation and conditional independence, are assumed.
    Used throughout Section I-B and II for deriving trajectory likelihoods and message passing.
  • domain assumption The environment is a Markov decision process with transition distribution p(s_{t+1}|s_t,a_t) and reward distribution p(r_t|s_t,a_t).
    Stated in Section II-A.2 for Equation (2) and used in all subsequent derivations.
  • domain assumption The optimal policy can be expressed as the posterior over actions given an auxiliary optimality variable O_t with p(O_t=1|s_t,a_t)=exp(r(s_t,a_t)), requiring rewards to be negative.
    Introduced in Section III-B.1 around Figure 7 and used to derive the message passing posterior in Equations (35)-(36).
  • domain assumption The variational family q_phi(z|x) is expressive enough to approximate the true posterior (e.g., the policy network class).
    Relied on in Sections III-B.4 and III-C for the variational approximations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/LYIPUVHP

@misc{pith2026190809381,
  author       = {Pith},
  title        = {Pith review of: Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYIPUVHP}},
  note         = {Machine review of arXiv:1908.09381}
}
read the original abstract

Aiming at a comprehensive and concise tutorial survey, recap of variational inference and reinforcement learning with Probabilistic Graphical Models are given with detailed derivations. Reviews and comparisons on recent advances in deep reinforcement learning are made from various aspects. We offer detailed derivations to a taxonomy of Probabilistic Graphical Model and Variational Inference methods in deep reinforcement learning, which serves as a complementary material on top of the original contributions.

Figures

Figures reproduced from arXiv: 1908.09381 by the authors.

Figure 2
Figure 2. Illustration of State, Action and Reward Trajectory [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Concept of Reinforcement Learning 1) RL Concepts, Terminology and Convention: As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Directed Acyclic Graph For Markov Decision Process [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Probabilistic Graphical Model for POMDP belief state can be updated in a Bayesian way in Equation (6). B. Value Function, Bellman Equation, Policy Iteration Define state value function of state s ∈ S in Equation (7), where the corresponding Bellman Equation is derived …
Figure 5
Figure 5. Figure 5: General Policy Iteration η(s) = X k=0 γ kP π (s0 → s, k + 1) (17) = h(s) +X s,a ¯ γη(¯s)πθ(a|s¯)P(s|s, a ¯ ) (18) Suppose all episodes start from an auxiliary initial state s0, which with probability h(s), jumps to different state s ∈ S without reward. h(s) characteriz…
Figure 7
Figure 7. Figure 7: Optimal Policy as posterior on actions: p(at|st, Ot:T = 1) update the state value function Q(s, a)’s estimation. Such an on-policy process has been shown to be empirically effective in the large state actions spaces [27]. B. Variational Inference on ”optimal” Policies …
Figure 6
Figure 6. Figure 6: Restricted Boltzmann Machine Value and Policy [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: Probabilistic Graphical Model For VIME I(X; Y | Z) = Z x,y,z p(z)p(x, y|z) log p(x, y|z) p(x|z)p(y|z) dxdydz = − Z x,y,z p(z)p(x, y|z) log p(x|z)dxdzdy+ + Z x,y,z p(x, y, z) log p(x|y, z)dxdzdy = H(X | Z) − H(X | Y, Z) (49) in Equation (51), which is first proposed in …
Figure 9
Figure 9. Figure 9: Graphical Model for Variation State Tabulation [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 21 canonical work pages

  1. [13]

    Reinforcement learning and control as probabilistic infer- ence: Tutorial and review,

    S. Levine, “Reinforcement learning and control as probabilistic infer- ence: Tutorial and review,” arXiv preprint arXiv:1805.00909 , 2018

    arXiv 2018

  2. [1]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, p. 529, 2015

    2015

  3. [2]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning, 2015, pp. 1889–1897

    work page 2015

  4. [3]

    Reinbo: Machine learning pipeline search and configuration with bayesian optimization embedded reinforcement learning,

    X. Sun, J. Lin, and B. Bischl, “Reinbo: Machine learning pipeline search and configuration with bayesian optimization embedded reinforcement learning,” 2019

    work page 2019

  5. [4]

    Vime: Variational information maximizing exploration,

    R. Houthooft, X. Chen, Y . Duan, J. Schulman, F. De Turck, and P. Abbeel, “Vime: Variational information maximizing exploration,” in Advances in Neural Information Processing Systems , 2016, pp. 1109– 1117

    work page 2016

  6. [5]

    Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation

    D. Corneil, W. Gerstner, and J. Brea, “Efficient model-based deep reinforcement learning with variational state tabulation,” arXiv preprint arXiv:1802.04325, 2018

    work page Pith review arXiv 2018

  7. [6]

    Variational inference: A review for statisticians,

    D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: A review for statisticians,” Journal of the American Statistical Associa- tion, vol. 112, no. 518, pp. 859–877, 2017

    work page 2017

  8. [7]

    C. M. Bishop, Pattern recognition and machine learning . springer, 2006

    2006

Show all 32 references
  1. [8]

    Training restricted boltzmann machines: An introduction,

    A. Fischer and C. Igel, “Training restricted boltzmann machines: An introduction,” Pattern Recognition, vol. 47, pp. 25–39, 01 2014

    2014

  2. [9]

    Variational resampling based assessment of deep neural networks under distribution shift,

    X. Sun, A. Gossmann, Y . Wang, and B. Bischl, “Variational resampling based assessment of deep neural networks under distribution shift,” 2019

    2019

  3. [10]

    Weight uncertainty in neural networks,

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural networks,” arXiv preprint arXiv:1505.05424, 2015

    2015 arXiv

  4. [11]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  5. [12]

    R. S. Sutton, A. G. Barto et al., Introduction to reinforcement learning. MIT press Cambridge, 1998, vol. 2, no. 4

    1998

  6. [14]

    Maximum entropy-regularized multi- goal reinforcement learning,

    R. Zhao, X. Sun, and V . Tresp, “Maximum entropy-regularized multi- goal reinforcement learning,” arXiv preprint arXiv:1905.08786 , 2019

    1905 arXiv

  7. [15]

    Planning and acting in partially observable stochastic domains,

    L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,”Artificial i ntelligence, vol. 101, no. 1-2, pp. 99–134, 1998

    1998

  8. [16]

    Deep reinforcement learning with double q-learning,

    H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Thirtieth AAAI conference on artificial intelligence, 2016

    2016

  9. [17]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015

    2015 arXiv

  10. [18]

    Deterministic policy gradient algorithms,

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” 2014

    2014

  11. [19]

    Asynchronous methods for deep rein- forcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep rein- forcement learning,” in International conference on machine learning , 2016, pp. 1928–1937

    2016

  12. [20]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  13. [21]

    Universal value func- tion approximators,

    T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value func- tion approximators,” in International Conference on Machine Learning , 2015, pp. 1312–1320

    2015

  14. [22]

    A lesson learned from pmf based approach for semantic recommender system,

    N. Kushwaha, X. Sun, B. Singh, and O. Vyas, “A lesson learned from pmf based approach for semantic recommender system,” Journal of Intelligent Information Systems , vol. 50, no. 3, pp. 441–453, 2018

    2018

  15. [23]

    Hindsight expe- rience replay,

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba, “Hindsight expe- rience replay,” in Advances in Neural Information Processing Systems , 2017, pp. 5048–5058

    2017

  16. [24]

    Prioritized experience replay,

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952 , 2015

    2015 arXiv

  17. [25]

    High dimensional restrictive federated model selection with multi-objective bayesian optimization over shifted distributions,

    X. Sun, A. Bommert, F. Pfisterer, J. Rahnenf ¨uhrer, M. Lang, and B. Bischl, “High dimensional restrictive federated model selection with multi-objective bayesian optimization over shifted distributions,” arXiv preprint arXiv:1902.08999, 2019

    1902 arXiv

  18. [26]

    Approximately optimal approximate rein- forcement learning,

    S. Kakade and J. Langford, “Approximately optimal approximate rein- forcement learning,” in ICML, vol. 2, 2002, pp. 267–274

    2002

  19. [27]

    Reinforcement learning with factored states and actions,

    B. Sallans and G. E. Hinton, “Reinforcement learning with factored states and actions,” Journal of Machine Learning Research , vol. 5, no. Aug, pp. 1063–1088, 2004

    2004

  20. [28]

    Reinforcement learning with deep energy-based policies,

    T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in Proceedings of the 34th Interna- tional Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 1352–1361

    2017

  21. [29]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” arXiv preprint arXiv:1801.01290 , 2018

    2018 arXiv

  22. [30]

    Planning to be surprised: Op- timal bayesian exploration in dynamic environments,

    Y . Sun, F. Gomez, and J. Schmidhuber, “Planning to be surprised: Op- timal bayesian exploration in dynamic environments,” in International Conference on Artificial General Intelligence . Springer, 2011, pp. 41– 51

    2011

  23. [31]

    Tutorial on variational autoencoders,

    C. Doersch, “Tutorial on variational autoencoders,” arXiv preprint arXiv:1606.05908, 2016

    2016 arXiv

  24. [32]

    The concrete distribution: A continuous relaxation of discrete random variables,

    C. J. Maddison, A. Mnih, and Y . W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” arXiv preprint arXiv:1611.00712, 2016

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.