Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Switching the KL direction in SAC turns its policy update into an explicit formula.

desk verdict The forward-KL projection derivation is correct but standard, and the Bidirectional SAC idea is reasonable, but the VDN-a critic's marginal recovery is unidentifiable and the experiments are too thinly reported to support the 30% claim. read the letter →

arxiv 2506.01639 v1 pith:IVH46KBH submitted 2025-06-02 cs.LG cs.AI

Yixian Zhang , Huaze Tang , Changxu Wei , Wenbo Ding This is my paper · ORCID
classification cs.LGcs.AI
keywords softactor-criticforwardKLdivergencereversemaximumentropyreinforcementlearningpolicyprojectionvaluedecompositionnetworkscontinuouscontrolsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard Soft Actor-Critic improves its Gaussian policy by minimizing reverse KL divergence, which has no closed-form solution and is updated by noisy gradient steps. This paper shows that minimizing the forward KL divergence instead yields an explicit optimal projection: the new policy's mean and variance are exactly the mean and variance of the Boltzmann distribution's action marginals. The authors build Bidirectional SAC, which starts each policy update at that exact forward-KL projection and then refines it with reverse KL, preserving SAC's improvement guarantee while removing the wobbly starting point. Experiments on continuous-control benchmarks report faster convergence and up to a 30% increase in episodic reward over standard SAC. The claim matters because it identifies a simple, non-parametric anchor for policy improvement in maximum-entropy reinforcement learning.

What carries the argument

The load-bearing object is the forward-KL projection identity: for a diagonal Gaussian policy, the optimal forward-KL projection of the Boltzmann distribution is obtained by setting each action dimension's mean and variance to the corresponding marginal moments, computed by numerical integration. The VDN-a critic, adapted from value decomposition networks, decomposes the global Q-function into per-action-dimension subnetworks plus an auxiliary interaction network so that each action marginal can be estimated independently. The Bidirectional SAC objective then uses the forward projection as both the initialization and a regularizer for the reverse-KL policy improvement step.

What would settle it

Train the VDN-a critic on a fixed Q-function whose Boltzmann distribution has a known bimodal marginal; if the mean and variance computed by Simpson integration of the learned marginal deviate from the true values by more than the integration error, the projection anchor is miscalibrated. Selecting a continuous-control task with strong action-dimension coupling and comparing Bidirectional SAC's episodic reward against standard SAC with identical hyperparameters would also test the claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that the intractable projection step in SAC becomes explicit when the KL direction is flipped. For a Gaussian policy with diagonal covariance, the minimizer of $D_{\mathrm{KL}}(q(\cdot|s_t)\|\pi'(\cdot|s_t))$ is the product of one-dimensional Gaussians whose mean and variance are the mean and variance of the corresponding marginal of the Boltzmann target $q(a|s_t) \propto \exp(Q(s_t,a)/\alpha)$. The paper further claims that an actor initialized at this projection and then refined by reverse-KL minimization, through the Bidirectional SAC objective $D_{\mathrm{KL}}(\pi\|q) + \epsilon(\|f^* - f\|^2 + \|\Sigma^* - \Sigma\|^2)$, converges faster and reaches higher reward than standard SAC's purely gradient-based reverse-KL update. A value-decomposition-style critic with per-action-dimension subnetworks and an auxiliary interaction network is proposed to learn the marginals needed for the projection.

Load-bearing premise

The forward-KL anchor is only as good as the critic's estimate of each action dimension's share of the target distribution, and the paper's proof of correctness requires an auxiliary network that has fully learned the interactions between action dimensions.

Editorial extensions

If this is right

  • Forward SAC alone obtains an explicit, gradient-free policy projection and outperforms standard SAC on simpler continuous-control tasks.
  • Bidirectional SAC inherits the reverse-KL guarantee of monotonic Q improvement while starting from a better-conditioned initialization, giving faster convergence and higher asymptotic reward.
  • The per-dimension marginal decomposition allows the forward projection to be computed with one-dimensional numerical integration, avoiding the curse of dimensionality.
  • If the VDN-a critic captures the first two moments of each Boltzmann marginal, the same bidirectional update should transfer to other maximum-entropy algorithms that use the same Gaussian policy family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not test this, but the same moment-matching projection could initialize other maximum-entropy or trust-region policy algorithms, not just SAC.
  • Because the method only needs the first two moments of each marginal, its occasional failure on bimodal marginals may not hurt control performance; a direct comparison of policies matched on mean and variance would test this.
  • The result reframes SAC's reverse-KL gradient step as a noisy estimator of an exact moment-matching step, suggesting that variance reduction in policy updates, rather than a new objective, may be the primary source of the reported gains.
  • A natural extension the paper leaves implicit is using higher-order marginal moments or non-Gaussian policy families when the Boltzmann target is strongly multimodal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates the use of forward KL divergence in the SAC framework. It derives that, for diagonal Gaussian policies, the forward-KL optimal projection of a Boltzmann target onto the policy family is given by the marginal means and variances of the target distribution (Section 3.2). To compute these marginals efficiently, the paper introduces VDN-a, a single-agent adaptation of Value Decomposition Networks, and proposes two algorithms: Forward SAC, which updates the policy directly via the explicit projection, and Bidirectional SAC, which uses the forward projection to initialize and regularize the standard reverse-KL actor update (Section 4.2). Experiments on MuJoCo and Box2D environments claim that Bidirectional SAC improves sample efficiency and achieves up to a 30% increase in episodic rewards over SAC and other baselines (Section 5.1).

Significance. The explicit forward-KL projection formula in Section 3.2 is a valid and potentially useful observation: for a factorization of the Gaussian policy into independent diagonal components, the optimal projection is exactly the moment-matching solution. The proposed combination of forward and reverse KL in Bidirectional SAC is conceptually appealing, as it offers a concrete way to initialize the actor closer to the target before applying the reverse-KL update. However, the practical value of the method rests on the VDN-a critic's ability to recover the true Boltzmann marginals, and this is not established. Appendix C.2.4 proves marginal recovery only from an unverified 'sufficiently learned' condition, and the decomposition underlying VDN-a is not identifiable. Therefore, the central algorithmic claim is conditional; the theoretical contribution is sound in isolation but not yet connected to the proposed algorithm in a rigorous way.

major comments (3)
  1. [Section 3.3 / Appendix C.2.4] The claim that the VDN-a critic recovers the marginal Boltzmann distributions is not established. The additive decomposition Q(s,a)=Σ_j Qsub_j(s,a_j)+U_aux(s,a) is subject to a gauge freedom: for arbitrary functions g_j(a_j), replacing Qsub_j by Qsub_j+g_j and U_aux by U_aux-Σ_j g_j leaves the total Q invariant but changes each exp(Qsub_j) and hence all estimated marginals. Thus, a perfectly learned total Q does not uniquely determine the marginal Q components. The recovery proof in Appendix C.2.4 begins with the assumption exp(Qhat_j)=K_j M^(j), which is exactly the property that needs to be shown; the 'sufficiently learned U_aux' condition is a restatement, not a mechanism, of the desired outcome. Since the forward-projection moments f_i^* and Sigma_i^* in Eq. (7) are computed from these marginals, the central anchor of Bidirectional SAC is miscalibrated unless an identifiability constraint is imposed or a separate training objective enforces the decomposition.
  2. [Appendix C.2.4, Figure 9] The paper concedes that VDN-a's approximation fails on bimodal marginal structures. The Boltzmann distribution in continuous control can be multimodal, and the paper does not quantify how often or how severely the estimated mean/variance deviate from the true moments in such cases. The statement that 'the first two moments are often still well-captured' is not supported by any numerical evidence. This is a load-bearing concern because the forward projection uses only these moments, and a miscalibrated mean or variance can mislead both the initialization and the regularization in Bidirectional SAC.
  3. [Section 5.1, Figure 4] The empirical claim of 'up to a 30% increase in episodic rewards' is not supported by standard statistical rigor. The learning curves are presented without error bars, and Appendix F does not report the number of independent runs or the variance across seeds. Without this information, the comparison against SAC, A2C, PPO, and DDPG is not verifiable, and the claim of 'significantly outperforms' is not substantiated. The paper should report means and standard deviations over at least 5-10 seeds for all environments and baselines.
minor comments (6)
  1. [Abstract / Section 3.2] The abstract and introduction state that the explicit projection holds 'for Gaussian policies'; the derivation in Section 3.2 specifically assumes a diagonal covariance matrix. Please state this assumption in the abstract or at the beginning of Section 3.2 to avoid overgeneralization.
  2. [Equation (12) / Equation (13)] Equation (12) introduces the integration bound b as a hyperparameter, and Equation (13) uses a Simpson partition count I, but no concrete values are given anywhere in the paper. Please provide these values or a sensitivity analysis.
  3. [Section 3.2] In Section 3.2, the notation Σ^i_φ is used both for the standard deviation in the Gaussian density and for the variance in the optimal projection formula. This is confusing; please use distinct symbols (e.g., σ for standard deviation and σ^2 for variance).
  4. [Appendix C.2.4, Figures 8 and 9] The captions for Figures 8 and 9 contain "(Assumed: ...)", which appears to be a placeholder. Please replace with proper captions that state precisely what is plotted (e.g., true vs. estimated marginal densities).
  5. [Section 5.1] The paper does not compare against modern SAC variants that also improve sample efficiency (e.g., REDQ, DroQ, or distributional SAC). Adding such comparisons would help isolate the contribution of the bidirectional mechanism.
  6. [Section 5.2, Figure 5] In Section 5.2, Figure 5 reports a qualitative comparison of estimated marginals to ground truth for a single state; please provide quantitative metrics (e.g., KL divergence, error in mean/variance) over a set of states to substantiate the claim that the first two moments are captured.

Circularity Check

1 steps flagged · score 4.0 of 10

Forward-KL projection is self-contained, but the VDN-a marginal-recovery theorem is definitional: it assumes the subnetwork identity it claims to prove.

  1. self definitional [Appendix C.2.4, 'Recovery of true marginal distributions' (also summarized in Section 3.3)]
    "If exp( Qhat_j(a_j; s_t) ) = K_j(s_t) M^(j)(a_j; s_t) for some K_j(s_t) > 0, the denominator becomes K_j(s_t) ∫_{A_j} M^(j)(a'_j; s_t) da'_j. Since ∫_{A_j} M^(j)(a'_j; s_t) da'_j = Z(s_t), then: qhat_j(a_j|s_t) = M^(j)(a_j; s_t) / Z(s_t) = q_j(a_j|s_t). Thus, if each (1/α) Qsub_j(s_t, a_j) effectively learns ln(M^(j)(a_j; s_t)) (up to an additive constant), and U_aux isolates dependencies, then this procedure recovers the true marginal PDFs."

    The recovery proof starts by assuming exp(Qhat_j) = K_j M^(j), which is exactly the property that the VDN-a subnetwork must learn. No training signal on the total Q forces this split: Q_total = sum_j Qsub_j + U_aux is invariant under Qsub_j -> Qsub_j + g_j, U_aux -> U_aux - sum_j g_j, so the same total critic yields arbitrary estimated marginals. Thus 'sufficiently learned U_aux' cannot select the true ln M^(j) decomposition. The final identity qhat_j = q_j follows by normalization from the assumed equality, i.e., the conclusion is the antecedent restated. Figure 9 concedes that the approximation fails on bimodal marginals, so the condition is not only unproved but empirically violated in some cases.

full rationale

Most of the derivation chain is self-contained and non-circular. Section 3.2 explicitly differentiates D_KL(q||pi') with respect to the Gaussian mean and variance and obtains f_i* and Sigma_i* as the marginal moments of q; this is a standard calculus result and does not depend on fitting or on the authors' prior work. The Bidirectional SAC objective in Equation (7) uses these moments as targets, which is algorithmic design rather than circular prediction. The empirical evaluation against SAC, PPO, and DDPG is independent of the disputed theorem. The one definitional step is Appendix C.2.4: the claim that VDN-a recovers the true Boltzmann marginals is conditional on exp(Qhat_j) proportional to M^(j), which is precisely the unidentified representation the network must learn. Because the critic loss supervises only the sum, the decomposition is non-identifiable up to additive gauge transformations, so the 'sufficiently learned U_aux' condition is not derived from the architecture or loss. This weakens the theoretical justification of the VDN-a-based forward projection, but it does not make the forward-KL moment projection or the benchmark results circular. There are no load-bearing self-citations in the paper. Score 4 reflects one definitional step in a supporting theorem while the central empirical and mathematical contributions retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central derivation rests on a diagonal Gaussian assumption and on the VDN-a architecture being able to isolate marginal Boltzmann components. The free parameters are not reported, which weakens both reproducibility and empirical soundness.

free parameters (4)
  • epsilon in Eq. (7)
    Weight of the forward-KL projection penalty in the Bidirectional SAC objective; no value or tuning schedule is given.
  • integration bound b in Eq. (12)
    Truncation bound for numerical integration over tanh-transformed actions; affects the estimated mean and variance.
  • Simpson partition count I in Eq. (13)
    Number of subintervals for Simpson's rule; controls numerical integration accuracy and is not reported.
  • temperature alpha in Eq. (1)
    Entropy coefficient in the maximum entropy objective; not specified or auto-tuned in the experiments.
assumptions (5)
  • domain assumption Diagonal Gaussian policy family
    Section 3.2 assumes the policy factorizes into independent one-dimensional Gaussians; correlations between action dimensions are delegated to the neural networks.
  • standard math Sklar's theorem and copula decomposition
    Used in Appendix C.2 to decompose the Boltzmann potential into marginal terms plus a copula interaction term.
  • ad hoc to paper Auxiliary network captures the copula interaction term
    Marginal recovery in Appendix C.2.4 requires U_aux to learn the log-copula term and constants; no guarantee is proven, and Figure 9 shows failure cases.
  • domain assumption Boltzmann distribution is numerically integrable with Simpson's rule over a bounded interval
    Eq. (12) truncates integration to [-b, b] and assumes Simpson's rule accurately approximates the moments; no error control is provided.
  • domain assumption Reverse-KL gradient updates in SAC improve expected Q
    Section 4.1 relies on the SAC improvement guarantee, but with function approximation and sampling it is approximate; the paper itself acknowledges instability.
invented entities (2)
  • VDN-a critic architecture
    purpose: Estimate per-action-dimension marginal Q values so Boltzmann marginals can be computed via one-dimensional numerical integration.
    Only validated on the paper's own environments; no external benchmark or theoretical guarantee that the decomposition recovers true marginals.
  • Auxiliary interaction network U_aux
    purpose: Learn the copula and interaction terms so subnetworks can isolate ln M^(j) terms.
    The key condition for marginal recovery; the paper admits it can fail on bimodal marginals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning." pith.science (2026). https://pith.science/paper/IVH46KBH

@misc{pith2026250601639,
  author       = {Pith},
  title        = {Pith review of: Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IVH46KBH}},
  note         = {Machine review of arXiv:2506.01639}
}
abstract

The Soft Actor-Critic (SAC) algorithm, a state-of-the-art method in maximum entropy reinforcement learning, traditionally relies on minimizing reverse Kullback-Leibler (KL) divergence for policy updates. However, this approach leads to an intractable optimal projection policy, necessitating gradient-based approximations that can suffer from instability and poor sample efficiency. This paper investigates the alternative use of forward KL divergence within SAC. We demonstrate that for Gaussian policies, forward KL divergence yields an explicit optimal projection policy -- corresponding to the mean and variance of the target Boltzmann distribution's action marginals. Building on the distinct advantages of both KL directions, we propose Bidirectional SAC, an algorithm that first initializes the policy using the explicit forward KL projection and then refines it by optimizing the reverse KL divergence. Comprehensive experiments on continuous control benchmarks show that Bidirectional SAC significantly outperforms standard SAC and other baselines, achieving up to a $30\%$ increase in episodic rewards, alongside enhanced sample efficiency.

Figures

Figures reproduced from arXiv: 2506.01639 by the authors.

Figure 1
Figure 1. Optimization of the parameterized distribution [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The structure of VDN-a network To compute the optimal values f i (st) ∗ and Σ i∗ effi￾ciently, we introduce the VDN-a network as the critic network to learn the marginal Q values Qi(st, ai t ) for each action dimension a i t . This approach is inspired by Value Decomposition Networks (VDN), where a brief introduction to it is provided in Appendix C.1. Similar to VDN, which designs an independent sub￾network for each… view at source ↗
Figure 3
Figure 3. Conceptual illustration of policy optimization dynamics for (Reverse) SAC and Forward [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The episodic reward in MuJoCo environments for Bidirectional SAC, Forward SAC, and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The comparison result in marginal distribution learned by VDN-a networks to the true [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The comparison result for the updated policy in Forward SAC and standard SAC to the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Framework of the original Value Decomposition Network (VDN) for MARL. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: VDN-a performance in approximating components related to a unimodal marginal distribu [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: VDN-a performance in approximating components related to a bimodal marginal distribu [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The episodic reward in MuJoCo and Box2D environments for Bidirectional SAC, Forward [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 35 canonical work pages

  1. [1]

    Deriving and improving cma-es with information geometric trust regions

    Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis Paulo Reis, and Gerhard Neumann. Deriving and improving cma-es with information geometric trust regions. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 657--664, 2017

    work page 2017

  2. [2]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018 a

    work page 2018

  3. [3]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018 b

    arXiv 2018

  4. [4]

    Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experiences

    Chayan Banerjee, Zhiyong Chen, and Nasimul Noman. Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experiences. IEEE Transactions on Neural Networks and Learning Systems, 35 0 (3): 0 3121--3129, 2022

    work page 2022

  5. [5]

    Sukhatme

    Sumeet Batra, Bryon Tjanaka, Matthew Christopher Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav S. Sukhatme. Proximal policy gradient arborescence for quality diversity reinforcement learning. In International Conference on Learning Representations, 2024

    work page 2024

  6. [6]

    Box2d: A 2d physics engine for games

    Erin Catto. Box2d: A 2d physics engine for games. URL: http://www. box2d. org, 2011

    work page 2011

  7. [7]

    Greedification operators for policy optimization: Investigating forward and reverse kl divergences

    Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23 0 (253): 0 1--79, 2022

    2022

  8. [8]

    Using expectation-maximization for reinforcement learning

    Peter Dayan and Geoffrey E Hinton. Using expectation-maximization for reinforcement learning. Neural Computation, 9 0 (2): 0 271--278, 1997

    work page 1997

Show all 46 references
  1. [9]

    Soft actor-critic for navigation of mobile robots

    Junior Costa de Jesus, Victor Augusto Kich, Alisson Henrique Kolling, Ricardo Bedin Grando, Marco Antonio de Souza Leite Cuadros, and Daniel Fernando Tello Gamarra. Soft actor-critic for navigation of mobile robots. Journal of Intelligent & Robotic Systems, 102 0 (2): 0 31, 2021

    2021

  2. [10]

    Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors

    Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on Neural Networks and Learning Systems, 33 0 (11): 0 6584--6598, 2021 a

    2021

  3. [11]

    Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors

    Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (11): 0 6584--6598, 2021 b

    2021

  4. [12]

    Virel: A variational inference framework for reinforcement learning

    Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [13]

    Brax - a differentiable physics engine for large scale rigid body simulation

    Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  6. [14]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 2021

    2021

  7. [15]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34nd International Conference on Machine Learning, pages 1352--1361, 2017

    2017

  8. [16]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35nd International Conference on Machine Learning, pages 1861--1870, 2018 a

    2018

  9. [17]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018 b

    2018 arXiv

  10. [18]

    The curse of dimensionality for numerical integration of smooth functions

    Aicke Hinrichs, Erich Novak, Mario Ullrich, and H Wo \'z niakowski. The curse of dimensionality for numerical integration of smooth functions. Mathematics of Computation, 83 0 (290): 0 2853--2863, 2014

    2014

  11. [19]

    Graph soft actor–critic reinforcement learning for large-scale distributed multirobot coordination

    Yifan Hu, Junjie Fu, and Guanghui Wen. Graph soft actor–critic reinforcement learning for large-scale distributed multirobot coordination. IEEE Transactions on Neural Networks and Learning Systems, 36 0 (1): 0 665--676, 2025

    2025

  12. [20]

    Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo \ A G o GM Ara \ A s jo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 ...

    2022

  13. [21]

    Accelerating reinforcement learning with value-conditional state entropy exploration

    Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. Advances in Neural Information Processing Systems, 36, 2024

    2024

  14. [22]

    Optimistic reinforcement learning by forward kullback--leibler divergence optimization

    Taisuke Kobayashi. Optimistic reinforcement learning by forward kullback--leibler divergence optimization. Neural Networks, 152: 0 169--180, 2022

    2022

  15. [23]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020

    2020

  16. [24]

    Continuous control with deep reinforcement learning

    TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  17. [25]

    Constrained variational policy optimization for safe reinforcement learning

    Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644--13668, 2022

    2022

  18. [26]

    Algorithm 145: Adaptive numerical integration by simpson's rule

    William Marshall McKeeman. Algorithm 145: Adaptive numerical integration by simpson's rule. Communications of the ACM, 5 0 (12): 0 604, 1962

    1962

  19. [27]

    On principled entropy exploration in policy optimization

    Jincheng Mei, Chenjun Xiao, Ruitong Huang, Dale Schuurmans, and Martin M \"u ller. On principled entropy exploration in policy optimization. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3130--3136, 2019

    2019

  20. [28]

    Importance sampling techniques for policy optimization

    Alberto Maria Metelli, Matteo Papini, Nico Montali, and Marcello Restelli. Importance sampling techniques for policy optimization. Journal of Machine Learning Research, 21 0 (141): 0 1--75, 2020

    2020

  21. [29]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

    2015

  22. [30]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1928--1937, 2016

    1928

  23. [31]

    Improving policy gradient by exploring under-appreciated rewards

    Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.09321, 2016

    2016 arXiv

  24. [32]

    Stable-baselines3: Reliable reinforcement learning implementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 0 (268): 0 1--8, 2021

    2021

  25. [33]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889--1897, 2015

    2015

  26. [34]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  27. [35]

    Monte carlo sampling methods

    Alexander Shapiro. Monte carlo sampling methods. Handbooks in Operations Research and Management Science, 10: 0 353--425, 2003

    2003

  28. [36]

    V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control

    H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conferen...

    2020

  29. [37]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017

    2017 arXiv

  30. [38]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, 2018

    2018

  31. [39]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026--5033, 2012

    2012

  32. [40]

    Probabilistic inference for solving discrete and continuous state markov decision processes

    Marc Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state markov decision processes. In Proceedings of the 23rd international conference on Machine learning, pages 945--952, 2006

    2006

  33. [41]

    Self-play reinforcement learning guides protein engineering

    Yi Wang, Hui Tang, Lichao Huang, Lulu Pan, Lixiang Yang, Huanming Yang, Feng Mu, and Meng Yang. Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, 5 0 (8): 0 845--860, 2023

    2023

  34. [42]

    Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation

    Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. Advances in Neural Information Processing Systems, 30, 2017

    2017

  35. [43]

    Monocular vision approach for soft actor-critic based car-following strategy in adaptive cruise control

    Jiachen Yang, Jiankun Peng, Quanwei Zhang, Weiqi Chen, and Chunye Ma. Monocular vision approach for soft actor-critic based car-following strategy in adaptive cruise control. Expert Systems with Applications, page 125999, 2024

    2024

  36. [44]

    Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning

    Qisong Yang, Thiago D Sim \ a o, Simon H Tindemans, and Matthijs TJ Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10639--10646, 2021

    2021

  37. [45]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1433--1438, 2008

    2008

  38. [46]

    Wasserstein gradient flows for optimizing gaussian mixture policies

    Hanna Ziesche and Leonel Rozo. Wasserstein gradient flows for optimizing gaussian mixture policies. Advances in Neural Information Processing Systems, 36, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.