Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

Daeho Kim; Hyoseok Hwang; Jeong Woon Lee; Kyoleen Kwak

REVIEW 5 major objections 5 minor 4 references

Policy jitter in actor-critic RL is driven by the critic's geometry, not the actor's, and a regularizer that stabilizes the Q-gradient field by shrinking mixed partials while preserving action curvature achieves smooth policies without touc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:16 UTC pith:GOGT44BL

load-bearing objection The math is standard, but the critic-centric framing and PAVE regularizer are a legitimate new idea; the empirical validation is confounded and the curvature penalty does not enforce the theory it claims, so the paper overreaches but deserves referee time. the 5 major comments →

arxiv 2601.22970 v2 pith:GOGT44BL submitted 2026-01-30 cs.LG cs.AI

Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

Jeong Woon Lee , Kyoleen Kwak , Daeho Kim , Hyoseok Hwang This is my paper

classification cs.LG cs.AI

keywords actor-criticpolicy smoothnessQ-function geometryimplicit differentiationHessian regularizationcontinuous controlLipschitz continuityreinforcement learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that high-frequency oscillations in continuous actor-critic reinforcement learning stem from the critic's differential geometry, not from the actor. By implicit differentiation, the sensitivity of the optimal action is exactly the inverse action Hessian times the mixed state–action Hessian, yielding a Lipschitz bound L ≤ M/µ where M bounds the mixed partials and µ lower-bounds the curvature. To act on this, the authors introduce PAVE, which regularizes the critic by penalizing mixed-partial volatility, aligning Q-gradients along transitions, and explicitly preserving curvature to prevent over-smoothing. Experiments on six environments show that PAVE achieves smoothness and task performance comparable to policy-side regularizers without altering the actor, and it degrades gracefully under observation noise. The underlying shift in perspective is that stabilizing the learning signal is more principled than forcing the policy to be smooth.

Core claim

The central claim is that the smoothness of an actor-critic policy is governed by the Q-function's geometry via the Implicit Policy Jacobian (Lemma 4.1): Jπ(s) = -(∇²_aa Qθ(s,a*(s)))^{-1} ∇²_sa Qθ(s,a*(s)). This yields a Lipschitz bound (Proposition 4.2) L ≤ M/µ, where M bounds the spectral norm of the mixed partials and µ is the lower bound on the negative-definiteness of the action Hessian. The paper further asserts that conventional policy-side regularization treats the symptom: the actor cannot be smoother than the greedy policy a*(s) without sacrificing returns. PAVE therefore minimizes M and keeps µ away from zero through three critic auxiliary losses—Mixed-Partial Regularization, Vect

What carries the argument

The key object is the implicit-function-theorem derived ratio of the Q-function's Hessians: policy sensitivity equals -(∇²_aa Q)^{-1} ∇²_sa Q, giving the M/µ Lipschitz bound. This factorization drives the design of PAVE, whose machinery consists of three critic-side losses: Mixed-Partial Regularization (finite-difference proxy for ∇²_sa Q) suppresses the numerator M, Vector Field Consistency aligns ∇_a Q along transitions to tame temporal chattering, and Curvature Preservation (Hutchinson trace on ∇²_aa Q) keeps the denominator µ bounded away from zero, preventing the inverse-Hessian amplification that would otherwise explode sensitivity.

Load-bearing premise

The theoretical bound assumes that the learned actor is exactly the greedy maximizer a*(s) and that the critic is twice differentiable with a strictly negative-definite action Hessian at interior maxima; in practice, the actor is only an approximate maximizer and the Curvature Preservation loss only enforces concavity along a few random directions, so the uniform M/µ bound is not guaranteed to hold.

What would settle it

For a set of trained actor-critic agents, measure the empirical policy Jacobian norm ∥∇_s πϕ(s)∥ and compare it to the theoretically predicted L = M/µ computed from the same critic. If agents with low M/µ still exhibit high-frequency oscillations or if the Jacobian norm exceeds the predicted bound by a large margin, then the critic's geometry is not the dominant mechanism controlling policy smoothness.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Since both deterministic (TD3-style) and stochastic (SAC-style) actors ascend ∇_a Q, stabilizing the critic's geometry can smooth policies across actor types without modifying the actor's update rule.
Regularizing the critic alone can match or exceed the smoothness and returns of policy-side regularizers, suggesting that the critic's learning signal, not the policy parameterization, is the limiting factor for action continuity.
Minimizing M alone is insufficient; preserving curvature µ is what prevents over-smoothing collapse, confirming the two-term structure of the bound.
The robustness to observation noise suggests that a flattened Q-gradient field decouples input perturbations from action generation, which could improve sim-to-real transfer.
The regularizers use only O(k+d) computations per update via finite differences and Hutchinson trace, making the method scalable to high-dimensional control.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The M/µ bound is a testable diagnostic: practitioners could compute these two quantities from a trained critic to predict how jittery the induced policy will be, before deployment—something the paper does not explicitly propose.
The Fisher-divergence framing hints at a deeper link between score-based generative modeling and RL; the same tools used to estimate smooth score fields could be adapted to construct smoother Q-fields, potentially generalizing to stochastic policies where the greedy-maximizer assumption is relaxed.
The choice of activation function (SiLU for C² continuity) is not an engineering detail but a prerequisite for the theory; this suggests that other architectural choices affecting second-order derivatives may be more consequential for policy smoothness than previously recognized.
If the causal claim holds, then offline RL and imitation learning could also benefit from critic-side geometric regularization to produce deployable, non-oscillatory policies even when the dataset itself is noisy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The math is standard, but the critic-centric framing and PAVE regularizer are a legitimate new idea; the empirical validation is confounded and the curvature penalty does not enforce the theory it claims, so the paper overreaches but deserves referee time.

read the letter

Quick take: the paper's math is correct but standard — Lemma 4.1 is the usual implicit-function result and Proposition 4.2 is a one-line spectral-norm bound. The novelty is in the framing and in the PAVE loss combination, and that is legitimate. If the critic's geometry really drives policy instability, then regularizing the critic instead of the actor is a sensible bet, and the authors build a coherent framework around it.

What the paper does well: the derivation is clear, the bound L ≤ M/µ is useful intuition, and the three losses map naturally onto that bound. The experiments cover six environments and two base algorithms, with an ablation and a robustness study. The ablation in Table 4 is the most instructive part: adding curvature preservation recovers performance that pure smoothing loses, which is consistent with the M/µ story.

The soft spots are empirical plus one theoretical gap. First, the main comparison is confounded: PAVE uses SiLU while the baselines use ReLU, so part of the smoothness gain could simply be the smoother activation. The component analysis does include a SiLU baseline, but the main tables do not — that needs to be fixed. Second, Curvature Preservation uses a single Hutchinson projection to penalize v^T H_aa v + δ. One random projection cannot certify that all eigenvalues are below −δ, so the condition in Proposition 4.2 is not actually enforced. The paper's language about "guaranteeing" bounded inverse Hessian norm is an overclaim; the regularizer is a heuristic proxy. Third, hyperparameters are tuned per environment and five-seed variance is high; several return differences are within noise. There is also a small factual slip in Appendix A calling SiLU piecewise linear.

None of this kills the paper. The idea is good and the evidence is suggestive, but the claim that the bound "fundamentally" governs policy smoothness goes beyond what the experiments show. A serious referee should push for a controlled activation comparison, direct measurement of M and µ during training, and code release.

This is for anyone working on smoothness in continuous-control RL or on critic-side regularization. Worth a reading-group discussion. I would send it to peer review — it deserves referee time — but the theoretical framing should not be accepted as proven.

Referee Report

5 major / 5 minor

Summary. The paper argues that action oscillations in continuous actor-critic RL are caused by the critic's geometry rather than policy irregularities. Using implicit differentiation, it derives the Jacobian of the greedy policy as Jπ(s) = -(∇²_aa Qθ)⁻¹ ∇²_sa Qθ and bounds its Lipschitz constant by L ≤ M/μ, where M bounds the mixed state–action Hessian and μ lower-bounds the action Hessian's negative definiteness. The proposed method PAVE regularizes the critic with three terms: Mixed-Partial Regularization (MPR) to suppress M, Vector Field Consistency (VFC) for temporal alignment, and Curvature Preservation (Curv) to keep μ away from zero. Experiments on six Gymnasium environments under TD3 and SAC report smoothness comparable to policy-side regularizers while maintaining task performance.

Significance. If validated, the paper's central insight — that critic differential geometry governs policy smoothness — would provide a principled alternative to policy-side regularization. The IFT-based derivation is elegant and the M/μ bound is a clean, falsifiable statement. The paper also contributes a practical critic-regularization pipeline whose components are computationally cheap. However, the empirical validation and the implementation of the curvature condition currently fall short of establishing the theoretical mechanism. The method shows promise, but the current evidence does not yet justify the strong claims made in the paper.

major comments (5)

[§5.3, Eq. (15)] The Curvature Preservation loss uses a single Rademacher projection per sample: LCurv = E[max(0, v^T ∇²_aa Q v + δ)]. The Formal Justification claims this 'stochastically enforces λmax(∇²_aa Q) ≤ -δ'. This is false. A one-dimensional random projection cannot certify the maximum eigenvalue; the hinge on v^T H v can be zero even when a positive eigenvalue exists, because E[v^T H v] = tr(H) and the trace can be negative while the spectrum is indefinite. Thus the trained critic may have indefinite action Hessians at sampled states, violating the strict concavity assumption of Proposition 4.2. The paper's assertion that the inverse Hessian norm 'remains bounded' is therefore unsupported. A spectral-norm penalty, multiple projections, or explicit eigenvalue computation is needed to justify the claim.
[§6.1, Tables 1–3] PAVE uses SiLU activation for the critic, while all baselines use ReLU. The activation function itself affects policy smoothness and value-field geometry. The component analysis in §6.5 uses a SiLU baseline, but the main tables do not include policy-side regularizers (CAPS, GRAD, ASAP) trained with SiLU. Without these controls, the reported smoothness gains cannot be attributed to the regularization terms rather than the smoother activation. This is a load-bearing confound for the central empirical claim.
[§6.5, Table 4, Appendix B] The method is evaluated with hyperparameters λ1, λ2, λ3 tuned per environment, with values spanning orders of magnitude (e.g., λ1 = 0.1 to 2.0 in TD3; λ3 = 0.01 to 2.0). This heavy per-environment tuning weakens the practical claim and suggests the method is sensitive. The sensitivity plots in Appendix B show that performance degrades sharply away from the chosen values. The paper would benefit from a single hyperparameter setting across all environments, or at least a discussion of the tuning burden and sensitivity.
[Appendix A.1] The visualization of the mixed-partial Hessian norm uses a finite-difference approximation of the gradient difference. For ReLU networks (and piecewise-linear critics), the true second derivative is zero almost everywhere, yet the figures show spikes. The paper acknowledges this but still labels the quantity ∥∇²_sa Q∥. This means the empirical 'smoothness' being visualized is not the theoretical M in Proposition 4.2. Either the proxy should be redefined (e.g., as a gradient Lipschitz measure) and its relation to the bound made explicit, or the theoretical connection is unvalidated.
[§4.2, Proposition 4.2] The bound L ≤ M/μ applies to the exact greedy policy a*(s) = argmax_a Qθ(s,a). In practice the actor is an approximate maximizer, and its Jacobian is not directly related to the inverse action Hessian. The paper claims the parameterized actor 'cannot exceed the smoothness of a*' without a proof, and the empirical smoothness measurements are taken on the actor, not on a*. The theoretical-to-empirical transfer requires an error analysis for the approximate actor, or at least a discussion of when the bound approximately holds.

minor comments (5)

[§4.2, Eq. (5)] The proof refers to 'Theorem 4.1' but the statement is Lemma 4.1. Please correct.
[Appendix A.1] The statement that 'standard neural networks with ReLU or SiLU activations are piecewise linear' is inaccurate; SiLU is smooth (C∞). This matters because the paper relies on SiLU for C² continuity.
[Figure 1] The y-axis label appears garbled: '||s,a Q||'. Please clarify the exact quantity plotted.
[Tables 1–3] Results are reported over only five seeds, with large standard deviations. No significance tests are provided. Given the variability, many reported differences are within one standard deviation; this limits the strength of the conclusions.
[§6.6] The complexity claim of O(k+d) is plausible for the regularizers, but the actual throughput in Table 5 shows a 3–4× slowdown (e.g., 25 FPS vs 97 FPS for LunarLander). This is a practical cost that should be discussed more thoroughly, especially for real-time deployment.

Circularity Check

0 steps flagged

No significant circularity: the policy-Jacobian and Lipschitz bound are derived from stated assumptions via standard implicit differentiation and spectral-norm inequalities; the empirical smoothness metric is an independent FFT-based score, and the only self-citation (ASAP, Kwak & Hwang 2026) is not load-bearing.

full rationale

The derivation chain is self-contained. Lemma 4.1 obtains Jπ(s) = -(∇²_aaQ)^{-1}∇²_saQ from the first-order condition ∇_aQ(s,a*(s)) = 0 by implicit differentiation (Eqs. 2-4), and Proposition 4.2 applies the sub-multiplicative spectral-norm inequality to get L ≤ ||(∇²_aaQ)^{-1}||·||∇²_saQ|| ≤ M/µ (Eqs. 6-9). No step restates an input: the bounds M and µ are hypotheses of the theorem, not quantities fitted to data, and the theorem is never inverted to fit constants. The empirical validation uses the independent FFT-based Smoothness Score per Mysore et al.'s protocol and environment rewards; neither equals the regularized losses LMPR/LVFC/LCurv, so no 'prediction' reduces by construction to a fitted quantity. The design-to-theory loop (PAVE minimizes M and penalizes vanishing µ; smoother policies result) is confirmatory engineering, not a definitional equivalence. The only self-citation is (Kwak & Hwang, 2026) (ASAP), whose authors overlap with this paper; it is used for related-work context, for the general point that temporal coherence matters, and for implementation reuse, but the VFC objective itself is defined in-paper (Eq. 13) via a score-matching analogy, so the citation is not load-bearing. Two correctness caveats, neither circular: Eq. 15's single Hutchinson/Rademacher projection does not certify the uniform spectral condition λmax(∇²_aaQ) ≤ −δ that Prop. 4.2 assumes, and Appendix A.1's statement that SiLU is piecewise linear is technically wrong; these are theory-implementation gaps, not input-output reductions.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The theoretical result relies on standard calculus and an idealized smoothness/concavity assumption that is only heuristically enforced. All free parameters are tuned per environment, and the central bound is not directly verified.

free parameters (5)

λ1 (MPR weight) = 0.1–2.0 (per environment, TD3); 0.1–2.0 (SAC)
Tuned per environment, reported in Tables 7–8; significantly affects smoothness/return trade-off.
λ2 (VFC weight) = 0.005–0.1 (TD3); 5e-4–0.5 (SAC)
Tuned per environment; controls temporal gradient alignment strength.
λ3 (Curv weight) = 0.01–2.0 (per environment)
Tuned per environment; loss ablation shows it is required to maintain returns.
δ (sharpness margin in Curv) = Not reported
Defined as positive minimum curvature in Eq. 15, no value given; effectively a free parameter.
σ (noise scale for MPR perturbation) = Not reported in hyperparameter tables
Used in Algorithm 1 (ϵ∼N(0,σ²I)) but not specified; affects MPR proxy accuracy.

axioms (4)

standard math Implicit Function Theorem for the first-order condition ∇_a Q(s,a*(s))=0
Used in Lemma 4.1 to derive the policy Jacobian; requires a zero of the gradient and an invertible action Hessian.
domain assumption Qθ is twice continuously differentiable with respect to actions and states
Needed for the Hessians in the theory. The paper uses SiLU specifically to ensure C², but the comparison baselines use ReLU, which has zero second derivatives almost everywhere.
domain assumption The optimal action a*(s) is an interior point of the action space and a strict local maximum
The IFT requires an interior solution; many continuous control environments use bounded actions (e.g., tanh), where the argmax can lie at the boundary.
ad hoc to paper Curvature Preservation via Hutchinson's trace estimator enforces λ_max(∇²_aaQ) ≤ -δ
The paper claims this stochastically enforces the spectral constraint, but penalizing a few random projections does not guarantee a bound on the maximum eigenvalue.

pith-pipeline@v1.3.0-alltime-deepseek · 17497 in / 8548 out tokens · 92580 ms · 2026-08-03T06:16:12.194904+00:00 · methodology

0 comments

read the original abstract

Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment. Current approaches attempt to enforce smoothness by directly regularizing the policy's output. We argue that this approach treats the symptom rather than the cause. In this work, we theoretically establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. By applying implicit differentiation to the actor-critic objective, we prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). To empirically validate this theoretical insight, we introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework that treats the critic as a scalar field and stabilizes its induced action-gradient field. PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature. Experimental results demonstrate that PAVE achieves smoothness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance, without modifying the actor.

Figures

Figures reproduced from arXiv: 2601.22970 by Daeho Kim, Hyoseok Hwang, Jeong Woon Lee, Kyoleen Kwak.

**Figure 1.** Figure 1: Comprehensive 3D visualization of the mixed-partial Hessian norm ∥∇2 saQ∥ in LunarLander. While baseline methods exhibit highly irregular landscapes with sharp spikes, PAVE effectively stabilizes the Q-gradient field, providing a smooth and stable landscape. formance. Notably, in the Walker task, PAVE achieved the highest cumulative return of 5563 while simultaneously recording the best smoothness score of… view at source ↗

**Figure 2.** Figure 2: Comprehensive 3D visualization of the mixed-partial Hessian norm ∥∇2 saQ∥ across six Gymnasium environments. Each row corresponds to an environment, and each column represents a different stabilization method. While baseline methods exhibit highly irregular landscapes with sharp spikes (indicating an unstable learning signal), PAVE effectively paves the Q-gradient field, providing a smooth and stable lands… view at source ↗

**Figure 3.** Figure 3: Comprehensive 3D visualization of the mixed-partial Hessian norm ∥∇2 saQ∥ across six Gymnasium environments. Each row corresponds to an environment, and each column represents a different stabilization method. While baseline methods exhibit highly irregular landscapes with sharp spikes (indicating an unstable learning signal), PAVE effectively paves the Q-gradient field, providing a smooth and stable lands… view at source ↗

**Figure 4.** Figure 4: Hyperparameter Sensitivity on Pendulum. For each subplot, only the target parameter was varied while the other two were held fixed at their default settings [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

**Figure 5.** Figure 5: Learning curves of the TD3 algorithm across various Gymnasium environments. 0 1 2 3 4 5 Step 1e5 −200 −100 0 100 200 300 Episode Reward Lunar BASE CAPS GRAD ASAP PAVE 0.0 0.2 0.4 0.6 0.8 1.0 Step 1e5 −1200 −1000 −800 −600 −400 −200 Episode Reward Pendulum BASE CAPS GRAD ASAP PAVE 0 1 2 3 4 5 Step 1e5 −45 −40 −35 −30 −25 −20 −15 Episode Reward Reacher BASE CAPS GRAD ASAP PAVE 0.0 0.2 0.4 0.6 0.8 1.0 Step 1e… view at source ↗

**Figure 6.** Figure 6: Learning curves of the SAC algorithm across various Gymnasium environments. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith

[395]

9 Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Sokoli´c, J., Giryes, R., Sapiro, G., and Rodrigues, M

Pmlr, 2014. 9 Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Sokoli´c, J., Giryes, R., Sapiro, G., and Rodrigues, M. R. Robust large margin deep neural networks.IEEE Trans- actions on Signal Processing, 65(16):4265–4280, 2017. Song, X., Duan, J., Wang, W., Li, S. E., Chen, C., Cheng, B., Zhang, B., Wei, J., and Wang, X. S. Lipsnet:...

2014
[2019]

dominant axes

URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper. pdf. Takase, R., Yoshikawa, N., Mariyama, T., and Tsuchiya, T. Stability-certified reinforcement learning control via spec- tral normalization.Machine Learning with Applications, 10:100409, 2022. Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics ...

Pith/arXiv arXiv 2019
[2025]

URL https://openreview.net/forum? id=KZo2XhcSg6. Song, Y . and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alch´e-Buc, F., Fox, E., and Garnett, R. (eds.),Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,
[3744]

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S

PMLR, 2021. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. InInternational conference on machine learning, pp. 1861–1870. Pmlr, 2018. Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsou- nis, V ., Koltun, V ., and Hutter, M. Learning agile and d...

arXiv 2021

[1] [395]

9 Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Sokoli´c, J., Giryes, R., Sapiro, G., and Rodrigues, M

Pmlr, 2014. 9 Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Sokoli´c, J., Giryes, R., Sapiro, G., and Rodrigues, M. R. Robust large margin deep neural networks.IEEE Trans- actions on Signal Processing, 65(16):4265–4280, 2017. Song, X., Duan, J., Wang, W., Li, S. E., Chen, C., Cheng, B., Zhang, B., Wei, J., and Wang, X. S. Lipsnet:...

2014

[2] [2019]

dominant axes

URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper. pdf. Takase, R., Yoshikawa, N., Mariyama, T., and Tsuchiya, T. Stability-certified reinforcement learning control via spec- tral normalization.Machine Learning with Applications, 10:100409, 2022. Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics ...

Pith/arXiv arXiv 2019

[3] [2025]

URL https://openreview.net/forum? id=KZo2XhcSg6. Song, Y . and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alch´e-Buc, F., Fox, E., and Garnett, R. (eds.),Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,

[4] [3744]

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S

PMLR, 2021. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. InInternational conference on machine learning, pp. 1861–1870. Pmlr, 2018. Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsou- nis, V ., Koltun, V ., and Hutter, M. Learning agile and d...

arXiv 2021