Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots

Alberto Sinigaglia; Gian Antonio Susto; Marco Cal\`i; Niccol\`o Turcato; Ruggero Carli

REVIEW 5 major objections 6 minor 16 references

Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a Soft Actor-Critic policy with Separable Natural Evolution Strategies directly on the true task score improves underactuated robot control beyond RL alone.

desk verdict Useful hybrid RL+ES finetuning study with a real result for acrobot and cartpole, but the pendubot headline number is a best-of-five selection effect; report the means. read the letter →

arxiv 2507.10030 v1 pith:E5LVT4QF submitted 2025-07-14 cs.RO

Marco Calì , Alberto Sinigaglia , Niccolò Turcato , Ruggero Carli , Gian Antonio Susto This is my paper · ORCID

classification cs.RO

keywords ControlSystemsDeepLearningEvolutionaryalgorithmsReinforcementUnderactuatedSoftActor-CriticSeparableNaturalEvolutionStrategySwing-up

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a two-stage recipe for control tasks whose true objective is a sparse, trajectory-level score that is hard to optimize directly with reinforcement learning. First a Soft Actor-Critic (SAC) agent is trained on a dense surrogate reward; then the policy is fine-tuned with the Separable Natural Evolution Strategy (SNES) directly on the true score. On the 2nd AI Olympics RealAIGym benchmark, the resulting EvolSAC controllers reach performance scores of 0.524 (acrobot) and 0.596 (pendubot), above SAC alone (0.504 and 0.567) and above the listed baselines, with only a small drop in robustness. On cartpole, the same fine-tuning cuts average swing-up time from 2.085 s to 1.135 s. The paper's point is that a cheap black-box optimization step can align an RL-trained policy with the objective that actually matters, without sacrificing robustness.

What carries the argument

The load-bearing object is the SNES fine-tuning step applied to all parameters of the SAC policy network. Separable Natural Evolution Strategy is a zero-order optimizer that keeps a separate step size per parameter dimension and updates the parameter vector by sampling Gaussian perturbations; the step sizes adapt through a log-normal rule. Because SNES evaluates whole trajectories, it can optimize the sparse competition score directly, including trajectory-level terms like success flags, which a per-transition reward cannot see. A second mechanism is the noise-injection procedure used during SNES evaluation: the agent samples its greedy action, undoes the tanh transform, adds Gaussian noise, and re-squashes, which keeps the policy stochastic and prevents SNES from overfitting to a single brittle trajectory.

What would settle it

Run the same SNES fine-tuning from SAC checkpoints taken at 500, 1000, and 2000 episodes; if the improvement over the un-finetuned SAC policy shrinks or vanishes as the starting policy worsens, the near-optimal warm start is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that evolutionary fine-tuning directly on the evaluation score repairs the bias introduced by surrogate rewards in deep RL. Starting from a SAC policy trained on a dense reward, the paper applies SNES to the policy parameters and optimizes the raw competition score, with noise added before the tanh squashing to prevent the stochastic policy from collapsing to a deterministic one. In the AI Olympics double-pendulum benchmark this raises the best performance score from 0.504 to 0.524 in the acrobot and from 0.567 to 0.596 in the pendubot, while final score (performance plus robustness) also improves; robustness scores decline only from 0.700 to 0.692 and 0.800 to 0.796. On cartpole, SAC+SNES reduces average swing-up time from 2.085 s to 1.135 s, a 45.6 percent reduction, and TD3+SNES improves similarly, while additional SAC or TD3 training alone gives much smaller gains. The paper's stated assumption is that the SAC policy is already near-optimal, so SNES only has to make local adjustments.

Load-bearing premise

The method depends on the SAC-trained policy already being close to optimal, because SNES fine-tuning only makes local parameter adjustments and cannot escape a poor region of policy space.

Editorial extensions

If this is right

On the AI Olympics double pendulum, EvolSAC outperforms SAC and all listed baselines on performance score for both acrobot (0.524 vs 0.504) and pendubot (0.596 vs 0.567).
On cartpole, SAC+SNES cuts average swing-up time from 2.085 s to 1.135 s, and TD3+SNES from 2.525 s to 1.345 s, while continuing SAC or TD3 training alone yields smaller gains.
Robustness is largely preserved: the acrobot robustness score drops from 0.700 to 0.692 and the pendubot from 0.800 to 0.796, so final competition scores still improve.
Because the fine-tuning phase only needs the policy as a black box, the same recipe should transfer to other model-free and model-based RL algorithms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not run is to use the same two-phase recipe on other trajectory-level metrics, such as maximum energy, maximum velocity, or maximum action, where surrogate rewards are equally biased.
The local-search assumption implies the method will shine when RL already solves the task; for harder tasks, one could test restarting SNES from several SAC checkpoints or interleaving ES with continued RL to escape poor basins.
Treating the added noise during ES evaluation as an explicit robustness regularizer suggests a testable variant that optimizes the competition score plus a robustness penalty in the same ES loop.
The cartpole improvement comes from a low-cost ES refinement; a hardware test would show whether the gains survive real-world dynamics mismatch, since evaluation here is in simulation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is straightforward and worth taking seriously: train SAC on a surrogate reward, then finetune the policy with SNES directly on the sparse competition score. The paper does this cleanly and reports more than most empirical RL papers, including mean and standard deviation over five seeds in Table 2, a comparison against continued SAC/TD3 training, and a robustness metric that is not directly optimized. That alone puts it ahead of the typical competition-paper baseline. What is actually new is the application to the AI Olympics tasks: SAC+SNES beats SAC and the listed baselines on acrobot both in mean performance (0.5249 vs 0.4972) and in the selected best case (0.524 vs 0.504). The cartpole result is also meaningful: average swing-up time drops from 2.085 s to 1.135 s, and the comparison against continued training shows the gain is not just diminishing-returns noise. The robustness scores barely drop, which is a genuinely non-obvious finding given that SNES has no explicit robustness mechanism. The soft spot is the pendubot headline. The paper reports 0.596 for EvolSAC in Table 4, but that is the best of five seeds, selected by highest average of performance and robustness. The mean from Table 2 is 0.5441 ± 0.0288 versus SAC's 0.5207 ± 0.0235. Those overlap heavily; a quick t-test on the reported numbers gives p ≈ 0.19. So the broad claim that evolutionary finetuning significantly improves performance on both double-pendulum configurations rests on the acrobot result plus a selected best case for pendubot. That is not enough. The paper should report per-seed raw scores, confidence intervals, or at least be explicit that the pendubot gain is a selection effect, not a reliable mean improvement. The circularity concern is real but not fatal. SNES directly optimizes the competition score, so higher scores on that metric are expected. What saves the paper is the acrobot mean improvement, the robustness maintenance, and the comparison against continued SAC training, which shows the finetuning does something beyond just extra optimization time. The method is also clearly limited by the near-optimality assumption: if the SAC policy is poor, SNES local search will not escape the basin. The authors state this assumption honestly. Who is this for? Anyone working on RL-to-ES hybrid pipelines or on the AI Olympics benchmark specifically. It deserves a serious referee. The experiments are reproducible in principle, but no code or raw per-seed data is provided, which should be required before acceptance. My recommendation: send it to review, require the authors to report means rather than best-of-N for all headline claims, add significance testing, and release code and per-seed results. The core method is sound and the acrobot result is solid; the pendubot claim needs to be reframed or supported.

Referee Report

5 major / 6 minor

Summary. The paper proposes Evolutionary SAC (EvolSAC), a two-phase method for control of underactuated systems: a SAC policy is first trained on a dense surrogate reward, and its parameters are then fine-tuned with Separable Natural Evolution Strategies (SNES) directly optimizing the sparse task score (swing-up time on cartpole; the RealAIGym AI Olympics performance score on acrobot and pendubot). On cartpole, SAC+SNES reduces average swing-up time from 2.085 s to 1.135 s, versus 1.565 s for continued SAC training. On the double-pendulum benchmark, EvolSAC is reported to raise the acrobot performance score from 0.504 to 0.524 and the pendubot score from 0.567 to 0.596, with small reductions in robustness (0.700 to 0.692 and 0.800 to 0.796), and to exceed listed baselines. The authors claim that evolutionary fine-tuning significantly improves performance while maintaining high robustness, and they explicitly assume that the initial SAC policy is near-optimal.

Significance. If the claims hold, the paper is a useful empirical contribution: the two-phase surrogate-RL/ES scheme is simple, plausible, and directly addresses the mismatch between dense surrogate rewards and sparse trajectory-level evaluation metrics. The cartpole comparison against continued SAC training is a well-chosen control, the acrobot mean improvement (0.4972 ± 0.0088 to 0.5249 ± 0.0085) is statistically credible, and the final competition scores are competitive with official baselines. The authors also deserve credit for explicitly acknowledging the near-optimal-initialization assumption and for reporting robustness even though it is not part of the optimized objective. However, the central statistical evidence on the double-pendulum benchmark is weaker than the abstract suggests: the headline pendubot gain is a best-of-5 selection effect, no significance tests or per-seed data are reported, and the robustness comparison is also selection-based. With per-seed reporting and appropriately tempered claims, this would be a sound empirical paper; as written, the claims outpace the evidence. No code repository is mentioned.

major comments (5)

[Section 4.2, Tables 2 and 4] The headline pendubot improvement (0.596 vs 0.567) is a best-of-5 selection effect. Table 2 reports means over five trained agents: SAC 0.5207 ± 0.0235 and EvolSAC 0.5441 ± 0.0288; a two-sample t-test on these statistics gives t ≈ 1.4 and p ≈ 0.19, i.e., the mean improvement is not significant. Table 4, by contrast, reports the 'optimal agent' selected as the highest average of performance and robustness over the five seeds, so 0.596 is the maximum of a small sample rather than the central tendency; selecting the maximum inflates it more for the higher-variance EvolSAC distribution. Because the abstract claims that evolutionary fine-tuning 'significantly improves agent performance' for the competition tasks, and the pendubot result is one of the two supporting cases, this is load-bearing. Please report per-seed scores and paired differences with confidence intervals or a significance test, and if the pendubot effect is not significant, temper the claim accordingly. For acrobot the mean improvement (0.4972 to 0.5249) is significant, so the issue is specific to pendubot but affects the abstract's general statement.
[Section 4.2, Tables 3 and 4] The robustness-maintenance claim ('minimal reduction in robustness scores, which drop from 0.700 to 0.692 for the acrobot and from 0.800 to 0.796 for the pendubot') is also based on the selected optimal agents, not on the mean over seeds, and no per-seed robustness values or tests are given. In addition, the SAC rows of Tables 3 and 4 (0.504 and 0.567) appear to be the same 'top performance scores' reported in the SAC training paragraph, i.e., also the best of five runs; comparing two selection maxima is a valid protocol only if stated as such and analyzed accordingly. Please supplement Tables 3 and 4 with the mean ± standard deviation robustness results, state explicitly how each entry was selected, and confirm whether the SAC baselines in these tables are best-of-5 values.
[Section 4.2, Table 2] The sentence introducing Table 2 — '5 agents, each trained via SAC with a specific set of hyperparameters' — is ambiguous. If the five agents use different hyperparameter settings, then the mean and standard deviation in Table 2 do not estimate the performance of a fixed configuration, and the comparison with SAC would mix hyperparameter effects with the effect of SNES fine-tuning. If all five use the identical configuration from Table 1, the sentence should say so. This ambiguity matters because the table is the only non-selected evidence for the double-pendulum claim.
[Section 3, noise-injection procedure] The claim that injecting noise of variance σ² during SNES evaluation 'prevents the posterior from collapsing to a Dirac delta distribution' and thereby preserves robustness is asserted but never tested: there is no ablation comparing SNES fine-tuning with and without this noise injection, and σ² itself is measured from a trained SAC trajectory, so it is a data-dependent design choice. Since robustness preservation is a central part of the paper's claim ('while maintaining high robustness'), please add an ablation or, at minimum, per-seed robustness statistics for the variant without noise injection.
[Section 4.2, SNES agent training] The paper correctly acknowledges that the method is contingent on the starting point: 'we assume that the initial SAC agent has approximated a near-optimal policy, with a discrepancy amenable to further optimization via SNES.' This assumption is load-bearing because it limits the approach to local refinement, yet the abstract and introduction frame the method more generally ('in some cases, policies may require refinement'). Please state this limitation explicitly in the abstract and conclusions, and if feasible report one experiment that quantifies how the fine-tuning gain degrades when SNES is initialized from an earlier, less-converged checkpoint.

minor comments (6)

[Section 4.1, Figure 2] The left-column y-axis label is 'returns' while the right column is labeled 'score [s]', but the text reports swing-up times in seconds; please make the axes consistent and state whether the plotted quantity is a rolling window of swing-up time or reward.
[Section 4.1] The cartpole comparison reports average swing-up times (2.085 s to 1.135 s vs continued SAC 1.565 s) for 10 agents but gives no variance, confidence intervals, or test; the effect is large, but a paired or two-sample summary statistic would substantiate the word 'significant'.
[Section 4.2, Table 1] Several standard training details are missing: network architecture, discount factor, replay buffer size, entropy target for SAC, and episode length for the double-pendulum tasks; the learning rate 0.001 appears only in the text.
[Section 3] Clarify whether the σ² noise injection is applied only during SNES candidate rollouts or also at deployment of the final EvolSAC controller.
[Section 4.2] Clarify how the normalized action at ∈ [−1, 1] maps to the physical torque limit τmax = 3 N·m (for example, through a scaling factor).
[Section 4.1] The statement that 'SNES alone does not discover viable solutions within this time frame' would benefit from reporting how many generations and environment evaluations were used in that ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: score improvements are the SNES objective, but robustness and continued-training comparisons supply independent content.

full rationale

The paper contains no derivation chain whose output is equivalent to its input. EvolSAC is an empirical pipeline: SAC trains on a surrogate reward, then SNES optimizes policy parameters directly against the competition score (13). Reporting improved values of that same score is the outcome of the optimizer, not a model-derived prediction; a finite-sample zero-order optimizer can fail, so the positive result is an empirical finding. Independent content exists in the robustness metric, which is not part of the SNES objective, and in the comparison against continued SAC/TD3 training and external baselines. The competition score and baseline numbers are cited from Wiebe et al. (2025), which includes overlapping authors, but that source is an external competition report and is not used to justify a uniqueness claim or to define away the result. The best-of-five-seed selection in Tables 3-4 and the non-significant pendubot mean difference are statistical-reporting concerns, not circularity; no load-bearing self-citation or ansatz-via-citation chain was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method relies on a small set of hand-tuned reward weights and thresholds plus the structural assumption that the SAC policy provides a good starting point. No new physical entities or forces are postulated. The competition score function is taken from the cited competition paper and is not independently derived here.

free parameters (5)

Reward weights α, β, ρ1, ρ2, ϕ1, ϕ2, η = 2, 1, 0.1, 0.02, 0.15, 0.15, 0.02 (Table 1)
Manual hyperparameters balancing energy, torque and smoothness; the central result depends on this tuning.
Height threshold yth = 0.375 m acrobot, 0.35 m pendubot
Hand-chosen regime switch; affects which reward terms apply.
Torque limit τmax = 3.0 N·m
Chosen as a balanced setting after observing worse results at 1.5 and 5.0; directly affects what policies are learnable.
SNES population size and variance = 40, 0.01 (0.02 cartpole)
Set by hand; controls scale of evolutionary search.
SNES noise variance σ² = measured from SAC trajectory
Derived from SAC action distribution, but remains a design choice that shapes the robustness of the finetuned policy.

assumptions (5)

standard math Markov Decision Process formulation and Bellman equations
Used in both RL and ES optimization; standard background.
domain assumption Simulator faithfully represents the real underactuated systems
All evaluations are in simulation (RealAIGym); the paper claims robustness but does not validate on real hardware.
domain assumption Competition score function from Wiebe et al. (2025) is the correct target
Paper optimizes eq. (13) as given by the cited competition rules.
ad hoc to paper Initial SAC policy is near-optimal
Stated in Section 4.2: "we assume that the initial SAC agent has approximated a near-optimal policy, with a discrepancy amenable to further optimization via SNES." If false, local ES search fails.
ad hoc to paper Gaussian noise injection prevents posterior collapse without hurting performance
Introduced in the 'On the agent robustness' paragraph to prevent overfitting to a single trajectory; no independent evidence that this is the minimal or best strategy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots." pith.science (2026). https://pith.science/paper/E5LVT4QF

@misc{pith2026250710030,
  author       = {Pith},
  title        = {Pith review of: Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E5LVT4QF}},
  note         = {Machine review of arXiv:2507.10030}
}

read the original abstract

Deep Reinforcement Learning (RL) has emerged as a powerful method for addressing complex control problems, particularly those involving underactuated robotic systems. However, in some cases, policies may require refinement to achieve optimal performance and robustness aligned with specific task objectives. In this paper, we propose an approach for fine-tuning Deep RL policies using Evolutionary Strategies (ES) to enhance control performance for underactuated robots. Our method involves initially training an RL agent with Soft-Actor Critic (SAC) using a surrogate reward function designed to approximate complex specific scoring metrics. We subsequently refine this learned policy through a zero-order optimization step employing the Separable Natural Evolution Strategy (SNES), directly targeting the original score. Experimental evaluations conducted in the context of the 2nd AI Olympics with RealAIGym at IROS 2024 demonstrate that our evolutionary fine-tuning significantly improves agent performance while maintaining high robustness. The resulting controllers outperform established baselines, achieving competitive scores for the competition tasks.

Figures

Figures reproduced from arXiv: 2507.10030 by the authors.

**Figure 2.** Comparison between SAC + SNES and TD3 + SNES on the cartpole swing-up task. normally distributed, and it’s optimized to reward highentropy distributions. For this reason, the solution found by SAC will already be robust to noise on the action sampled. The SNES algorithm is employed to align the agent’s behavior with the actual reward function rather than the surrogate one. However, optimizing this reward directly c… view at source ↗

**Figure 3.** State and input trajectories of the EvolSAC con [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: State and input trajectories of the EvolSAC con [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[1]

Amadio, F., Dalla Libera, A., Antonello, R., Nikovski, D., Carli, R., and Romeres, D. (2022). Model-based policy search using monte carlo gradient estimation with real systems application. IEEE Transactions on Robotics, 38(6), 3879--3898

work page 2022
[2]

Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6), 26--38

work page 2017
[3]

B \"a ck, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press

work page 1996
[4]

Chatzilygeroudis, K., Rama, R., Kaushik, R., Goepp, D., Vassiliades, V., and Mouret, J.B. (2017). Black-box data-efficient policy search for robotics. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 51--58. IEEE

work page 2017
[5]

Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

work page 2018
[6]

Haarnoja, T. et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML. PMLR

work page 2018
[7]

and Ostermeier, A

Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2), 159--195

work page 2001
[8]

Lillicrap, T.P. et al. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

arXiv 2015

Show all 16 references

[9]

Schaul, T., Glasmachers, T., and Schmidhuber, J. (2011). High dimensions and heavy tails for natural evolution strategies. In Proceedings of the 13th annual conference on Genetic and evolutionary computation

2011
[10]

Silver, D. et al. (2014). Deterministic policy gradient algorithms. In International conference on machine learning. PMLR

2014
[11]

Turcato, N., Libera, A.D., Giacomuzzo, G., Carli, R., and Romeres, D. (2024). Learning control of underactuated double pendulum with model-based reinforcement learning. arXiv preprint arXiv:2409.05811

2024 arXiv
[12]

Wang, Q., Hong, Z., and Zhong, Y. (2022). Learn to swim: Online motion control of an underactuated robotic eel based on deep reinforcement learning. Biomimetic Intelligence and Robotics, 2(4), 100066

2022
[13]

ai olympics with realaigym

Wiebe, F., Turcato, N., Dalla Libera, A., Zhang, C., Vincent, T., Vyas, S., Giacomuzzo, G., Carli, R., Romeres, D., Sathuluri, A., Zimmermann, M., Belousov, B., Peters, J., Kirchner, F., and Kumar, S. (2024). Reinforcement learning for athletic intelligence: Lessons from the 1...

2024
[14]

Wiebe, F., Turcato, N., Libera, A.D., Choe, J.S.B., Choi, B., Faust, T.L., Maraqten, H., Aghadavoodi, E., Calì, M., Sinigaglia, A., Giacomuzzo, G., Romeres, D., kook Kim, J., Susto, G.A., Vyas, S., Mronga, D., Belousov, B., Peters, J., Kirchner, F., and Kumar, S. (2025). Reinf...

2025 arXiv
[15]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence afte...
[16]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Amadio, F., Dalla Libera, A., Antonello, R., Nikovski, D., Carli, R., and Romeres, D. (2022). Model-based policy search using monte carlo gradient estimation with real systems application. IEEE Transactions on Robotics, 38(6), 3879--3898

work page 2022

[2] [2]

Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6), 26--38

work page 2017

[3] [3]

B \"a ck, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press

work page 1996

[4] [4]

Chatzilygeroudis, K., Rama, R., Kaushik, R., Goepp, D., Vassiliades, V., and Mouret, J.B. (2017). Black-box data-efficient policy search for robotics. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 51--58. IEEE

work page 2017

[5] [5]

Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

work page 2018

[6] [6]

Haarnoja, T. et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML. PMLR

work page 2018

[7] [7]

and Ostermeier, A

Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2), 159--195

work page 2001

[8] [8]

Lillicrap, T.P. et al. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

arXiv 2015