Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Scaling Policy Gradient Quality-Diversity with Massive Parallelization via Behavioral Variations

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ASCII-ME shows that a critic-free policy-gradient operator can make MAP-Elites fast and parallel-friendly for deep-network policies.

desk verdict Genuinely new non-AC PG variation operator with strong empirical speedups, but the update rule's theoretical basis is shaky and the code is withheld. read the letter →

arxiv 2501.18723 v1 pith:3OHECYTO submitted 2025-01-30 cs.NE cs.AIcs.LGcs.RO

Konstantinos Mitsides , Maxence Faldor , Antoine Cully This is my paper · ORCID
classification cs.NEcs.AIcs.LGcs.RO
keywords quality-diversityMAP-ElitespolicygradientsneuroevolutionmassiveparallelizationbehavioralvariationsreinforcementlearningASCII-ME
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a MAP-Elites-style archive of diverse neural-network policies can be evolved with policy gradients alone, bypassing the actor-critic setup in which a separately trained value-function critic guides updates and that slows existing policy-gradient quality-diversity methods. The proposed operator, ASCII, interpolates between two action sequences using per-time-step reward-to-go differences and maps the result back into policy parameters through the transpose of the policy Jacobian. The paper's claim is that this critic-free signal keeps sample efficiency competitive while making the algorithm about five times faster on average and robust to large batch sizes on a single GPU. In the reported experiments, ASCII-ME builds a diverse archive in under 250 seconds and achieves a 25% higher archive-quality score than the strongest actor-critic baseline on two locomotion tasks.

What carries the argument

The load-bearing object is the performance-based weight matrix $\boldsymbol{Z}_{\boldsymbol{x}_i,\boldsymbol{x}_j} = \mathrm{diag}(z_0(\boldsymbol{x}_i,\boldsymbol{x}_j)\boldsymbol{I}, \ldots, z_{H-1}(\boldsymbol{x}_i,\boldsymbol{x}_j)\boldsymbol{I})$ from Eq. (10). It decides, at every time step, how strongly the parent's action should be pulled toward the target's action: the pull is strong when the target's reward-to-go is higher, the states are similar, and the actions are close, and it is zeroed out by the clipping rule when the parent is already outperforming the target and the actions diverge. The transpose Jacobian $\boldsymbol{J}_{\boldsymbol{x}_i}^T$ then maps this action-space change back to the genotype. Together these pieces replace the trained critic that other policy-gradient QD methods need.

What would settle it

Run ASCII-ME on a locomotion task while injecting a state-distribution shift: make the target policy's evaluation start from states with systematically higher rewards (e.g., downhill or closer to a reward source) while its actions are known to be worse. If ASCII still pulls the parent's actions toward the target's, the reward-to-go comparison is being driven by state luck. A more direct version: at each time step, estimate the true advantage of the target action over the parent action at the same state, and check whether the sign of that advantage agrees with the sign of the weighted $\Delta G_t$; chance-level agreement would falsify the mechanism.

Watch

Extended reading notes

Core claim

At the center of the paper is a new operator, ASCII (Action Sequence Crossover with performance-Informed Interpolation), integrated into MAP-Elites as ASCII-ME. For a parent policy with genotype $\boldsymbol{x}_i$ and a target policy sampled from a buffer, ASCII first computes the imaginary actions the parent would take along the target's states, $\tilde{\boldsymbol{a}}_t = \boldsymbol{\mu}_{\boldsymbol{x}_i}(\boldsymbol{s}^j_t)$, and compares them with the target's actual actions $\boldsymbol{a}^j_t$. A per-time-step weight $z_t$ combines the reward-to-go advantage $\Delta G_t = G^j_t - G^i_t$ with a cosine similarity between states, a squared-exponential action kernel, and a PPO-like clipping gate; the parent genotype is then moved by $\boldsymbol{x}'_i = \boldsymbol{x}_i + \lambda_2 \sum_t z_t(\boldsymbol{a}^j_t - \tilde{\boldsymbol{a}}^i_t) \nabla_{\boldsymbol{x}} \boldsymbol{\mu}_{\boldsymbol{x}}(\boldsymbol{s}^j_t)|_{\boldsymbol{x}_i}$. The claim is that this mechanism substitutes for a critic: it uses the time-step structure of the MDP directly and can be computed in parallel for an entire evaluation batch, so ASCII-ME maintains competitive sample efficiency while scaling to large batch sizes.

Load-bearing premise

The load-bearing premise is that the reward-to-go difference between two policies at the same time step, after the cosine and kernel corrections, indicates which policy's action is better at that step; this can fail because the two rewards-to-go are gathered from different state distributions, so a higher value may reflect a luckier or systematically different state rather than a superior action.

Editorial extensions

If this is right

  • On Hopper Uni and Walker Uni, with a one-million-evaluation budget, ASCII-ME reports a 25% higher archive-quality metric than DCRL-ME while running, on average, about five times faster.
  • Under larger parallel evaluation batches, ASCII-ME's final performance stays nearly flat (mean coefficient of variation around 2% across tasks), whereas the actor-critic baselines degrade (mean CVs of 13% and 8%), so its speed can be scaled up without retuning.
  • On the omnidirectional tasks, ASCII-ME reaches 100% descriptor coverage even though it does not use descriptor information, and the 0%-Iso+LineDD variant finds no solutions, indicating the ASCII operator carries the search in those tasks.
  • A 50/50 mix of ASCII and Iso+LineDD is the most robust configuration; replacing either operator entirely lowers average performance.
  • The paper also notes a boundary: on Ant Uni, the largest task, ASCII-ME's maximum fitness is lower than the actor-critic baselines, and it attributes this to critics making more accurate performance estimates than its direct reward-to-go comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the reward-to-go comparison is used directly, replacing the cosine and kernel corrections with an explicit state-distribution correction would be the natural next test; if it improves ASCII-ME on large state/action spaces, it would confirm that trajectory-comparability is the limiting factor.
  • Editorial inference: ASCII-ME's batch-size stability should carry over to other archive-based QD designs as a plug-in operator, since it has no critic update schedule tied to iteration count; a concrete prediction is that descriptor-conditioned archives would retain the scaling behavior.
  • Editorial inference: The buffer's low-quality trajectories help rather than hurt, which suggests that ASCII benefits from broad state-action coverage, not just elite states; combining ASCII with novelty- or coverage-based sampling from the buffer may improve sample efficiency further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ASCII-ME, a MAP-Elites variant for neuroevolution that replaces part of the genetic mutation operator with a new operator, ASCII. ASCII interpolates between two action sequences, weighting the interpolation at each time step by a reward-to-go difference, a cosine state-similarity term, and an action kernel, and then maps the action-space change to the genotype space via the transpose of the policy Jacobian. The authors claim that this yields a policy-gradient-based, non-actor-critic variation operator that scales with massive parallelization, and they report experiments on five Brax locomotion tasks showing that ASCII-ME achieves competitive or better QD scores than PGA-ME, DCRL-ME, MEMES, PPGA, and MAP-Elites, while being substantially faster and more robust to batch-size increases.

Significance. If the empirical results hold, ASCII-ME is a practically valuable addition to the QD-RL toolbox: it offers a simple, parallelizable, non-AC alternative to current gradient-based QD operators, with strong runtime and sample-efficiency numbers, and it is evaluated with 20 seeds, Holm-Bonferroni-corrected p-values, and a careful batch-size analysis. The paper also provides a useful study of operator synergy and of how AC-based baselines degrade under massive parallelization. However, the theoretical framing of ASCII as a policy-gradient operator is not well supported: the central performance signal in Eq. (11) compares rewards-to-go from different state distributions, and the Jacobian-transpose mapping is a heuristic, so the paper's central claim of a principled non-AC policy-gradient operator is not established.

major comments (4)
  1. [Section 3.3.1, Eq. (11)] The core signal ΔG_t = G^j_t − G^i_t is not a valid action-advantage estimate. G^i_t is the return collected under the mutated policy's own trajectory from state s^i_t, not the return of the action ã_t = μ_i(s^j_t) evaluated at the target policy's state s^j_t. The cosine similarity in Eq. (12) is nonnegative after the max with b and thus can only rescale ΔG_t, not correct for the state mismatch. Consequently, β_t in Eq. (14), and therefore the weight z_t, can have the wrong sign or magnitude whenever the two state sequences differ, which is the generic case. The paper itself acknowledges this in Section 3.3.2 ('To address this discrepancy') and Section 3.3.3 ('To mitigate the inaccuracy'), but the remaining correction is heuristic. The claim in Section 1 and Section 5 that ASCII is a policy-gradient-based operator is therefore not justified by the presented derivation.
  2. [Section 3.2.2, Eq. (6), and Algorithm 1 (ASCII function)] The mapping from action-space changes to genotype changes uses the transpose of the policy Jacobian rather than a proper inverse, so the intended action-space variation is generally not realized. This would be acceptable as a heuristic if it were presented as such, but the paper presents it as the mechanism that 'maps these behavioral changes into the solution space' (Section 1). Moreover, the ASCII loop iterates e=32 times on the same genotype (Algorithm 1), but the archived states and rewards-to-go (s^i_t, G^i_t) are those of the original genotype when it entered the archive; after the first inner update, the Jacobian is evaluated for a policy whose stored trajectory data are stale. This means the repeated Jacobian steps do not correspond to repeated valid policy-gradient steps, and the effect of e on performance is not isolated in any experiment.
  3. [Section 4.4 and Table 3] The optimal batch size for each algorithm is selected using an efficiency score computed from the very same runs that are then used to report the main results in Section 4.5.1. This is a form of selection bias: the comparison is not made at a fixed, pre-specified configuration but at the configuration that maximizes each algorithm's own efficiency-score on the test tasks. For ASCII-ME, the selected batch size (4096) yields noticeably better efficiency scores than the smaller sizes (Table 3), so part of the reported five-times-faster and 25%-higher-QD claims may reflect tuning to the benchmarks rather than an intrinsic advantage. The authors should either report results for a fixed batch size across all algorithms, use a validation set for batch-size selection, or explicitly discuss the optimism introduced by this selection procedure.
  4. [Section 4.5.2 and Appendix A.3] The scalability analysis for AC-based baselines considers three actor-critic training strategies and selects strategy 3 based on the efficiency score, again using the same benchmark runs. This is a reasonable practical choice, but it conflates algorithmic scalability with hyperparameter adjustment. The conclusion that 'ASCII-ME offers straightforward scalability' (Section 4.5.2) is fair, but the claim that AC-based methods 'cannot' scale without performance loss is too strong given that only one of three tested strategies is reported in the main comparison, and the other strategies were aborted on some tasks due to computational cost. A more balanced statement would acknowledge that strategy 1 and 2 achieved higher QD scores in some cases.
minor comments (5)
  1. [Algorithm 1] The pseudocode has a type mismatch: the call `addition(ˆx_i, X, B)` on line 5 does not match the function signature `function addition(X, B, ˆx_i)` defined on line 8. Also, the notation `trajectory𝒙𝒋` in the ASCII function is not defined; it should be clear that this is the trajectory sampled from the buffer B.
  2. [Table 4] Several hyperparameters are listed without a clear mapping to the symbols in the main text: for example, 'Noise variance σ²=4' seems to be the action kernel variance, but the text in Section 3.3.3 uses σ² without connecting it to this table value. Please add explicit cross-references.
  3. [Figure 2, caption] The caption says 'Checkpoints show the number of evaluations completed by each algorithm at that point,' but it is not obvious which markers correspond to which checkpoint values; please make the checkpoints explicit or add annotations to the curves.
  4. [Section 4.5.2] The coefficient-of-variation values (2% and 3% for ASCII-ME and ME) are reported as means across tasks, but the distribution of CVs across tasks and seeds is not shown; a small table or a sentence with the range would make the claim about stability more informative.
  5. [Conclusion, Section 5] The paper states that ASCII-ME is 'the first policy gradient based QD algorithm to operate independently of actor-critic methods.' This is a strong novelty claim that would benefit from a more precise definition of 'policy gradient' in this context, since the ASCII update is not obtained by differentiating a scalar objective with respect to the policy parameters in the standard REINFORCE sense.

Circularity Check

0 steps flagged · score 1.0 of 10

No meaningful circularity: the ASCII update is self-contained and independently evaluated; the main risks are state-mismatch validity and test-set tuning, not circularity.

full rationale

ASCII-ME's update rule (Eqs. 5-15) uses time-step reward-to-go differences between a mutated policy and a target policy, scaled by cosine similarity and an action kernel, then maps the resulting action-sequence change to the genotype via the policy Jacobian. Nothing in these equations is defined in terms of the QD score, coverage, or archive inclusion that the experiments report; offspring are evaluated independently and only then compete for archive cells. No parameter is fitted to the reported QD score and then reported as a prediction; the batch-size selection in Sec. 4.4 uses an efficiency score on the same benchmark, which is a selection-bias concern rather than a circular derivation. The heavy citation of the authors' own QDax/DCRL-ME/MEMES infrastructure is used as benchmark comparison and related work, not as an unverified premise that forces the conclusion. The main methodological vulnerability—Eq. 11 compares rewards-to-go from mismatched state distributions and Eq. 6 uses J^T rather than an inverse—is a correctness/threat-to-validity issue, not circularity, because the update does not presuppose the reported outcome. No self-definitional, fitted-input-as-prediction, self-citation-load-bearing, uniqueness-import, or renamed-ansatz pattern is present. Score 1 due only to the omnipresence of the authors' own prior benchmarks and the tuning-on-test-set caveat, neither of which is load-bearing.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central algorithm depends on several tuned constants, none of which are derived from theory. The strongest nonstandard assumptions are about the comparability of single-episode reward-to-go across policies and about the Jacobian transpose as an inverse. There are no invented entities.

free parameters (6)
  • lambda_2 step size = alpha/(H*sigma^2), alpha=3e-3, sigma^2=4
    Controls the magnitude of the ASCII parameter update; tuned in Appendix A.4.
  • cosine minimum threshold b = 0.25
    Prevents zero or negative state-similarity weights in Eq. (12); tuned.
  • clipping ratio epsilon = 0.8
    Gates which time steps contribute to the update in Eq. (15); tuned.
  • action kernel variance sigma^2 = 4
    Bandwidth of the squared exponential kernel in Eq. (13); tuned.
  • ASCII gradient iterations e = 32
    Number of parameter update steps per offspring in the ASCII loop; tuned.
  • evaluation batch size and GA/PG split = k=4096, k_GA=2048
    Selected by maximizing an efficiency score on the same benchmark suite (Section 4.4, Appendix B); directly affects the runtime and QD-score claims.
assumptions (4)
  • domain assumption Reward-to-go values from different trajectories are comparable at the same time step after cosine and kernel corrections.
    Eq. (11)-(13) treat G^j_t - G^i_t as a per-time-step performance signal even though G^i_t and G^j_t come from different state sequences.
  • ad hoc to paper The transpose of the policy Jacobian is a valid inverse for mapping action-space changes to genotype changes.
    Eq. (6) uses J^T instead of a pseudo-inverse; no justification is given for why this produces the intended action changes.
  • domain assumption Single-episode stored rewards-to-go are reliable enough to guide mutations.
    The archive and buffer store rewards-to-go from one evaluation episode, while tasks have stochastic initial state sampling (Section 4.1).
  • ad hoc to paper The e-step ASCII loop can update the policy without re-evaluating fitness or rewards-to-go between iterations.
    Algorithm 1 recomputes imaginary actions at fixed target states but does not update the reward-to-go comparison during the e iterations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Policy Gradient Quality-Diversity with Massive Parallelization via Behavioral Variations." pith.science (2026). https://pith.science/paper/3OHECYTO

@misc{pith2026250118723,
  author       = {Pith},
  title        = {Pith review of: Scaling Policy Gradient Quality-Diversity with Massive Parallelization via Behavioral Variations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OHECYTO}},
  note         = {Machine review of arXiv:2501.18723}
}
read the original abstract

Quality-Diversity optimization comprises a family of evolutionary algorithms aimed at generating a collection of diverse and high-performing solutions. MAP-Elites (ME), a notable example, is used effectively in fields like evolutionary robotics. However, the reliance of ME on random mutations from Genetic Algorithms limits its ability to evolve high-dimensional solutions. Methods proposed to overcome this include using gradient-based operators like policy gradients or natural evolution strategies. While successful at scaling ME for neuroevolution, these methods often suffer from slow training speeds, or difficulties in scaling with massive parallelization due to high computational demands or reliance on centralized actor-critic training. In this work, we introduce a fast, sample-efficient ME based algorithm capable of scaling up with massive parallelization, significantly reducing runtimes without compromising performance. Our method, ASCII-ME, unlike existing policy gradient quality-diversity methods, does not rely on centralized actor-critic training. It performs behavioral variations based on time step performance metrics and maps these variations to solutions using policy gradients. Our experiments show that ASCII-ME can generate a diverse collection of high-performing deep neural network policies in less than 250 seconds on a single GPU. Additionally, it operates on average, five times faster than state-of-the-art algorithms while still maintaining competitive sample efficiency.

Figures

Figures reproduced from arXiv: 2501.18723 by the authors.

Figure 1
Figure 1. The ASCII-ME method employs two distinct vari [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Main metrics for ASCII-ME and baselines across tasks, with each algorithm running for the duration required for the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. QD score (left) and runtime (right) for ASCII-ME and all baselines with varying batch sizes across tasks, after one [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Accumulated number of solutions added to the archive for Iso+LineDD variation operator and PG variation operator [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: QD score of ASCII-ME with different mutation pro [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: QD score of ASCII-ME and ASCII-ME (Archive) after [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: QD score (left) and runtime (right) of different actor-critic strategies with varying batch sizes across tasks, after one [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 19 canonical work pages

  1. [1]

    Alberto Alvarez, Steve Dahlskog, Jose Font, and Julian Togelius. 2022. Interactive Constrained MAP-Elites: Analysis and Evaluation of the Expressiveness of the Feature Dimensions. IEEE Transactions on Games 14, 2 (2022), 202–211. https: //doi.org/10.1109/TG.2020.3046133

    work page arXiv 2022

  2. [2]

    Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav Sukhatme

    Sumeet Batra, Bryon Tjanaka, Matthew C. Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav Sukhatme. 2024. Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning. arXiv:2305.13795 [cs.LG] https: //arxiv.org/abs/2305.13795

    arXiv 2024

  3. [3]

    Felix Chalumeau, Bryan Lim, Raphael Boige, Maxime Allard, Luca Grillotti, Manon Flageat, Valentin Macé, Arthur Flajolet, Thomas Pierrot, and Antoine Cully. 2023. QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Acceleration. arXiv:2308.03665 [cs.AI] https: //arxiv.org/abs/2308.03665 8

    work page Pith review arXiv 2023

  4. [4]

    Felix Chalumeau, Thomas Pierrot, Valentin Macé, Arthur Flajolet, Karim Be- guir, Antoine Cully, and Nicolas Perrin-Gilbert. 2023. Assessing Quality- Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Prob- lems. arXiv:2211.13742 [cs.NE] https://arxiv.org/abs/2211.13742

    work page Pith review arXiv 2023

  5. [5]

    Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean- Baptiste Mouret. 2020. Quality-Diversity Optimization: a novel branch of sto- chastic optimization. arXiv:2012.04322 [cs.NE] https://arxiv.org/abs/2012.04322

    arXiv 2020

  6. [6]

    Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. 2020. Scal- ing MAP-Elites to deep neuroevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (GECCO ’20, Vol. 2) . ACM, 67–75. https://doi.org/10.1145/3377930.3390217

    work page arXiv 2020

  7. [7]

    Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. 2015. Robots that can adapt like animals. Nature 521, 7553 (May 2015), 503–507. https://doi.org/10.1038/nature14422

    doi:10.1038/nature14422 2015

  8. [8]

    Antoine Cully and Yiannis Demiris. 2017. Quality and Diversity Optimization: A Unifying Modular Framework. arXiv:1708.09251 [cs.NE] https://arxiv.org/abs/ 1708.09251

    arXiv 2017

Show all 48 references
  1. [9]

    Stanley, and Jeff Clune

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune

  2. [10]

    Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    2024 arXiv

  3. [11]

    Maxence Faldor, Félix Chalumeau, Manon Flageat, and Antoine Cully. 2023. MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy. arXiv:2303.03832 [cs.NE] https://arxiv.org/abs/2303.03832

    2023 arXiv

  4. [12]

    Maxence Faldor, Félix Chalumeau, Manon Flageat, and Antoine Cully. 2024. Syn- ergizing Quality-Diversity with Descriptor-Conditioned Reinforcement Learning. arXiv:2401.08632 [cs.NE] https://arxiv.org/abs/2401.08632

    2024 arXiv

  5. [13]

    Manon Flageat, Bryan Lim, and Antoine Cully. 2024. Enhancing MAP-Elites with Multiple Parallel Evolution Strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’24) . ACM, 1082–1090. https: //doi.org/10.1145/3638529.3654089

    2024

  6. [14]

    Smith, and Antoine Cully

    Manon Flageat, Bryan Lim, Luca Grillotti, Maxime Allard, Simón C. Smith, and Antoine Cully. 2022. Benchmarking Quality-Diversity Algorithms on Neu- roevolution for Reinforcement Learning. arXiv:2211.02193 [cs.NE] https: //arxiv.org/abs/2211.02193

    2022 arXiv

  7. [15]

    Fontaine and Stefanos Nikolaidis

    Matthew C. Fontaine and Stefanos Nikolaidis. 2021. Differentiable Quality Diversity. CoRR abs/2106.03894 (2021). arXiv:2106.03894 https://arxiv.org/abs/ 2106.03894

    2021 arXiv

  8. [16]

    Fontaine and Stefanos Nikolaidis

    Matthew C. Fontaine and Stefanos Nikolaidis. 2023. Covariance Matrix Adapta- tion MAP-Annealing. arXiv:2205.10752 [cs.LG] https://arxiv.org/abs/2205.10752

    2023 arXiv

  9. [17]

    Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K

    Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover

  10. [18]

    Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem

    C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. 2021. Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. http://github.com/google/brax

    2021

  11. [19]

    Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. CoRR abs/1802.09477 (2018). arXiv:1802.09477 http://arxiv.org/abs/1802.09477

    2018 arXiv

  12. [20]

    Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. 2017. Aerodynamic Design Exploration through Surrogate-Assisted Illumination. https://doi.org/10. 2514/6.2017-3330

    2017

  13. [21]

    Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. 2018. Data-efficient design exploration through surrogate-assisted illumination. Evol. Comput. 26, 3 (Sept. 2018), 381–410. https://doi.org/10.1162/evco_a_00231

    2018 doi

  14. [22]

    Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. 2019. Are quality diversity algorithms better at generating stepping stones than objective-based search?. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO ’...

    2019 doi

  15. [23]

    Yannakakis

    Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Geor- gios N. Yannakakis. 2019. Procedural Content Generation through Quality Diversity. In 2019 IEEE Conference on Games (CoG) . IEEE, 1–8. https://doi.org/ 10.1109/cig.2019.8848053

    2019

  16. [24]

    Nikolaus Hansen. 2007. The CMA Evolution Strategy: A Comparing Review . Vol. 192. 75–102. https://doi.org/10.1007/3-540-32494-1_4

    2007 doi

  17. [25]

    Vijay Konda and John Tsitsiklis. 1999. Actor-Critic Algorithms. In Advances in Neural Information Processing Systems , S. Solla, T. Leen, and K. Müller (Eds.), Vol. 12. MIT Press. https://proceedings.neurips.cc/paper_files/paper/1999/file/ 6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

    1999

  18. [26]

    Bryan Lim, Maxime Allard, Luca Grillotti, and Antoine Cully. 2022. Accelerated Quality-Diversity for Robotics through Massive Parallelism.CoRR abs/2202.01258 (2022). arXiv:2202.01258 https://arxiv.org/abs/2202.01258

    2022 arXiv

  19. [27]

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. 2021. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. CoRR abs/2108.10470 (20...

    2021 arXiv

  20. [28]

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. http://arxiv.org/abs/1312.5602 arXiv:1312.5602 [cs]

    2013 arXiv

  21. [29]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015 doi

  22. [30]

    Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites. arXiv:1504.04909 [cs.AI] https://arxiv.org/abs/1504.04909

    2015 arXiv

  23. [31]

    Olle Nilsson and Antoine Cully. 2021. Policy Gradient Assisted MAP-Elites. In The Genetic and Evolutionary Computation Conference . Lille, France. https: //doi.org/10.1145/3449639.3459304

    2021

  24. [32]

    Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert

  25. [33]

    Pugh, Lisa B

    Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. 2016. Quality Diversity: A New Frontier for Evolutionary Computation. Frontiers in Robotics and AI 3 (2016). https://doi.org/10.3389/frobt.2016.00040

    2016

  26. [34]

    Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv:1703.03864 [stat.ML] https://arxiv.org/abs/1703.03864

    2017 arXiv

  27. [35]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Ham- bro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. 2024. Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Pro...

    2024 arXiv

  28. [36]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  29. [37]

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approxima- tion. In Advances in Neural Information Processing Systems , S. Solla, T. Leen, and K. Müller (Eds.), Vol. 12. MIT Press. h...

    1999

  30. [38]

    Fontaine, David H

    Bryon Tjanaka, Matthew C. Fontaine, David H. Lee, Aniruddha Kalkar, and Stefanos Nikolaidis. 2023. Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing. arXiv:2210.02622 [cs.RO] https://arxiv.org/abs/2210.02622

    2023 arXiv

  31. [39]

    Fontaine, Julian Togelius, and Stefanos Niko- laidis

    Bryon Tjanaka, Matthew C. Fontaine, Julian Togelius, and Stefanos Niko- laidis. 2022. Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning. CoRR abs/2202.03666 (2022). arXiv:2202.03666 https://arxiv.org/abs/2202.03666

    2022 arXiv

  32. [40]

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv preprint arXiv:2407.17032 (2024)

    2024 arXiv

  33. [41]

    Vassiiis Vassiliades and Jean-Baptiste Mouret. 2018. Discovering the elite hy- pervolume by leveraging interspecies correlation. In Proceedings of the Ge- netic and Evolutionary Computation Conference (Kyoto, Japan) (GECCO ’18) . Association for Computing Machinery, New York, ...

    2018

  34. [42]

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, and Jürgen Schmidhuber

  35. [43]

    efficiency score

    Ronald J. Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Mach. Learn. 8, 3–4 (May 1992), 229–256. https://doi.org/10.1007/BF00992696 9 A SUPPLEMENTARY MATERIALS A.1 ASCII-ME Pseudocode In this section, we provide the ...

    1992 doi

  36. [2011]

    arXiv:1106.4487 [stat.ML] https://arxiv.org/ abs/1106.4487

    Natural Evolution Strategies. arXiv:1106.4487 [stat.ML] https://arxiv.org/ abs/1106.4487

    arXiv

  37. [2017]

    CoRR abs/1707.06347 (2017)

    Proximal Policy Optimization Algorithms. CoRR abs/1707.06347 (2017). arXiv:1707.06347 http://arxiv.org/abs/1707.06347

    2017 arXiv

  38. [2019]

    CoRR abs/1912.02400 (2019)

    Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space. CoRR abs/1912.02400 (2019). arXiv:1912.02400 http://arxiv.org/abs/1912.02400

    2019 arXiv

  39. [2021]

    Nature 590, 7847 (Feb

    First return, then explore. Nature 590, 7847 (Feb. 2021), 580–586. https: //doi.org/10.1038/s41586-020-03157-9

    2021 doi

  40. [2022]

    In Workshop on Agent Learning in Open-Endedness (ALOE) at ICLR 2022

    DIVERSITY POLICY GRADIENT FOR SAMPLE EFFI-CIENT QUALITY- DIVERSITY OPTIMIZATION. In Workshop on Agent Learning in Open-Endedness (ALOE) at ICLR 2022 . virtual, Vatican City. https://hal.science/hal-03753541

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.