Pith. sign in

REVIEW 3 major objections 3 minor 20 references

Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that NSR-ES, a quality-diversity variant of OpenAI-ES that averages novelty with fitness, can train a Decision Transformer in the MuJoCo Humanoid environment to final performance comparable to the objective-based…

desk verdict Honest, modest empirical study of novelty-based ES on Decision Transformers; the 'comparable to OpenAI-ES' claim needs a proper baseline and statistical support. read the letter →

arxiv 2502.06301 v2 pith:O5HSNSVZ submitted 2025-02-10 cs.LG cs.NE

Matyáš Lorenc , Roman Neruda This is my paper · ORCID
classification cs.LGcs.NE
keywords evolutionstrategiesnoveltysearchqualitydiversityDecisionTransformerreinforcementlearningMuJoCoHumanoidNSR-ESNS-ES
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether evolution strategies driven by behavioral novelty, rather than purely by task reward, can train transformer-based policies for reinforcement learning. It finds that pure novelty search (NS-ES) makes progress on a Decision Transformer in the MuJoCo Humanoid environment but would need many more iterations to become competitive. The quality-diversity variant NSR-ES, which combines novelty with the task objective, trains the transformer to final-agent performance comparable to the objective-based OpenAI-ES baseline, at roughly three times the number of iterations. Pretraining the transformer through behavior cloning did not accelerate novelty-driven training and was generally harmful. The result matters because transformer policies are about five times larger than the feed-forward networks usually trained with evolution strategies, and it suggests that quality-diversity signals can guide high-dimensional search in the absence of a dense gradient.

What carries the argument

The central machinery is NSR-ES: a metapopulation of Gaussian distributions over the policy-network parameters, a behavior archive that stores each distribution mean's final (x, y) position, novelty computed as the average Euclidean distance to the k nearest archived behaviors, and a combined fitness-and-novelty score that determines which member of the metapopulation to improve. The test architecture is the Decision Transformer, a causal transformer with return-to-go conditioning that predicts actions from a sequence of return, state, and action tokens. The contrast between NS-ES (novelty only) and NSR-ES (novelty plus fitness) isolates the contribution of the objective signal to training a large sequence model.

What would settle it

Run OpenAI-ES and NSR-ES on the same Decision Transformer in MuJoCo Humanoid with identical hyperparameters, population sizes, seeds, and evaluation protocol, then compare final fitness and distance traveled; if the unpublished OpenAI-ES baseline is not reproduced or NSR-ES falls clearly behind it, the central comparability conclusion fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that NSR-ES, a quality-diversity evolution strategy, can train a Decision Transformer policy directly in the MuJoCo Humanoid environment, with final distances traveled comparable to those produced by the objective-based OpenAI-ES baseline, while using a four-times-larger population and about three times more iterations than the feed-forward case. Pure novelty search with NS-ES also trains the transformer, but too slowly to be practical under the tested budget. Pretraining the Decision Transformer via behavior cloning did not speed up novelty-based training and usually hurt performance, whether with NS-ES or NSR-ES. The authors also propose a future transfer method that reuses the behavior archive from a smaller trained population to avoid re-exploring simple behaviors.

Load-bearing premise

The central comparison with OpenAI-ES rests on baseline numbers taken from the authors' unpublished earlier paper, so the claim that NSR-ES matches OpenAI-ES holds only if those earlier runs used the same hyperparameters, population sizes, seeds, and evaluation details as the NSR-ES runs reported here.

Editorial extensions

If this is right

  • If the claim holds, derivative-free training of transformer-based reinforcement-learning policies is feasible without gradients, using quality-diversity search rather than objective-only evolution.
  • NSR-ES offers an exploration bonus over OpenAI-ES at roughly three times the computing cost, which may help in environments with deceptive or sparse rewards where a pure objective signal leads to local optima.
  • Behavior-cloning pretraining of a Decision Transformer does not bootstrap novelty-based evolution, so any speedup for larger models must come from a different transfer mechanism, such as reusing the behavior archive.
  • Pure novelty search alone is not a practical trainer for models of this size under the tested budget, setting a scalability boundary for novelty-only methods.
  • The results suggest that the final-position behavior characteristic is sufficient to train a transformer policy in the Humanoid environment, but richer behavior characteristics may be needed in other domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit is that the advantage of NSR-ES over NS-ES points to the fitness term anchoring the metapopulation in a useful region while novelty prevents premature convergence; directly testing NSR-ES on sparse-reward versions of MuJoCo would probe this mechanism.
  • The failed pretraining suggests that gradient-trained weights place the search in a region where the novelty archive is already exhausted or uninformative; the authors' proposed archive-transfer method is directly testable by reusing the saved behavior archive during a new training run.
  • A neighboring question the paper does not address is whether the result transfers to other behavior characteristics, such as full state trajectories, which would likely matter in environments where final position is uninformative.
  • Because the OpenAI-ES baseline curves come from an unpublished report, an independent replication with published baseline curves is the natural next step before relying on the comparability conclusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper reports experiments with two novelty-based evolution strategies, NS-ES and NSR-ES, applied to feed-forward policies and to Decision Transformer policies in the MuJoCo Humanoid environment, both from scratch and with pretrained initializations. The main positive result is that NSR-ES trains Decision Transformers to final-agent performance that the authors describe as comparable to an OpenAI-ES baseline, whereas NS-ES makes progress but would need more compute and pretraining is reported as unhelpful. The paper is an empirical study with ten runs per condition and includes a code/data repository.

Significance. If the central comparison holds, the paper provides a useful data point: a quality-diversity evolution strategy can scale to transformer-based policies and match an objective-based evolution strategy in final performance, while retaining the diversity-oriented exploration that is the motivation for novelty-based methods. The paper is transparent about negative results, reports ten-run aggregations with median/quartile and percentile intervals, and releases code and data, all of which are strengths. The main risks are the reliance on an unpublished baseline and the asymmetric best-of-metapopulation selection rule in the headline comparison; both are verification issues that can be addressed with additional data and analysis rather than by reworking the core method.

major comments (3)
  1. [Section III.B, Fig. 3] The central claim that NSR-ES achieves Decision Transformer performance comparable to OpenAI-ES is supported only by Fig. 3. The caption states that for algorithms with a metapopulation, 'the agent with the best average distance in the final metapopulation was chosen as a solution,' while for OpenAI-ES the plotted point is the solution of a single Gaussian. This is not an apples-to-apples comparison: selecting the best member of a diverse metapopulation can inflate reported performance relative to a single final mean. Moreover, Fig. 3 shows single bars with no error bars or significance tests, so 'comparable' is not established at the reported level of certainty. Please report the distribution of final metapopulation members across the ten runs (e.g., box or violin plots), state the number of evaluation episodes underlying each point, and make explicit whether the selected solution corresponds to the algorithm's actual output policy or to a post-hoc selection.
  2. [Section III, baseline paragraph; Ref. [16]] The OpenAI-ES baseline and the TD3 reference shown in Fig. 2a are taken from reference [16], which is listed as unpublished. The manuscript does not provide the baseline reward curves, their variance, the exact hyperparameters, the number of workers, or the evaluation protocol for those runs. Since the paper's main positive result is a comparison to that baseline, this is a verification gap rather than a minor citation issue. Please include the full baseline data in the appendix or repository, or weaken the cross-algorithm claims to what can be verified from the present manuscript.
  3. [Section IV, Discussion] The Discussion states that NSR-ES 'requires more computation than OpenAI-ES' and 'only a threefold increase in computation (thrice the iterations),' but Sections III.A and III.B state that for Decision Transformers the population size was quadrupled compared with the feed-forward experiments. If iterations are tripled and the population is quadrupled, the number of function evaluations is about twelve times larger, not three times, unless OpenAI-ES also used the quadrupled population, which is not stated. Please reconcile this cost accounting, since the claim of comparable final performance at a modest compute premium depends on it.
minor comments (3)
  1. [Section II.B] The phrase 'the greater the larger the model employed' is ungrammatical and should be rephrased, for example as 'the stronger this effect, the larger the model'.
  2. [Section I] The sentence 'A fairly simple, yet highly effective examples of such algorithms' should read 'a fairly simple, yet highly effective example'.
  3. [Fig. 2 caption] The caption uses different aggregation methods for evaluation results (median and quartiles) and runtime results (mean and percentile intervals); please justify this choice briefly in the text, since a reader may otherwise infer that run-to-run variance is being masked in one of the two plots.

Circularity Check

1 steps flagged · score 4.0 of 10

The NSR-ES results are new measurements, so there is no equation-level circularity; however, the central 'comparable to OpenAI-ES' claim rests on an OpenAI-ES baseline imported from the authors' own unpublished prior work.

  1. self citation load bearing [Section III, baseline paragraph; reference [16]]
    "All these objective-based results were adopted from an earlier paper [16]. — [16] Matyáš Lorenc. Utilizing evolution strategies to train transformers in reinforcement learning, 2025. unpublished."

    The paper's headline comparison is that NSR-ES reaches final-agent performance 'comparable' to OpenAI-ES on Decision Transformers. The OpenAI-ES side of that comparison is not measured or reported in this paper; the paper states that 'all these objective-based results were adopted from an earlier paper [16]', and reference [16] is an unpublished manuscript by the same first author. Thus the benchmark against which NSR-ES is judged is an input taken from the authors' own prior work, not from the present experiments or from an independent source. The conclusion 'NSR-ES is comparable to OpenAI-ES' therefore depends on accepting a self-citation as the reference measurement; if that baseline is removed or is not directly comparable, the comparison has no supported denominator.

full rationale

The paper is an empirical study, not a derivation: no equation is fitted to define the conclusion, and no predicted quantity is equivalent to its input by construction. The NS-ES and NSR-ES training curves, runtimes, and final distances are new measurements, so the central result retains independent content. The main circularity-adjacent issue is the load-bearing use of reference [16] for the OpenAI-ES baseline: the baseline is the authors' own unpublished prior work, and the paper gives no baseline curves, hyperparameters, or variance for those runs, so the 'comparable to OpenAI-ES' claim cannot be independently checked from the present text. Separately, Figure 3 selects the best final metapopulation member for NS-ES and NSR-ES while OpenAI-ES is represented by a single mean, which can inflate the metapopulation algorithms; this is an evaluation bias rather than a circular step. The Discussion's claim of a 'threefold increase in computation (thrice the iterations)' is also hard to reconcile with the stated fourfold population increase for Decision Transformers, but that affects the cost comparison, not circularity. Overall, the NSR-ES result itself is not forced by construction, so the circularity score is moderate rather than high.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new theoretical quantities or entities are introduced; all components are taken from prior work. The central empirical claim rests on standard benchmark choices, the behavioral characterization, and an unpublished baseline from the authors' own prior study.

free parameters (3)
  • DT population multiplier = 4
    The Decision Transformer is almost five times larger than the feed-forward model, so the authors quadrupled the population size for DT experiments (Sections III-A and III-B). This hand-chosen scaling affects the function evaluation count and thus the comparison to the feed-forward runs.
  • Pretraining learning rate and noise deviation = 0.01
    When seeding training with a pretrained Decision Transformer, both the learning rate and noise deviation were reduced to 0.01 (Sections III-A and III-B). These are hand-picked and affect whether pretraining appears helpful.
  • Desired return-to-go = 7000 (scaled by 1/1000 in MuJoCo)
    Each Decision Transformer episode is conditioned on an initial desired return of 7000, following the original Decision Transformer paper [14]. This is a standard choice, not fitted here, but it does constrain what the trained policy can express.
assumptions (5)
  • domain assumption The MuJoCo Humanoid environment is a suitable representative of continuous control benchmarks.
    Section III states the choice was made because Humanoid is 'the most complex and challenging among the standard ones.' The central claims about NS-ES and NSR-ES scaling to transformers are drawn from this single environment.
  • domain assumption The agent's final (x,y) position is an adequate behavior characteristic for novelty search in Humanoid.
    Section III defines behavior as final x,y coordinates with Euclidean distance, following the standard for MuJoCo in prior literature. The novelty signal driving both algorithms depends entirely on this representation.
  • domain assumption OpenAI-ES baseline results from reference [16] are valid and directly comparable.
    Section III adopts all objective-based results from [16], which is listed as unpublished. The comparability of NSR-ES and OpenAI-ES outcomes is contingent on this unverifiable baseline.
  • domain assumption Decision Transformer hyperparameters from the original paper are appropriate for online training with evolution strategies.
    Section III states the same hyperparameter values as [14] were used for Humanoid; no tuning is reported, so the model's performance under ES is assumed to be unaffected by the choice of this configuration.
  • standard math First-order rescaling approximates the natural gradient in OpenAI-ES.
    Section II-A relies on the known result [18] that dividing by variance approximates natural gradient descent for Gaussian distributions. This is background, not a contribution, and is not tested in the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning." pith.science (2026). https://pith.science/paper/O5HSNSVZ

@misc{pith2026250206301,
  author       = {Pith},
  title        = {Pith review of: Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5HSNSVZ}},
  note         = {Machine review of arXiv:2502.06301}
}
read the original abstract

In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.

Figures

Figures reproduced from arXiv: 2502.06301 by the authors.

Figure 1
Figure 1. Decision Transformer architecture [14] The most important component, to which the transformers owe their success, is a self-attention layer, which is used repeatedly throughout the network. For each input sequence element, the self-attention constructs a "key", a "query", and a "value". Next, an i-th output element is obtained as a linear combination of all values, with each value weighted in proportion to the produ… view at source ↗
Figure 2
Figure 2. OpenAI-ES, NS-ES and NSR-ES used on a simple feed-forward model (FF), a Decision Transformer (DT), [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the average distances that the resulting agents trained by the examined algorithms were able to travel [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

  1. [16]

    Utilizing evolution strategies to train transformers in reinforcement learning, 2025

    Matyáš Lorenc. Utilizing evolution strategies to train transformers in reinforcement learning, 2025. unpublished

    work page 2025

  2. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, 2 edition, 2018

    work page 2018

  3. [2]

    De Jong.Evolutionary Computation

    Kenneth A. De Jong.Evolutionary Computation. The MIT Press, 2016

    work page 2016

  4. [3]

    Friedrich Frommann Verlag, Stuttgart-Bad Cannstatt, Germany, 1973

    Ingo Rechenberg.Evolutionsstrategie — Optimierung technischer Sys- teme nach Prinzipien der biologischen Evolution. Friedrich Frommann Verlag, Stuttgart-Bad Cannstatt, Germany, 1973

    work page 1973

  5. [4]

    Efficacy of modern neuro-evolutionary strategies for continuous control optimization.Fron- tiers in Robotics and AI, 7, 2020

    Paolo Pagliuca, Nicola Milano, and Stefano Nolfi. Efficacy of modern neuro-evolutionary strategies for continuous control optimization.Fron- tiers in Robotics and AI, 7, 2020

    work page 2020

  6. [5]

    Venkate- sha Prasad, and Chris Verhoeven

    Amjad Yousef Majid, Serge Saaybi, Vincent Francois-Lavet, R. Venkate- sha Prasad, and Chris Verhoeven. Deep reinforcement learning versus evolution strategies: A comparative survey.IEEE Transactions on Neural Networks and Learning Systems, 35(9):11939–11957, 2024

    work page 2024

  7. [6]

    Combining evolution and deep reinforcement learning for policy search: A survey.ACM Trans

    Olivier Sigaud. Combining evolution and deep reinforcement learning for policy search: A survey.ACM Trans. Evol. Learn. Optim., 3(3), September 2023

    work page 2023

  8. [7]

    Stanley.Novelty Search and the Problem with Objectives, pages 37–56

    Joel Lehman and Kenneth O. Stanley.Novelty Search and the Problem with Objectives, pages 37–56. Springer New York, New York, NY , 2011

    work page 2011

Show all 20 references
  1. [8]

    Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone.Evol. Comput., 19(2):189–223, jun 2011

    2011

  2. [9]

    Pugh, Lisa B

    Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation.Frontiers in Robotics and AI, 3, 2016

    2016

  3. [10]

    Stanley, and Jeff Clune

    Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. InProceedings of the 32nd International Conferenc...

    2018

  4. [11]

    Evolution strategies as a scalable alternative to reinforcement learning.arXiv, 2017

    Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning.arXiv, 2017

    2017

  5. [12]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  6. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  7. [14]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345, 2021

    2021 arXiv

  8. [15]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. InAdvances in Neural Information Processing Systems, volume 34, pages 1273–1286. Curran Associates, Inc., 2021

    2021

  9. [17]

    Natural evolution strategies.Journal of Machine Learning Research, 15(27):949–980, 2014

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies.Journal of Machine Learning Research, 15(27):949–980, 2014

    2014

  10. [18]

    First-order and second-order variants of the gradient descent in a unified framework

    Thomas Pierrot, Nicolas Perrin-Gilbert, and Olivier Sigaud. First-order and second-order variants of the gradient descent in a unified framework. In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter, editors,Proceedings of the International Conference on Artificia...

    2021

  11. [19]

    MuJoCo: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012

    2012

  12. [20]

    OpenAI Gym.arXiv, 06 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym.arXiv, 06 2016

    2016

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.