{"id":"9a2cf452-1896-431b-b114-051d4dede8d5","arxiv_id":"2502.06301","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NSR-ES, a quality-diversity evolution strategy, trains Decision Transformers on Humanoid comparably to OpenAI-ES, while NS-ES alone learns too slowly and pretraining does not accelerate novelty-based training.","lead":"The authors test two novelty-based evolution strategies, NS-ES and NSR-ES, as trainers for Decision Transformer policies in the MuJoCo Humanoid reinforcement learning environment. They find that NSR-ES can train these large models comparably to reward-driven OpenAI-ES, while pure novelty search NS-ES is much slower and pretraining fails to help.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparing the best NSR-ES metapopulation member against OpenAI-ES's single mean (Fig. 3) biases the 'comparable performance' claim; no error bars or significance tests are given.","rationale":"The reader flagged the unpublished OpenAI-ES baseline (reference [16]) as the weakest assumption, and that is a genuine reproducibility concern. However, the more load-bearing issue is internal to the presented comparison: even with the baseline data in hand, Fig. 3 compares the best member of NSR-ES's metapopulation to OpenAI-ES's single mean solution, which is an asymmetric evaluation that can manufacture apparent parity. Without confidence intervals or significance testing across the ten runs, the headline claim is not robust. I also noted a possible inconsistency in compute accounting: the population was quadrupled for the transformer (Section III.A) yet the Discussion says only a threefold increase in computation relative to OpenAI-ES, suggesting total sample complexity may be 12x, not 3x, unless the OpenAI-ES baseline also used a quadrupled population. This does not contradict the central feasibility claim (NSR-ES can train a DT), but it does change the strength of the 'comparable' statement. The paper is honest about mixed results, provides a GitHub repository with code and data, and the negative pretraining results are clearly reported, so the contribution remains a modest empirical study. For these reasons the conditional verdict stands, but the authors should be asked to re-analyze the comparison symmetrically and supply baseline details.","tokens_in":11392,"tokens_out":6001,"duration_ms":50424,"concrete_test":"Recompute the Fig. 3 comparison using a symmetric evaluation rule: replace the best-of-metapopulation NSR-ES score with the mean (or a fixed randomly chosen member) of each final NSR-ES metapopulation, and add bootstrap 95% confidence intervals across the ten runs for both NSR-ES and OpenAI-ES on Decision Transformers. If OpenAI-ES's DT mean lies outside the NSR-ES interval, the 'comparable' claim fails under fair evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NSR-ES trains Decision Transformers to performance comparable to OpenAI-ES is supported by Fig. 3. But the figure's caption and Section III state that for NSR-ES, 'the agent with the best average distance in the final metapopulation was chosen as a solution' — a best-of-population selection. OpenAI-ES has no metapopulation; its solution is presumably the final mean of a single Gaussian. Comparing the best member of a diverse set against a single mean inflates NSR-ES performance and is not an apples-to-apples comparison. Moreover, Fig. 3 shows only single aggregated bars with no confidence intervals or statistical tests across the ten runs, so the observed 'comparable' could be within noise. The abstract's claim 'performance appears as similar' therefore is not established by the presented evidence even if the unpublished OpenAI-ES baseline from [16] were fully supplied. A secondary inconsistency: Section III.A says the population size was quadrupled for the Decision Transformer, while the Discussion claims only a 'threefold increase in computation (thrice the iterations)' relative to OpenAI-ES; if population was also quadrupled, total function evaluations are ~12x, not 3x, unless OpenAI-ES also used 4x population, which is not stated. This makes the cost of the claimed comparability unclear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports experiments with two novelty-based evolution strategies, NS-ES and NSR-ES, applied to feed-forward policies and to Decision Transformer policies in the MuJoCo Humanoid environment, both from scratch and with pretrained initializations. The main positive result is that NSR-ES trains Decision Transformers to final-agent performance that the authors describe as comparable to an OpenAI-ES baseline, whereas NS-ES makes progress but would need more compute and pretraining is reported as unhelpful. The paper is an empirical study with ten runs per condition and includes a code/data repository.","tokens_in":11669,"tokens_out":4863,"duration_ms":44726,"significance":"If the central comparison holds, the paper provides a useful data point: a quality-diversity evolution strategy can scale to transformer-based policies and match an objective-based evolution strategy in final performance, while retaining the diversity-oriented exploration that is the motivation for novelty-based methods. The paper is transparent about negative results, reports ten-run aggregations with median/quartile and percentile intervals, and releases code and data, all of which are strengths. The main risks are the reliance on an unpublished baseline and the asymmetric best-of-metapopulation selection rule in the headline comparison; both are verification issues that can be addressed with additional data and analysis rather than by reworking the core method.","major_comments":[{"comment":"The central claim that NSR-ES achieves Decision Transformer performance comparable to OpenAI-ES is supported only by Fig. 3. The caption states that for algorithms with a metapopulation, 'the agent with the best average distance in the final metapopulation was chosen as a solution,' while for OpenAI-ES the plotted point is the solution of a single Gaussian. This is not an apples-to-apples comparison: selecting the best member of a diverse metapopulation can inflate reported performance relative to a single final mean. Moreover, Fig. 3 shows single bars with no error bars or significance tests, so 'comparable' is not established at the reported level of certainty. Please report the distribution of final metapopulation members across the ten runs (e.g., box or violin plots), state the number of evaluation episodes underlying each point, and make explicit whether the selected solution corresponds to the algorithm's actual output policy or to a post-hoc selection.","section":"Section III.B, Fig. 3"},{"comment":"The OpenAI-ES baseline and the TD3 reference shown in Fig. 2a are taken from reference [16], which is listed as unpublished. The manuscript does not provide the baseline reward curves, their variance, the exact hyperparameters, the number of workers, or the evaluation protocol for those runs. Since the paper's main positive result is a comparison to that baseline, this is a verification gap rather than a minor citation issue. Please include the full baseline data in the appendix or repository, or weaken the cross-algorithm claims to what can be verified from the present manuscript.","section":"Section III, baseline paragraph; Ref. [16]"},{"comment":"The Discussion states that NSR-ES 'requires more computation than OpenAI-ES' and 'only a threefold increase in computation (thrice the iterations),' but Sections III.A and III.B state that for Decision Transformers the population size was quadrupled compared with the feed-forward experiments. If iterations are tripled and the population is quadrupled, the number of function evaluations is about twelve times larger, not three times, unless OpenAI-ES also used the quadrupled population, which is not stated. Please reconcile this cost accounting, since the claim of comparable final performance at a modest compute premium depends on it.","section":"Section IV, Discussion"}],"minor_comments":[{"comment":"The phrase 'the greater the larger the model employed' is ungrammatical and should be rephrased, for example as 'the stronger this effect, the larger the model'.","section":"Section II.B"},{"comment":"The sentence 'A fairly simple, yet highly effective examples of such algorithms' should read 'a fairly simple, yet highly effective example'.","section":"Section I"},{"comment":"The caption uses different aggregation methods for evaluation results (median and quartiles) and runtime results (mean and percentile intervals); please justify this choice briefly in the text, since a reader may otherwise infer that run-to-run variance is being masked in one of the two plots.","section":"Fig. 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a short empirical study; its main contribution is narrow but potentially acceptable for this venue. The decisive issue is verification: the unpublished baseline in [16] and the best-of-metapopulation selection in Fig. 3 both bear directly on the headline claim. If the authors cannot release the baseline data, I would suggest reframing the contribution around the NS-ES versus NSR-ES comparison and treating the OpenAI-ES comparison as preliminary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honest empirical paper about whether novelty-based ES variants (NS-ES and NSR-ES) can train Decision Transformers. The negative results are reported cleanly, and the positive claim about NSR-ES is plausible but under-supported by the evidence as presented.\n\nWhat's genuinely new: applying NS-ES and NSR-ES to a transformer policy (Decision Transformer) in MuJoCo Humanoid, with ten runs per condition, medians/quartiles, and a replication of the feed-forward baselines. The finding that pretraining hurts both novelty-based algorithms is a useful data point. The paper is clearly written and doesn't oversell: it says NS-ES needs much more compute and that pretraining offers no benefit.\n\nThe main soft spot is the headline comparison. The OpenAI-ES baseline is taken from the authors' unpublished prior work [16]; the current paper gives no reward curves, hyperparameters, or variance for those runs. So 'comparable to OpenAI-ES' cannot be verified from this paper alone. On top of that, Fig. 3 selects the best member of the final metapopulation for NS-ES/NSR-ES, while OpenAI-ES is presumably a single mean, and there are no error bars or significance tests on those bars. That comparison is biased in favor of the novelty-based methods. The stress-test note is on point.\n\nThere is also a minor cost-accounting inconsistency: the Transformer runs quadruple the population size, and the Discussion says 'threefold increase in computation (thrice the iterations).' If both are true, function evaluations are ~12x, not 3x, unless the OpenAI-ES baseline also used a 4x population, which is not stated. That should be corrected or clarified.\n\nNone of these flaws sink the paper. The core observation that NSR-ES trains a Decision Transformer to a reasonable level, and that the DT/feed-forward gap is small, is still useful. It's a modest incremental contribution, not a claim that needs a revolution.\n\nWho it's for: people working on derivative-free RL or quality-diversity algorithms, and anyone thinking about scaling ES to transformer policies. A serious referee could help by asking for the baseline to be made explicit and for the statistical comparison to be tightened. I would send it to peer review, with the expectation of a revision.","headline":"Honest, modest empirical study of novelty-based ES on Decision Transformers; the 'comparable to OpenAI-ES' claim needs a proper baseline and statistical support.","tokens_in":12163,"tokens_out":3420,"would_cite":false,"duration_ms":29409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that NSR-ES, a quality-diversity variant of OpenAI-ES that averages novelty with fitness, can train a Decision Transformer in the MuJoCo Humanoid environment to final performance comparable to the objective-based…","keywords":["evolution strategies","novelty search","quality diversity","Decision Transformer","reinforcement learning","MuJoCo Humanoid","NSR-ES","NS-ES"],"falsifier":"Run OpenAI-ES and NSR-ES on the same Decision Transformer in MuJoCo Humanoid with identical hyperparameters, population sizes, seeds, and evaluation protocol, then compare final fitness and distance traveled; if the unpublished OpenAI-ES baseline is not reproduced or NSR-ES falls clearly behind it, the central comparability conclusion fails.","tokens_in":11212,"feed_emoji":"🦾","tokens_out":4073,"duration_ms":35964,"temperature":0.7,"pith_summary":"This paper asks whether evolution strategies driven by behavioral novelty, rather than purely by task reward, can train transformer-based policies for reinforcement learning. It finds that pure novelty search (NS-ES) makes progress on a Decision Transformer in the MuJoCo Humanoid environment but would need many more iterations to become competitive. The quality-diversity variant NSR-ES, which combines novelty with the task objective, trains the transformer to final-agent performance comparable to the objective-based OpenAI-ES baseline, at roughly three times the number of iterations. Pretraining the transformer through behavior cloning did not accelerate novelty-driven training and was generally harmful. The result matters because transformer policies are about five times larger than the feed-forward networks usually trained with evolution strategies, and it suggests that quality-diversity signals can guide high-dimensional search in the absence of a dense gradient.","feed_headline":"NSR-ES trains transformer policies as well as objective-based ES","feed_subtitle":"Novelty-aware NSR-ES reaches final-agent performance comparable to the reward-only baseline in MuJoCo Humanoid.","key_machinery":"The central machinery is NSR-ES: a metapopulation of Gaussian distributions over the policy-network parameters, a behavior archive that stores each distribution mean's final (x, y) position, novelty computed as the average Euclidean distance to the k nearest archived behaviors, and a combined fitness-and-novelty score that determines which member of the metapopulation to improve. The test architecture is the Decision Transformer, a causal transformer with return-to-go conditioning that predicts actions from a sequence of return, state, and action tokens. The contrast between NS-ES (novelty only) and NSR-ES (novelty plus fitness) isolates the contribution of the objective signal to training a large sequence model.","core_discovery":"On its own terms, the paper establishes that NSR-ES, a quality-diversity evolution strategy, can train a Decision Transformer policy directly in the MuJoCo Humanoid environment, with final distances traveled comparable to those produced by the objective-based OpenAI-ES baseline, while using a four-times-larger population and about three times more iterations than the feed-forward case. Pure novelty search with NS-ES also trains the transformer, but too slowly to be practical under the tested budget. Pretraining the Decision Transformer via behavior cloning did not speed up novelty-based training and usually hurt performance, whether with NS-ES or NSR-ES. The authors also propose a future transfer method that reuses the behavior archive from a smaller trained population to avoid re-exploring simple behaviors.","pith_inferences":["One implication the authors leave implicit is that the advantage of NSR-ES over NS-ES points to the fitness term anchoring the metapopulation in a useful region while novelty prevents premature convergence; directly testing NSR-ES on sparse-reward versions of MuJoCo would probe this mechanism.","The failed pretraining suggests that gradient-trained weights place the search in a region where the novelty archive is already exhausted or uninformative; the authors' proposed archive-transfer method is directly testable by reusing the saved behavior archive during a new training run.","A neighboring question the paper does not address is whether the result transfers to other behavior characteristics, such as full state trajectories, which would likely matter in environments where final position is uninformative.","Because the OpenAI-ES baseline curves come from an unpublished report, an independent replication with published baseline curves is the natural next step before relying on the comparability conclusion."],"forward_implications":["If the claim holds, derivative-free training of transformer-based reinforcement-learning policies is feasible without gradients, using quality-diversity search rather than objective-only evolution.","NSR-ES offers an exploration bonus over OpenAI-ES at roughly three times the computing cost, which may help in environments with deceptive or sparse rewards where a pure objective signal leads to local optima.","Behavior-cloning pretraining of a Decision Transformer does not bootstrap novelty-based evolution, so any speedup for larger models must come from a different transfer mechanism, such as reusing the behavior archive.","Pure novelty search alone is not a practical trainer for models of this size under the tested budget, setting a scalability boundary for novelty-only methods.","The results suggest that the final-position behavior characteristic is sufficient to train a transformer policy in the Humanoid environment, but richer behavior characteristics may be needed in other domains."],"supporting_citations":[{"why":"Introduces the NS-ES and NSR-ES algorithms and provides the feed-forward model and experimental setup this paper extends to transformers.","marker":"[10]"},{"why":"Defines the OpenAI-ES algorithm, its natural-gradient update, and its parallelization scheme that NS-ES and NSR-ES build on.","marker":"[11]"},{"why":"Introduces the Decision Transformer architecture and supplies the Humanoid hyperparameters used in the transformer experiments.","marker":"[14]"},{"why":"Provides the unpublished OpenAI-ES baseline results on Decision Transformers that NSR-ES is compared against.","marker":"[16]"},{"why":"Supplies the natural evolution strategies background that justifies the parameter update rule used by OpenAI-ES and its variants.","marker":"[17]"},{"why":"Establishes the novelty search framework and behavior archive mechanism that NS-ES and NSR-ES rely on for their exploration signal.","marker":"[8]"},{"why":"Provides the MuJoCo physics engine that defines the Humanoid environment used for all experiments.","marker":"[19]"},{"why":"Provides the OpenAI Gym interface through which the environments and evaluation episodes are accessed.","marker":"[20]"}],"fun_headline_variants":["NSR-ES matches classic ES on transformer policies","Quality-diversity evolution scales to transformer RL","Pretrained models don't speed novelty-based training","NSR-ES: Novelty evolution works for Decision Transformers","Pure novelty search too slow for transformer RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison with OpenAI-ES rests on baseline numbers taken from the authors' unpublished earlier paper, so the claim that NSR-ES matches OpenAI-ES holds only if those earlier runs used the same hyperparameters, population sizes, seeds, and evaluation details as the NSR-ES runs reported here.","fun_headline_variants_meta":{"raw":{"variants":["NSR-ES matches classic ES on transformer policies","Quality-diversity evolution scales to transformer RL","Pretrained models don't speed novelty-based training","NSR-ES: Novelty evolution works for Decision Transformers","Pure novelty search too slow for transformer RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1432,"prompt_tokens":824,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":534}},"tokens_in":440,"tokens_out":608,"duration_ms":5965,"temperature":1.0,"reasoning_tokens":534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:52:19.897209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OpenAI-ES and NSR-ES on the same Decision Transformer in MuJoCo Humanoid with identical hyperparameters, population sizes, seeds, and evaluation protocol, then compare final fitness and distance traveled; if the unpublished OpenAI-ES baseline is not reproduced or NSR-ES falls clearly behind it, the central comparability conclusion fails.","supporting_citations":[{"cited_title":"Stanley, and Jeff Clune","cited_arxiv_id":null,"evidence_quote":"Introduces the NS-ES and NSR-ES algorithms and provides the feed-forward model and experimental setup this paper extends to transformers."},{"cited_title":"Evolution strategies as a scalable alternative to reinforcement learning.arXiv, 2017","cited_arxiv_id":null,"evidence_quote":"Defines the OpenAI-ES algorithm, its natural-gradient update, and its parallelization scheme that NS-ES and NSR-ES build on."},{"cited_title":"Utilizing evolution strategies to train transformers in reinforcement learning, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the unpublished OpenAI-ES baseline results on Decision Transformers that NSR-ES is compared against."},{"cited_title":"Natural evolution strategies.Journal of Machine Learning Research, 15(27):949–980, 2014","cited_arxiv_id":null,"evidence_quote":"Supplies the natural evolution strategies background that justifies the parameter update rule used by OpenAI-ES and its variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the novelty search framework and behavior archive mechanism that NS-ES and NSR-ES rely on for their exploration signal."},{"cited_title":"MuJoCo: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Provides the MuJoCo physics engine that defines the Humanoid environment used for all experiments."},{"cited_title":"OpenAI Gym.arXiv, 06 2016","cited_arxiv_id":null,"evidence_quote":"Provides the OpenAI Gym interface through which the environments and evaluation episodes are accessed."}],"review_version":1}