Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that evolution strategies can directly search the full parameter space of billion-parameter language models and outperform reinforcement-learning fine-tuning in sample efficiency, robustness across base models, resistance t

desk verdict Full-parameter ES on billion-scale LLMs is a real first and the engineering is solid, but the paper's own appendix shows the Countdown parameter changes are nearly indistinguishable from a random walk, and without a random-walk control the central claim is not yet proven. read the letter →

arxiv 2509.24372 v3 pith:RRELWZBV submitted 2025-09-29 cs.LG cs.AIcs.NE

Xin Qiu , Yulu Gan , Conor F. Hayes , Qiyao Liang , Yinggan Xu , Roberto Dailey , Elliot Meyerson , Babak Hodjat
show 1 more author
Risto Miikkulainen
This is my paper
classification cs.LGcs.AIcs.NE
keywords evolutionstrategiesLLMfine-tuningreinforcementlearningparameter-spaceexplorationrewardhackingsampleefficiencyzeroth-orderoptimizationrun-to-runstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to overturn the belief that evolution strategies cannot scale to modern LLM sizes. It reports the first full-parameter ES fine-tuning of billion-parameter models, with a population of only 30 perturbed copies, no dimensionality reduction, and no backpropagation. Across the Countdown reasoning task, ES beat PPO and GRPO on every tested model from 0.5B to 8B parameters, improving base accuracy by an average of 36.4% versus roughly 18–21% for RL, and reached RL-level performance with less than 20% of the training sample evaluations in most cases. In conciseness tuning, ES achieved better reward–KL tradeoffs than GRPO without any KL penalty, showed less reward hacking, and was more stable across random seeds. The sympathetic reading is that ES is not merely a viable alternative to RL but a different post-training mechanism: it explores a distribution of solutions in weight space rather than sampling actions token by token.

What carries the argument

The load-bearing mechanism is the ES update in Algorithm 2: sample N Gaussian noise vectors in the full weight space, evaluate each perturbed model's reward using greedy decoding, z-score normalize the rewards, and update the weights by the noise-weighted mean reward. In-place, layer-wise perturbation with stored random seeds keeps GPU memory low; parallel processes evaluate the population; and no gradient or backpropagation is ever computed. The paper attributes ES's qualitative advantages to this weight-space perturbation mechanism, which performs an implicit Gaussian convolution over the reward landscape and optimizes a solution distribution rather than a single policy, making isolated re

What would settle it

Rerun the same ES fine-tuning with N=30 and sigma=0.001, but randomly permute the perturbation directions across the reward-ranked population at each iteration so that the noise is independent of reward. If test accuracy improves by roughly the same amount, the update direction carries no reward information and the central optimization claim is refuted. Alternatively, compute the correlation between the ES parameter update and a finite-difference reward gradient on held-out prompts; a near-zero correlation at these settings would also refute the mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that a stripped-down natural evolution strategy can optimize models with billions of parameters by perturbing every weight with centered Gaussian noise, scoring each perturbed model, and stepping along the noise-weighted mean reward. On the Countdown benchmark this update, run with population size 30 and a single fixed set of hyperparameters, improved base models by more than RL methods that received per-model hyperparameter sweeps; ES also worked where RL stalled, such as on the 0.5B model. The authors argue that the advantage comes from parameter-space exploration: one noise draw determines the whole output trajectory, which smooths jagged, long-horizon reward land

Load-bearing premise

The load-bearing premise is that, with only 30 perturbed copies and noise scale 0.001, the ES update is a signal-bearing step toward higher reward rather than a random walk; the paper's own Appendix A.5 shows the Countdown parameter shifts are nearly indistinguishable from a random walk, so if the signal-to-noise ratio is too low, the reported accuracy gains cannot be attributed to ES optimization.

Editorial extensions

If this is right

  • ES fine-tuning becomes practical for tasks with only sparse, outcome-level rewards, since no token-level credit assignment or value model is needed.
  • ES can elicit reasoning behavior from smaller base models where PPO and GRPO make little or no progress, expanding which models can be usefully post-trained.
  • Removing backpropagation from post-training lowers memory requirements and allows population evaluations to be distributed across many GPUs, reducing the hardware bottleneck for very large models.
  • The run-to-run consistency of ES reduces the expected number of training runs needed to obtain a reliable deployed model.
  • Because ES optimizes a distribution of solutions, reward hacking is harder; this suggests a safer default when reward functions are imperfect or incomplete.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: The paper's Appendix A.5 reports that for most Countdown runs the final parameter shifts are nearly indistinguishable from a random walk; the authors attribute this to numerical precision. If that explanation is incomplete, the reported gains might come from selecting a lucky perturbation direction rather than from iterative ES optimization, and the mechanism would need direct verificat
  • Extension: A testable consequence of the paper's argument is that the effective dimension of the update subspace is small; if ES still works with far fewer than 30 seeds or with random perturbation subsets, the practical bottleneck is evaluation budget rather than parameter-count dimension.
  • Extension: Combining ES-style weight perturbation with explicit diversity or entropy penalties could clarify whether the reward-hacking resistance comes from the distribution-level objective or simply from small update steps; this is a natural bridge between ES and existing RL pipelines.
  • Extension: The conciseness result suggests ES may enable fine-tuning on purely internal or behavior-based objectives, such as confidence or semantic consistency, that action-space RL cannot directly optimize because it never modifies internal representations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes that evolution strategies (ES) can be scaled to full-parameter fine-tuning of billion-parameter LLMs, without dimensionality reduction, and reports comparisons with PPO and GRPO on the Countdown reasoning task and on a conciseness task. The central claims are that ES outperforms RL in accuracy, sample efficiency, robustness across base model families and sizes, resistance to reward hacking, and run-to-run stability. Algorithm 2 presents a memory-efficient, seed-based ES implementation, and the experimental section includes seven LLMs, learning curves, behavior analysis, and an appendix studying parameter magnitude shifts.

Significance. If the claims are validated, this is a notable empirical result: it would overturn the common assumption that ES cannot directly search billion-parameter spaces, and it would offer a backpropagation-free alternative to policy-gradient RL in settings with long-horizon, outcome-only rewards. The paper provides a concrete implementation and source code, uses a fixed ES hyperparameter setting across all models, and includes multiple-seed conciseness experiments, which are strengths. However, the main Countdown comparison lacks uncertainty estimates, and the paper's own appendix raises a concern about whether the ES update is distinguishable from a random walk; the evidence is therefore conditional rather than conclusive.

major comments (3)
  1. [A.5, Algorithm 2 (lines 19–25)] The paper reports that parameter magnitude shifts in the Countdown runs are 'nearly indistinguishable from a random walk' (Figures 7–8), yet the central claim is that ES performs reward-guided search. With N=30, σ=0.001, α=5e-4, the per-step signal-to-noise ratio is low, and the paper's 'benevolent' random-walk explanation is not tested. The authors should add a control with reward-unbiased noise (e.g., using random or shuffled rewards) and/or quantify the alignment between the empirical gradient estimate and the true reward gradient. Without such a control, the Countdown accuracy gains in Table 1 cannot be causally attributed to the ES reward signal.
  2. [Table 1, §4.1] Countdown results are reported as single accuracy numbers with no variance across seeds. This is a problem because the paper claims higher run-to-run stability for ES (§4.2, Table 2) and because some margins are modest (e.g., LLaMA-3.2-1B: ES 16.8 vs GRPO(8) 14.5). Please provide multiple seeds (at least three) and report mean ± std or confidence intervals for all entries in Table 1, and for the sample-efficiency comparison in Figure 6.
  3. [Abstract, §4.2, Table 4] The claim of 'reduced susceptibility to reward hacking' is supported by the conciseness experiment, which fine-tunes on only two training prompts (Table 4). This is a very narrow basis for a general behavioral conclusion. The authors should either expand the conciseness task to a more diverse set of prompts/tasks, or temper the abstract and conclusion claims to 'on the studied conciseness task'.
minor comments (5)
  1. [Appendix A.1] The sentence 'with a learning rate α = σ 2' is ambiguous and inconsistent with Table 2, where α = σ/2. Please clarify the intended relationship.
  2. [Figure 6 caption] The caption says 'two model families and six sizes' but Table 1 lists seven models (four Qwen and three LLaMA). Please correct the count.
  3. [§4.2] 'GRPO with β={0.0,0.01} sometimes hacked the reward' — 'sometimes' should be quantified (e.g., number of seeds/checkpoints where reward hacking was observed).
  4. [Equation (1)] The KL approximation is applied to per-token log-probabilities; please clarify whether a sum over tokens is included or whether the formula is per-token.
  5. [A.5] The term 'benevolent' random walk is informal and not operationalized. Suggest replacing it with a quantitative statement or removing it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the paper is an empirical comparison with only passing, non-load-bearing self-citations.

full rationale

The paper's central claims are empirical: ES fine-tuning at billion-parameter scale is compared with PPO/GRPO on Countdown and conciseness tasks, with test-set accuracies and rewards measured directly. The ES update in Algorithm 2 (theta_t,l <- theta_{t-1,l} + alpha * (1/N) * sum_n Z_n * epsilon_{n,l}) is a standard zeroth-order / NES-style estimator adapted from OpenAI ES; it is not fitted to the reported outcomes, and no fitted parameter is renamed as a prediction. ES uses a single fixed hyperparameter set for Countdown and alpha = sigma/2 for conciseness, so the reported numbers are evaluations rather than solutions of the paper's own equations. The only self-citations (Qiu & Miikkulainen, 2024; Meyerson et al., 2024) appear in a future-work sentence and a related-work list; neither is used to justify the central empirical claim. Appendix A.5 is the one passage that could look like a circular rescue: it states that parameter magnitude changes are 'nearly indistinguishable from random walk' and explains them as 'benevolent' with respect to the fine-tuning objective. That is an untested post-hoc interpretation and a missing random-walk control, which bears on correctness and causal attribution, but it is not a circular step: the Countdown gains are established on held-out test sets independently of the random-walk overlay, and the overlay is not an input to Algorithm 2. There is no uniqueness theorem, no ansatz imported by self-citation, and no definitional equivalence between inputs and outputs. Accordingly, no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on a small set of hand-chosen hyperparameters (N=30, sigma, alpha) and domain assumptions about the reward landscape, rather than on additional axioms or invented entities.

free parameters (5)
  • ES population size N = 30
    Chosen for all experiments; not tuned per model. The paper claims 30 suffices for billion-parameter search, but does not justify this choice.
  • ES noise scale sigma (Countdown) = 0.001
    Fixed for all Countdown runs; no sensitivity analysis reported.
  • ES learning rate alpha (Countdown) = 5e-4
    Fixed for all Countdown runs; alpha = sigma/2, but not justified.
  • ES sigma and alpha (Conciseness) = sigma={0.0005,0.001,0.0015}, alpha=sigma^2 per text but alpha=sigma/2 per Table 2
    Contradictory reporting between Appendix A.1 and Table 2; the actual values used are ambiguous.
  • RL baseline hyperparameters (beta, alpha) for PPO/GRPO = Grid-searched per model (4 settings)
    The claim that ES outperforms RL depends on the RL baselines being reasonably tuned; the grid is coarse and the best setting is selected per model.
assumptions (4)
  • domain assumption The ES gradient estimate from N=30 random Gaussian perturbations provides a useful search direction in billion-dimensional parameter space.
    This assumption underlies Algorithm 2; no theoretical or empirical evidence beyond the reported results.
  • domain assumption Greedy decoding gives a deterministic and informative reward signal for fitness evaluation.
    The method evaluates perturbed models with greedy decoding (Section 3.2, item 5). If greedy decoding is not representative, the reward signal could be misleading.
  • domain assumption The pre-trained models are good starting points such that small perturbations (sigma=0.001) yield meaningful reward differences.
    The algorithm relies on the reward landscape being locally smooth enough to correlate with perturbations.
  • standard math Standard probability and statistics (Gaussian noise, z-score normalization) are valid.
    Used throughout the ES algorithm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning." pith.science (2026). https://pith.science/paper/RRELWZBV

@misc{pith2026250924372,
  author       = {Pith},
  title        = {Pith review of: Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRELWZBV}},
  note         = {Machine review of arXiv:2509.24372}
}
read the original abstract

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Figures

Figures reproduced from arXiv: 2509.24372 by the authors.

Figure 1
Figure 1. Mean conciseness reward and mean KL divergence from the base model for each fine￾tuning checkpoint across different learning pa￾rameters. The Pareto front of ES (blue line) is higher and to the left of the GRPO Pareto front (black line) models, indicating that it found bet￾ter tradeoffs. ES discovers these solutions with￾out any KL divergence penalty, suggesting that it represents a distinctly different fine-tuning … view at source ↗
Figure 2
Figure 2. An example of a countdown task interaction. [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. GRPO behavior with different learning rates. (a) GRPO models trained using different [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Reward distributions in fine-tuning for conciseness with different learning rates [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Accuracy Improvement over Base Models with ES vs RL across Model Families. ES [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Training curves of ES and RL across two model families and six sizes in the countdown [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Parameter magnitude shift histograms for the Countdown task in Llama models optimized [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Parameter magnitude shift histograms for the Countdown task in Qwen models optimized [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Parameter magnitude shift histograms in conciseness fine-tuning in Qwen2.5-7B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mathematical perspective on genetic algorithms with optimization guided operators

    cs.NE 2026-06 unverdicted novelty 7.0 of 10

    Presents a query-complexity framework for genetic algorithms with guided operators and shows necessity of multiple operators and tight bounds for diversity in solution pools.

  2. Reward-Aware Population Scaling of Evolutionary Strategies in LLM Fine-Tuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Binary-reward ES collapses at N=2 because of z-score advantage normalization, not because two perturbation pairs are intrinsically too few.

  3. Why can genetic algorithms work in high-dimensional search spaces?

    cond-mat.stat-mech 2026-06 unverdicted novelty 6.0 of 10

    Elitist (1+M) genetic algorithms follow the loss gradient via mutation-selection, slowed only by noise in the effective-rank directions of the Hessian rather than the full parameter count.

  4. PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    PopuLoRA shows that co-evolving populations of LoRA adapters through cross-evaluated self-play can outperform compute-matched single-agent baselines on multiple code and math reasoning benchmarks.

  5. ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    ESSAM matches PPO and GRPO accuracy (~78%) on GSM8K math tasks but uses 10-18x less GPU memory and shows stronger generalization across datasets.

  6. Goal-Conditioned Supervised Learning for LLM Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    GCSL reframes LLM fine-tuning as supervised pursuit of quality thresholds using natural-language goals, outperforming SFT and DPO on toxicity, code, and recommendation tasks.

Reference graph

Works this paper leans on

83 extracted references · 7 canonical work pages · cited by 6 Pith papers

  1. [1]

    GPT -4 technical report

    Josh Achiam et al. GPT -4 technical report. arXiv:2303.08774, 2024

    arXiv 2024

  2. [2]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr...

    work page doi:10.18653/v1/2021.acl-lon 2021

  3. [3]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7 0 (2): 0 195--204, 2025. doi:10.1038/s42256-024-00975-8. URL https://doi.org/10.1038/s42256-024-00975-8

    doi:10.1038/s42256-024-00975-8 2025

  5. [5]

    Introducing Claude 4, 2025

    Anthropic. Introducing Claude 4, 2025. URL https://www.anthropic.com/news/claude-4

    2025

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  7. [7]

    Understanding pre-training and fine-tuning from loss landscape perspectives

    Huanran Chen, Yinpeng Dong, Zeming Wei, Yao Huang, Yichi Zhang, Hang Su, and Jun Zhu. Understanding pre-training and fine-tuning from loss landscape perspectives. arXiv:2505.17646, 2025. URL https://arxiv.org/abs/2505.17646

    arXiv 2025

  8. [8]

    On the weaknesses of reinforcement learning for neural machine translation

    Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1eCw3EKvH

    2020

Show all 83 references
  1. [9]

    Back to basics: benchmarking canonical evolution strategies for playing atari

    Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. Back to basics: benchmarking canonical evolution strategies for playing atari. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, pp.\ 1419–1426. AAAI Press, 2018. ISBN 9780999241127

    2018

  2. [10]

    Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents

    Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, ...

    2018

  3. [11]

    Bowman, Ethan Perez, and Evan Hubinger

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in...

    2024 arXiv

  4. [12]

    Knowledge fusion by evolving weights of language models

    Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. Knowledge fusion by evolving weights of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: AC...

    2024 doi

  5. [13]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, 2024

    2024

  6. [14]

    Reward shaping to mitigate reward hacking in RLHF

    Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in RLHF . arXiv:2502.18770, 2025. URL https://arxiv.org/abs/2502.18770

    2025

  7. [15]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective ST ars

    Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective ST ars. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=QGJ9ttXLTy

    2025

  8. [16]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, vol...

    2023

  9. [17]

    Deep learning, volume 1

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016

    2016

  10. [18]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities., 2025

    Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities., 2025. URL https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf

    2025

  11. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948, 2025 a . URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  12. [20]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning

    Daya Guo et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645 0 (8081): 0 633--638, 2025 b . doi:10.1038/s41586-025-09422-z. URL https://doi.org/10.1038/s41586-025-09422-z

    2025 doi

  13. [21]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024...

    2024

  14. [22]

    Completely derandomized self-adaptation in evolution strategies

    Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9 0 (2): 0 159--195, 2001. doi:10.1162/106365601750190398

    2001 doi

  15. [23]

    When evolution strategy meets language models tuning

    Bo Huang, Yuxin Jiang, Mingyang Chen, Yi Wang, Hongyang Chen, and Wei Wang. When evolution strategy meets language models tuning. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st Interna...

    2025

  16. [24]

    Neuroevolution for reinforcement learning using evolution strategies

    Christian Igel. Neuroevolution for reinforcement learning using evolution strategies. In Proceedings of the 2003 Congress on Evolutionary Computation, pp.\ 2588--2595, 2003

    2003

  17. [25]

    Do we need to verify step by step? rethinking process supervision from a theoretical perspective

    Zeyu Jia, Alexander Rakhlin, and Tengyang Xie. Do we need to verify step by step? rethinking process supervision from a theoretical perspective. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=4BfaPHfhJ0

    2025

  18. [26]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  19. [27]

    Derivative-free optimization for low-rank adaptation in large language models

    Feihu Jin, Yifan Liu, and Ying Tan. Derivative-free optimization for low-rank adaptation in large language models. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32: 0 4607–4616, October 2024. ISSN 2329-9290. doi:10.1109/TASLP.2024.3477330. URL https://doi.org/10.1109/TASLP.20...

    2024

  20. [28]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.or...

    2015 arXiv

  21. [29]

    Fine-tuning chatgpt for automatic scoring

    Ehsan Latif and Xiaoming Zhai. Fine-tuning chatgpt for automatic scoring. Computers and Education: Artificial Intelligence, 6: 0 100210, 2024. ISSN 2666-920X. doi:https://doi.org/10.1016/j.caeai.2024.100210. URL https://www.sciencedirect.com/science/article/pii/S2666920X24000110

    2024

  22. [30]

    Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O. Stanley. Es is more than just a traditional finite-difference approximator. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '18, pp.\ 450–457, New York, NY, USA, 2018. Association for Computing Mach...

    2018

  23. [31]

    Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution Through Large Models, pp.\ 331--366. Springer Nature Singapore, Singapore, 2024. ISBN 978-981-99-3814-8. doi:10.1007/978-981-99-3814-8_11. URL https://doi.org/10.1007/978-981-...

    2024 doi

  24. [32]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv:2412.19437, 2024

    2024 arXiv

  25. [33]

    Sparse me ZO : Less parameters for better performance in zeroth-order LLM fine-tuning, 2025

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse me ZO : Less parameters for better performance in zeroth-order LLM fine-tuning, 2025. URL https://openreview.net/forum?id=4Kw4KAoVnx

    2025

  26. [34]

    Utilizing evolution strategies to train transformers in reinforcement learning

    Matyáš Lorenc and Roman Neruda. Utilizing evolution strategies to train transformers in reinforcement learning. arXiv:2501.13883, 2025. URL https://arxiv.org/abs/2501.13883

    2025 arXiv

  27. [35]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing ...

    2023

  28. [36]

    Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K

    Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K. Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. ACM Trans. Evol. Learn. Optim., 4 0 (4), November 2024. doi:10.1145/3694791. URL https://doi.org/10.1145/3694791

    2024 doi

  29. [37]

    What is artificial superintelligence?, 2023

    Tim Mucci and Cole Stryker. What is artificial superintelligence?, 2023. URL https://www.ibm.com/think/topics/artificial-superintelligence

    2023

  30. [38]

    Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushme...

    2025 arXiv

  31. [39]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  32. [40]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  33. [41]

    Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz

    Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018. URL https://openreview.net/...

    2018

  34. [42]

    Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space

    Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024

    2024

  35. [43]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems,...

    2023

  36. [44]

    Rechenberg

    I. Rechenberg. Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Problemata (Stuttgart). Frommann-Holzboog, 1973. ISBN 9783772803741. URL https://books.google.com/books?id=-WAQAQAAMAAJ

    1973

  37. [45]

    Sebastian Risi and Kenneth O. Stanley. Deep neuroevolution of recurrent and discrete world models. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, pp.\ 456–462, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. ...

    2019

  38. [46]

    Pawan Kumar, Emilien Dupont, Francisco J

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with larg...

    2024 doi

  39. [47]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  40. [48]

    u ckstie , Martin Felder, and J \

    Thomas R \"u ckstie , Martin Felder, and J \"u rgen Schmidhuber. State-dependent exploration for policy gradient methods. In Walter Daelemans, Bart Goethals, and Katharina Morik (eds.), Machine Learning and Knowledge Discovery in Databases, pp.\ 234--249, Berlin, Heidelberg, 2...

    2008

  41. [49]

    u ckstie , Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and J \

    Thomas R \"u ckstie , Frank Sehnke, Tom Schaul, Daan Wierstra, Yi Sun, and J \"u rgen Schmidhuber. Exploring parameter space in reinforcement learning. Paladyn, 1 0 (1): 0 14--24, 2010. doi:10.2478/s13230-010-0002-4. URL https://doi.org/10.2478/s13230-010-0002-4

    2010 doi

  42. [50]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp.\ 2234–2242, Red Hook, NY, USA, 2016. Curra...

    2016

  43. [51]

    Evolution strategies as a scalable alternative to reinforcement learning

    Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017. URL https://arxiv.org/abs/1703.03864

    2017 arXiv

  44. [52]

    How well can a genetic algorithm fine-tune transformer encoders? a first approach

    Vicente Ivan Sanchez Carmona, Shanshan Jiang, and Bin Dong. How well can a genetic algorithm fine-tune transformer encoders? a first approach. In Shabnam Tafreshi, Arjun Akula, Jo \ a o Sedoc, Aleksandr Drozd, Anna Rogers, and Anna Rumshisky (eds.), Proceedings of the Fifth Wo...

    2024 doi

  45. [53]

    Approximating kl divergence, 2020

    John Schulman. Approximating kl divergence, 2020. URL http://joschu. net/blog/kl-approx. html, 2020

    2020

  46. [54]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  47. [55]

    Numerische Optimierung von Computermodellen mittels der Evo-lutionsstrategie, volume 26

    Hans-Paul Schwefel. Numerische Optimierung von Computermodellen mittels der Evo-lutionsstrategie, volume 26. 01 1977. ISBN 9783764308766. doi:10.1007/978-3-0348-5927-1

    1977 doi

  48. [56]

    Parameter-exploring policy gradients

    Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23 0 (4): 0 551--559, 2010. ISSN 0893-6080. doi:https://doi.org/10.1016/j.neunet.2009.12.004. URL https://www.sciencedi...

    2010 doi

  49. [57]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  50. [58]

    Sara Mahdavi, Joelle Barral, Dale Webster, Greg S

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y ...

    2023 arXiv

  51. [59]

    PRMB ench: A fine-grained and challenging benchmark for process-level reward models

    Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. PRMB ench: A fine-grained and challenging benchmark for process-level reward models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting ...

    2025 doi

  52. [60]

    J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37 0 (3): 0 332--341, 1992. doi:10.1109/9.119632

    1992 doi

  53. [61]

    A technical survey of reinforcement learning techniques for large language models

    Saksham Sahai Srivastava and Vaneet Aggarwal. A technical survey of reinforcement learning techniques for large language models. arXiv:2507.04136, 2025. URL https://arxiv.org/abs/2507.04136

    2025 arXiv

  54. [62]

    Stanley, and Jeff Clune

    Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv:1712.06567, 2017. URL https://api....

    2017 arXiv

  55. [63]

    BBT v2: Towards a gradient-free future with large language models

    Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. BBT v2: Towards a gradient-free future with large language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural L...

    2022 doi

  56. [64]

    Black-box tuning for language-model-as-a-service

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on M...

    2022

  57. [65]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018

    2018

  58. [66]

    Fine-tuning mt5-based transformer via cma-es for sentiment analysis

    Orlando Grabiel Toledano-López, Julio Madera, Hector González, Alfredo Simón-Cuevas, Thomas Demeester, and Erik Mannens. Fine-tuning mt5-based transformer via cma-es for sentiment analysis. In Manuel Montes y Gómez, Julio Gonzalo, Francisco Rangel, Marco Casavantes, Miguel Áng...

    2022

  59. [67]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023

    2023 arXiv

  60. [68]

    Solving math word problems with process- and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275, 2022. URL https://arxiv.org/abs/2211.14275

    2022 arXiv

  61. [69]

    Andrew Bagnell

    Anirudh Vemula, Wen Sun, and J. Andrew Bagnell. Contrasting exploration in parameter and action space: A zeroth order optimization perspective. In Proceedings of 22nd International Conference on Artificial Intelligence and Statistics (AISTATS '19), March 2019

    2019

  62. [70]

    When large language models meet evolutionary algorithms: Potential enhancements and challenges

    Chao Wang, Jiaxuan Zhao, Licheng Jiao, Lingling Li, Fang Liu, and Shuyuan Yang. When large language models meet evolutionary algorithms: Potential enhancements and challenges. Research, 8: 0 0646, 2025. doi:10.34133/research.0646. URL https://spj.science.org/doi/abs/10.34133/r...

    2025 doi

  63. [71]

    Natural evolution strategies

    Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp.\ 3381--3387, 2008. doi:10.1109/CEC.2008.4631255

    2008

  64. [72]

    Natural evolution strategies

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15 0 (27): 0 949--980, 2014. URL http://jmlr.org/papers/v15/wierstra14a.html

    2014

  65. [73]

    Bloomberggpt: A large language model for finance

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv:2303.17564, 2023

    2023 arXiv

  66. [74]

    Evolutionary computation in the era of large language model: Survey and roadmap

    Xingyu Wu, Sheng-Hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Evolutionary computation in the era of large language model: Survey and roadmap. IEEE Transactions on Evolutionary Computation, 29 0 (2): 0 534--554, 2025. doi:10.1109/TEVC.2024.3506731

    2025

  67. [75]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report. arXiv:2501.15383, 2025

    2025 arXiv

  68. [76]

    Xingwen Zhang, Jeff Clune, and Kenneth O. Stanley. On the relationship between the openai evolution strategy and stochastic gradient descent. arXiv:1712.06564, 2017. URL https://arxiv.org/abs/1712.06564

    2017 arXiv

  69. [77]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv:2501.07301, 2025. URL https://arxiv.org/abs/2501.07301

    2025 arXiv

  70. [78]

    Genetic prompt search via exploiting language model probabilities

    Jiangjiang Zhao, Zhuoran Wang, and Fangchun Yang. Genetic prompt search via exploiting language model probabilities. In Edith Elkind (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , pp.\ 5296--5305. International Joi...

    2023 doi

  71. [79]

    DPO meets PPO : Reinforced token optimization for RLHF

    Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. DPO meets PPO : Reinforced token optimization for RLHF . In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=IfWKVF6LfY

    2025

  72. [80]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  73. [81]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  74. [82]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  75. [83]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.