Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Training on synthesized evolution histories with execution rewards turns iterative refinement into a learned capability, beating the strongest baseline by 10–24 points absolute and 46.9% relative on algorithm optimization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:10 UTC pith:NDOIN3UK

load-bearing objection Promising training idea for self-evolution via synthesized trajectories and GRPO, but the undefined 'improvement rate' makes the headline numbers uncheckable; worth a revised resubmission, not acceptance as-is. the 4 major comments →

arxiv 2607.21971 v1 pith:NDOIN3UK submitted 2026-07-24 cs.LG cs.AIcs.CL

Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

Shujin Wu , Cheng Qian , Xiusi Chen , Heng Ji This is my paper
classification cs.LG cs.AIcs.CL
keywords self-evolutionmeta-skillsreinforcement learningiterative refinementtest-time scalingexecution-based rewardscompetitive programmingGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MetaEvolve argues that the effectiveness of test-time evolutionary refinement depends on meta-skills—reflecting on a solution's weaknesses, learning from prior attempts, and acting on environment feedback—and that these skills can be trained explicitly rather than left to an external search loop. The paper's route is to synthesize training data from static coding problems: for each program, it samples a few unrelated solutions, sorts them by an execution-based fitness score, and presents them as an 'evolution history' the model must improve upon. Training with reinforcement learning rewards only genuine fitness gains, so the model internalizes iterative refinement as a policy rather than a prompting trick. The reported result is that this transfers far beyond the training distribution, including to numerical algorithm-optimization tasks the model never saw. A sympathetic reader would take the paper to establish that self-evolution behavior is teachable and that dense execution feedback, not just binary correctness, is the key ingredient.

Core claim

On the paper's own terms, the central claim is that iterative self-evolution can be trained as a meta-skill. MetaEvolve constructs a contextual prompt containing a current program, its combined fitness score (correctness times a normalized inverse-runtime term), and a synthesized history of 1–3 prior attempts sorted from weakest to strongest, then fine-tunes the model with GRPO using a reward that is the fitness difference between the new and current program and −1 for stagnation. Across seven coding benchmarks this yields a 10.01% absolute improvement-rate gain over the strongest baseline in-distribution and 24.12% out-of-distribution; on AlgoTune, a suite of numerical algorithm-optimizatio

What carries the argument

The load-bearing object is the synthesized evolution trajectory: rather than collecting real multi-turn rollouts, the pipeline reuses a pool of independently sampled solutions to the same problem, scores them by execution (correctness and average runtime), and sorts them into a fake history of 'prior attempts' ending at the current program. The training reward is the execution-based fitness gain of the newly generated program over the current one, with a hard −1 for stagnation, and the policy is optimized with GRPO, a group-normalized advantage estimator. This design converts a multi-round evolutionary process into single-turn training signals while keeping the feedback dense and verifiable.

Load-bearing premise

The load-bearing premise is that a history made by sorting unrelated solutions to the same problem by fitness is a faithful proxy for real iterative refinement; if the model instead learns to exploit the artificial monotonic ordering—for instance by copying the best history entry—the transfer performance would not reflect a genuine learned meta-skill.

What would settle it

Run the same RL pipeline on real multi-turn rollouts of the model's own attempts on held-out coding problems and compare against the synthesized-history training; if the synthesized variant does not match or beat it, the proxy-history assumption fails. A second, sharper test: strip the current program from the prompt and present only the sorted history; if performance stays high, the model is likely exploiting the ordering rather than improving the given code.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Iterative refinement can be optimized as an explicit objective in post-training, rather than relying on prompting or external search scaffolding.
  • Meta-skills trained with dense execution rewards in coding transfer to open-ended domains where such feedback is scarce, including numerical algorithm optimization.
  • The approach improves not only final scores but solution novelty, suggesting evolution can be pushed toward structural exploration rather than local edits.
  • A correct-to-incorrect seed ratio near 80:20 is preferable; error recovery emerges from mixed training rather than from over-exposing the model to failures.
  • The same data-synthesis scheme can cheaply produce evolution-aware training data from any static corpus where a verifiable scoring function exists.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gains are genuine, a natural next step is to anneal from synthetic histories to real self-generated rollouts, which could compound the improvement and test whether the meta-skill deepens with practice.
  • The dependence on a continuous scoring function suggests the pipeline applies to any optimization domain with an evaluator—compiler optimization, physical simulation tuning, or theorem-proving proof-checkers—where binary success is too coarse.
  • Because Codeforces improvements are small, the method may plateau on problems that demand genuinely new algorithmic ideas; combining trained meta-skills with external memory or search over algorithm templates could be the next frontier.
  • The 80:20 seed-ratio result hints that efficiency refinement is easier to teach than correctness recovery; a curriculum that spikes incorrect examples only late in training might raise recovery rates further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. MetaEvolve proposes to train LLMs for iterative self-evolution by synthesizing multi-round refinement trajectories from static coding data, then applying GRPO with execution-based rewards. The trained model is run inside an AlphaEvolve-style evolutionary search at inference. The paper reports large gains over AlphaEvolve and other baselines on seven coding benchmarks and on AlgoTune, and argues that learned meta-skills transfer to out-of-distribution and open-ended tasks.

Significance. If the reported effects are real, the contribution is significant: it would demonstrate that RL on synthesized evolution trajectories can internalize reflective, feedback-driven revision rather than relying on an external scaffold. Strengths include verifiable execution-based rewards, use of public benchmarks and models, a detailed hyperparameter appendix, and a concrete novelty analysis. However, the current manuscript does not support its headline claims because the central metric is undefined and appears inconsistent across tables, the synthetic-history proxy is unvalidated, and the evaluation protocol lacks uncertainty estimates. These issues are load-bearing for the main thesis, so the contribution cannot yet be accepted on the evidence presented.

major comments (4)
  1. [§3.2.1, Tables 1–2] The key metric, 'improvement rate', is never defined. Table 1 reports 'Avg Impr. Rate' of 1.08–2.76% on 10 TACO problems, while Table 2 reports values of 33.93% (AlphaEvolve) and 50.62% (MetaEvolve) on TACO. If Table 2 uses the same metric, the numbers are irreconcilable; if it uses a different aggregation, the abstract's 10.01% in-distribution and 24.12% out-of-distribution gains (which are averages of Table 2 column differences: e.g., (17.56+16.69+7.07−1.29)/4 ≈ 10.01) rest on an undefined conversion. Tables 5 and 6 use 'improvement rate'/'recovery rate' similarly. The central empirical claim therefore cannot be checked as written.
  2. [§2.1.2, Evolution History Synthesis] The synthesized history is produced by independently sampling 1–3 other responses to the same question and sorting them by fitness. This assumes a set of unrelated solutions ordered by score is a faithful proxy for real iterative refinement, where each step is causally conditioned on the preceding one. No experiment validates this assumption (e.g., comparing against RL on real rollouts, shuffled histories, or histories without monotonic order). If the model learns to copy from the best history entry or exploit the artificial ordering, the claimed meta-skills would not transfer to genuine inference-time evolution. This is load-bearing for the core claim that MetaEvolve cultivates domain-agnostic meta-skills.
  3. [§3.1.1, evaluation protocol] Each benchmark is evaluated on only 50 randomly sampled problems, with no seeds, no standard errors, and no significance tests. Many reported differences in Table 2 are between 7 and 30 percentage points; with 50 problems, a 10-point difference has a standard error of roughly 7 percentage points (for p≈0.5). The paper should report bootstrap CIs or multiple evaluation samples. In addition, the in-distribution benchmarks are drawn from the same sources as the training data (PRIME-RL/Eurus-2-RL-Data), and no question-level deduplication is stated; the in-distribution gains may partly reflect memorization rather than learned meta-skills. The AlgoTune result also rests on only 8 tasks with no variance estimate.
  4. [Eq. (3) vs. Figure 1] Equation (3) defines correctness as binary (1 if all test cases pass, 0 otherwise), but Figure 1's history synthesis shows entries such as 'Correctness: 0 (Pass rate: 40%)' and 'Correctness: 0 (Pass rate: 75%)', implying a continuous pass rate is used as correctness when ranking prior attempts. This ambiguity affects the synthesized histories, the sort order, and the reward computation. Please clarify which correctness definition is used where; the discrepancy is material because the score directly determines training and evaluation.
minor comments (5)
  1. [§3.1.1] 'randomly sample 50 test cases' should be '50 problems'; each problem has many test cases.
  2. [Abstract] The claim of outperforming the strongest baseline by 10.01% absolute is an average over benchmark columns and hides the Codeforces degradation (−1.29 percentage points). The statement should be qualified.
  3. [§2.1.1] The thresholds τ_min/τ_max are described as 'predetermined by analyzing the score distribution' but no method or values are given; provide this for reproducibility.
  4. [Table 4] 'AlgoTune Score' is said to be a harmonic mean, but the formula is not given; specify how speedups are aggregated and confirm that higher is uniformly better.
  5. [Appendix A] The KL penalty coefficient β is set to 0; this should be justified, especially for a 14B model trained for 2 epochs where deviation from the reference policy is possible.

Circularity Check

0 steps flagged

No significant circularity: the central claims are tested on held-out benchmarks and do not reduce to the training objective by construction.

full rationale

METAEVOLVE's training reward (Eq. 4) is the difference in the combined score of Eq. 3, and the same score is used during inference-time evolutionary search. This is an alignment of the optimization objective with the evaluation metric, but it is not circular in the prohibited sense: no parameter is fitted to the evaluation benchmarks, and the headline generalization claims are supported by held-out benchmarks (Atcoder, Leetcode, USACO) and AlgoTune, which are not used in training. The synthesized evolution histories are an acknowledged proxy (prior attempts are independent responses sorted by fitness), but the paper does not derive the evaluation results from that construction; whether this proxy teaches genuine meta-skills is an empirical assumption, not a definitional reduction. Self-citations appear in the references (e.g., Acikgoz et al. 2026; Wang et al. 2024), but only as related work and not as load-bearing evidence for the core claim. The undefined 'improvement rate' is a serious verifiability issue, but it is a measurement/reporting concern rather than evidence that the derivation reduces to its own inputs. No circular step can be exhibited with a quote and an equation-level reduction, so the appropriate score is 0.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The method relies on several hand-set constants (diversity weights, fitness normalization, evolution hyperparameters, data ratio) and on the assumption that synthetic histories faithfully represent real iterative refinement. No new entities are introduced. The counts here show the empirical claims are contingent on many unvalidated design choices.

free parameters (7)
  • Diversity metric weights (Eq. 1) = 0.1, 10, 0.5
    Hand-chosen weights for character-length, line-count, and charset differences; not justified by theory or data.
  • Response selection weights (Eq. 2) = 0.3, 0.7
    Hand-chosen balance of correctness and diversity in the greedy selection.
  • Fitness score constants (Eq. 3) = 1e-6, 1000
    Arbitrary normalization and epsilon to cap the runtime score.
  • Correct-to-incorrect seed ratio = 80:20
    Selected from Table 5 ablation on the evaluation benchmarks themselves.
  • Evolution hyperparameters = rounds=10, sample=20, keepN=5, islands=5, migration=5, ratio=0.6:0.2:0.2
    Tuned on 10 TACO problems; TACO is later reported as a main benchmark.
  • History length N = N∈[1,3]
    Randomly sampled for each training example; no ablation.
  • Stagnation penalty = -1
    Fixed penalty in reward Eq. 4; not varied.
axioms (4)
  • standard math GRPO objective with KL coefficient β=0 is a valid RL update for policy optimization.
    Standard algorithm from Shao et al. 2024, used in Appendix A.
  • domain assumption Program runtime is a meaningful continuous reward signal that reflects solution quality beyond correctness.
    Assumed in §2 and Eq. 3; no validation that faster programs are better for all tasks.
  • ad hoc to paper Synthesized evolution histories from independently sampled responses sorted by fitness approximate true multi-round refinement trajectories.
    Introduced in §2.1.2; the model is trained on fake histories and expected to transfer to real inference-time evolution.
  • domain assumption Meta-skills learned on competitive programming transfer to open-ended tasks outside the training domain.
    Central hypothesis of the paper; tested on AlgoTune but not proven.

pith-pipeline@v1.3.0-alltime-deepseek · 14923 in / 17564 out tokens · 166782 ms · 2026-08-01T06:10:39.711840+00:00 · methodology

0 comments
read the original abstract

Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains. We hypothesize that the success of such evolution frameworks hinges on meta-skills, such as self-reflection with environment feedback, that enable effective multi-round refinement, yet are largely neglected by traditional post-training. To bridge this gap, we present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning (RL), and inference-time evolutionary search. Concretely, we ground MetaEvolve in coding, where program execution provides natural, continuous reward signals beyond binary correctness. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score (combining correctness and efficiency), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution. By training on large-scale code data, we aim to inspire generalizable domain-agnostic meta-skills that can transfer broadly to open-ended problems where such rich training signals are scarce. Across seven coding benchmarks, MetaEvolve outperforms the strongest baseline by 10.01% absolute on in-distribution tasks and 24.12% on out-of-distribution tasks. On open-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46.9% relative improvement. These results demonstrate that explicitly cultivating self-evolution meta-skills offers a principled path toward more capable and autonomously self-evolving AI.

Figures

Figures reproduced from arXiv: 2607.21971 by Cheng Qian, Heng Ji, Shujin Wu, Xiusi Chen.

Figure 1
Figure 1. Figure 1: Overview of the data generation pipeline of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative comparison of reasoning traces produced by AlphaEvolve and [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 22 linked inside Pith

  1. [1]

    Tool-r0: Self-evolving llm agents for tool-learning from zero data.arXiv preprint arXiv:2602.21320,

    Emre Can Acikgoz, Cheng Qian, Jonas H¨ubotter, Heng Ji, Dilek Hakkani-T¨ur, and Gokhan Tur. Tool-r0: Self-evolving llm agents for tool-learning from zero data.arXiv preprint arXiv:2602.21320,

    arXiv

  2. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025a

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025a. Daya Guo, Dejian Yang, He Zhang, Junxiao Song, Runxin Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, X...

    Pith/arXiv arXiv

  3. [8]

    Deepspeed ulysses: System optimiza- tions for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509,

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimiza- tions for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509,

    Pith/arXiv arXiv

  4. [10]

    Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852,

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852,

    Pith/arXiv arXiv

  5. [11]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand `es, and Tatsunori B Hashimoto. s1: Simple test-time scaling. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286–20332,

    2025

  6. [12]

    Alphaevolve: A coding agent for scientific and algorithmic discovery

    Alexander Novikov, Ng ˆan V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131,

    Pith/arXiv arXiv

  7. [14]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn

    URLhttps://arxiv.org/abs/2507.15887. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36,

    arXiv

  8. [16]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y K Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  9. [17]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu

    URL https://github.com/algorithmicsuperintelligence/openevolve. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  10. [18]

    Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952,

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952,

    Pith/arXiv arXiv

  11. [19]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

    Pith/arXiv arXiv

  12. [20]

    Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  13. [21]

    Large language model reasoning failures

    Peiyang Song, Pengrui Han, and Noah Goodman. Large language model reasoning failures. arXiv preprint arXiv:2602.06176,

    arXiv

  14. [22]

    Under review

    12 Preprint. Under review. Boxin Wang, Chankyu Lee, Nayeon Lee, Sheng-Chieh Lin, Wenliang Dai, Yang Chen, Yangyi Chen, Zhuolin Yang, Zihan Liu, Mohammad Shoeybi, et al. Nemotron-cascade: Scaling cascaded reinforcement learning for general-purpose reasoning models.arXiv preprint arXiv:2512.13607, 2025a. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexi...

    arXiv

  15. [23]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations, 2023c. Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baol...

    Pith/arXiv arXiv

  16. [24]

    Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms.arXiv preprint arXiv:2504.14655,

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms.arXiv preprint arXiv:2504.14655,

    Pith/arXiv arXiv

  17. [25]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  18. [27]

    Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446,

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl.arXiv preprint arXiv:2402.19446,

    Pith/arXiv arXiv

  19. [28]

    Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478,

    Pith/arXiv arXiv

  20. [29]

    Fine-tuning language models from human prefer- ences.arXiv preprint arXiv:1909.08593,

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences.arXiv preprint arXiv:1909.08593,

    Pith/arXiv arXiv 1909

  21. [31]

    We enable gradient checkpointing for memory efficiency. B Evolution Ablations We simultaneously evaluate three key hyperparameters: evolution rounds ∈ {5, 10, 15}, number of programs sampled per round ∈ {10, 20, 30}, and keep top N∈ {2, 5, 7}. For each hyperparameter, we vary its value across three candidates while holding the other two fixed, and confirm...

    2017

  22. [1989]

    doi: 10.1137/0218082

    ISSN 0097-5397. doi: 10.1137/0218082. URLhttps://doi.org/10.1137/0218082. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well?arXiv preprint arXiv:2503.24235,

    Pith/arXiv arXiv

  23. [2017]

    Under review

    10 Preprint. Under review. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456,

    Pith/arXiv arXiv

  24. [2019]

    Under review

    13 Preprint. Under review. A GRPO Training Algorithm FormulationWe perform RL using the Group Relative Policy Optimization (GRPO) algo- rithm (Shao et al., 2024): LGRPO(θ) =−E ∑G i=1min πθ(oi|x) πθold(oi|x) ˆAi, clip πθ(oi|x) πθold(oi|x), 1−ϵ, 1+ϵ ˆAi −βDKL[πθ∥πref] (5) where πθ is the current policy being optimized,πθold is the policy from the previous i...

    2024

  25. [2020]

    URLhttps://arxiv.org/abs/2009.10297. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco J R Ruiz, Jordan S Oldfield, Pengming Liao, Nando de Freitas, et al. Mathematical discoveries from program search with large language models.Nature, 625(7995):468–475,

    Pith/arXiv arXiv 2009

  26. [2021]

    Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering

    Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering. arXiv preprint arXiv:2506.09050,

    arXiv

  27. [2022]

    Qwen3-coder-next technical report.arXiv preprint arXiv:2603.00729,

    Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. Qwen3-coder-next technical report.arXiv preprint arXiv:2603.00729,

    Pith/arXiv arXiv

  28. [2023]

    Deltaevolve: Accelerating scientific discovery through momentum-driven evolution.arXiv preprint arXiv:2602.02919,

    Jiachen Jiang, Tianyu Ding, and Zhihui Zhu. Deltaevolve: Accelerating scientific discovery through momentum-driven evolution.arXiv preprint arXiv:2602.02919,

    arXiv

  29. [2024]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    System card: arXiv:2412.16720. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, volume 35,

    Pith/arXiv arXiv

  30. [2025]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307,

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307,

    Pith/arXiv arXiv

  31. [2026]

    Constitu- tional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073,

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073,

    Pith/arXiv arXiv