Pith. sign in

REVIEW 4 major objections 4 minor 68 references

Scaling Automatic Research Agents via World Models

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WMRL replaces real environment execution with a world model for RL training of research agents, cutting compute 3–4x and still beating full-execution RL.

desk verdict WMRL is a serious, well-thought method paper that plausibly cuts execution cost with a world model plus online debiasing, but the central convergence guarantee rests on a monotone-bias assumption the paper never validates. read the letter →

arxiv 2608.12564 v1 pith:ESSK5AHQ submitted 2026-08-12 cs.LG

Xiyuan Yang , Sheikh Sarwar , Jingru Cheng , Zhan Shi , Duanshun Li , Huiyuan Chen , Haiyang Zhang , Chenlei Guo
show 2 more authors
Jingrui He Zhenyu Liao
This is my paper · ORCID
classification cs.LG
keywords worldmodelsreinforcementlearningpost-trainingautomaticresearchagentsrewardbiascorrectionisotonicregressioncalibrationinverse-variancefusiongroup-relativepolicyoptimizationtrainingcomputebottleneck
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the true cost bottleneck in RL post-training of automatic-research agents is environment execution, not generation: generation batches across trajectories while each candidate solution must run in an isolated sandbox, so execution cost grows linearly and dominates training as trajectories scale. To remove that bottleneck, WMRL replaces execution with a language-model world model that predicts the outcome of a solution in a few forward passes, and it pays for the replacement by correcting the world model's bias and noise. The correction uses a small anchor stream—roughly ten percent of groups are graded by real execution—feeding a monotone recalibration map and an inverse-variance fusion of the two reward streams. The paper reports 3–4x less training compute at 4B and 9B agent scales while matching or exceeding real-execution GRPO on held-out tasks, and it proves in Theorem 4 that the bias term contracts and the variance term shrinks. If true, this turns expensive real execution from a per-trajectory requirement into a small, auditable stream that scales with generation.

What carries the argument

The load-bearing object is the anchor signal: a small fraction of each batch's groups are scored by both the world model and real execution, producing paired scores that drive two corrections. Online Debiasing uses isotonic regression to fit the monotone map that recasts world-model scores, exploiting Assumption 6 that the bias is a monotone distortion of the true score, $b(\tau)=\varphi(r(\tau))-r(\tau)$; Inverse-Variance Denoising fuses the two gradient estimate streams with inverse-variance weights, achieving the harmonic-mean variance of the two streams. These two mechanisms turn the permanent bias floor of Theorem 3 into a term that contracts as $1/(1+T/T_0)$ and divide the variance term by $1+V_{\mathrm{WM}}/V_E$ in the convergence bound.

What would settle it

Train WMRL with a world model whose bias is deliberately constructed to be non-monotone—for instance, one that reverses the ordering of two solution families while preserving a third—and measure whether the residual bias term fails to contract as anchor pairs accumulate; a second check is to track the measured residual $\hat{\eta}^2$ under a fast-changing policy and see whether the anchor weight saturates at its bound, in which case WMRL falls back to real-execution GRPO and the advertised compute saving disappears.

Watch

Extended reading notes

Core claim

The central claim is that an imperfect world model can replace the real environment in RL for AutoResearch agents without sacrificing final performance, provided its reward errors are corrected online. The paper models the world model's score as true score plus a bias and zero-mean noise, and shows the bias alone would leave a permanent $O(B^2)$ floor in the convergence bound while the noise inflates the variance term. Its two corrections remove that floor and shrink that variance: Online Debiasing fits a monotone map by isotonic regression on score pairs from anchor groups, turning the bias term into a contracting quantity, and Inverse-Variance Denoising fuses the anchor and world-model gradient estimates so the update variance is strictly lower than either stream alone. The resulting Theorem 4 guarantees convergence to the same optimum as real-execution RL, and the experiments confirm the speedup and the performance match at two scales and on a second domain.

Load-bearing premise

The load-bearing premise is that the world model's scoring bias is a monotone distortion of the true score that isotonic regression can learn from the accumulating anchor pairs, and that this distortion does not drift faster than the recalibration can track; if the bias is non-monotone or drifts quickly as the policy changes, the contraction guarantee collapses.

Editorial extensions

If this is right

  • Training compute for AutoResearch RL drops by 3.1–3.4x at the two scales tested, and leaderboard percentiles on held-out tasks rise, not fall, relative to real-execution GRPO.
  • Post-trained 4B and 9B agents outperform 48B and 120B open-weight agents on the same held-out evaluations, suggesting post-training on corrected world-model rewards can substitute for raw scale.
  • With both corrections active, the bias-induced error floor of world-model RL vanishes as training proceeds, so asymptotically the method converges to the same optimum as real-execution RL.
  • The inverse-variance fusion makes every gradient update strictly less noisy than either the anchor stream or the world-model stream alone.
  • The recipe transfers outside AutoResearch to embodied vision-language-action post-training, where a reward model predicting task success anchors a sparse environment signal and improves overall success rate by 3.8 points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same anchor-and-correct pattern should generalize to any setting where rewards are expensive to obtain but predictable from agent artifacts—real robotics, web agents, or wet-lab workflows—provided a thin ground-truth stream can be kept alive; the paper's theory does not depend on the AutoResearch specifics.
  • When execution outcomes depend on irreducible randomness invisible in the solution, the tracked residual $\hat{\eta}^2$ stays large and the anchor weight rises, so WMRL degrades gracefully toward full real-execution RL; one testable prediction is that the compute savings shrink exactly in those domains.
  • The monotone-distortion assumption implies a crisp failure mode worth probing: construct a world model whose bias reverses the ordering of two solution families; the paper's contraction guarantee should break, and the residual-bias term should stop shrinking even as anchor pairs accumulate.
  • Because the fusion weight is second-order insensitive to misspecification, the most consequential engineering choice is the single warmup measurement that fixes the variance-ratio scale; varying that measurement across runs would test how stable the 3–4x result is under that choice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes World Model RL (WMRL), a method for post-training AutoResearch agents that replaces expensive real-environment execution with a fast, batched world-model reward signal, while retaining a thin stream of real execution on roughly 10% of training groups (the anchor signal). Two corrections are introduced: Online Debiasing, which fits a monotone isotonic map from predicted to real scores on accumulated anchor pairs, and Inverse-Variance Denoising, which fuses the anchor and world-model gradient streams with inverse-variance weights. The authors prove convergence bounds showing that the permanent bias floor O(B^2) of training on world-model rewards contracts as O(B^2/(1+T/T0)) under a monotone-distortion assumption, and that the variance term is reduced by a harmonic-factor improvement. Experiments on MLE-Dojo, DSBench, and LIBERO-Long report a 3-4x reduction in GPU-hours while matching or exceeding real-execution GRPO, and ablations show both corrections contribute.

Significance. If the central claims hold, WMRL is a practically valuable recipe for scaling RL in settings where execution, not generation, is the bottleneck, and the paper is unusually transparent about its assumptions, proofs, prompts, and benchmark splits. The theoretical appendix contains real derivations under explicit regularity and monotone-bias assumptions, and the empirical protocol is described in enough detail to be reproducible. The significance is tempered by the fact that the main theoretical improvement and the main empirical performance gains rest on assumptions and measurements that are not validated in the manuscript: the monotonicity/stationarity of the world-model bias, and the statistical reliability of single-run benchmark numbers.

major comments (4)
  1. [Appendix B.3, Assumption 6 and Lemma 11] The contraction of the bias term in Theorem 4 is load-bearing and depends entirely on the unvalidated Assumption 6 that the world-model bias is a monotone distortion b(τ)=φ(r(τ))-r(τ) with a fixed non-decreasing φ, together with the coverage and separation conditions of Lemma 11 (q>0, Δφ>0). The world model in this paper is a prompted LLM sharing the agent's backbone, and its systematic error for a trajectory can plausibly depend on the solution code, API calls, task type, and many other features beyond the true score r(τ); if E[r̂|τ] is not a function of r(τ), no monotone map from r̂ to r can remove the bias, and the permanent O(M^2B^2) floor of Theorem 3 remains. The paper neither measures nor reports any diagnostic for these conditions. I ask the authors to provide a concrete empirical test of monotonicity (e.g., plotting anchor pairs and checking whether the fitted map is monotone, or measuring the residual non-monotonicity) and to discuss how φ varies across training steps; without this, the central theoretical claim is unsupported.
  2. [Section 5.2, Table 1] The central empirical claim that WMRL 'matches or exceeds' real-execution GRPO rests on single point estimates with no error bars, no multiple seeds, and no confidence intervals. With avg@8 and leaderboard percentile scores, differences of 1.2 to 4.8 points (e.g., Table 1, Qwen3.5-4B Ours vs GRPO on Img: 5.2 vs 4.3; Table 3 gains of 2.9-4.8 points) may well be within run-to-run noise. Please report at least three independent runs (or seeds) for the main comparisons, with means and standard errors, and state the number of evaluation repetitions used to compute avg@8. The same applies to Table 2 and Table 3, since the 'exceeding performance' statement is a key advertised contribution.
  3. [Section 3.3, Remark 8; Appendix D] The claim that the fusion rule carries 'no free parameter' is not supported by the actual implementation. The weight depends on c=1/η̂²_cal measured from a single warmup run, on an anchor fraction of about 10%, and on the clipping bound w_max=4 introduced in Appendix D. Theorem 4 is stated with exact variances, but training uses plug-in estimates; Remark 8 only shows local second-order insensitivity of the variance around the true weight, not robustness of the estimated c across runs. The paper should report the variability of c across warmup runs and show a sensitivity analysis for the anchor fraction and w_max, or explicitly list these as tuned parameters.
  4. [Theorem 4, Corollary 7] The claimed 'strictly improved' variance term requires more than the stated conditions of Theorem 4. The variance term of Theorem 4 uses the post-recalibration noise, while the variance term of Theorem 3 uses the raw world-model noise; the comparison is only guaranteed to be smaller if the recalibration map does not inflate noise, which is added separately in Corollary 7 as non-expansiveness. This assumption is not validated for isotonic regression with interpolation, and without it the variance comparison between the two theorems is not established by the proof as written. Please state explicitly which assumptions are needed for the term-by-term comparison and justify the non-expansiveness step, or weaken the claim accordingly.
minor comments (4)
  1. [Section 5.1, Table 1] The Nemotron-120B row contains the unreadable entry '18.87.620.5'; this appears to be a missing-space formatting error for '18.8 7.6 20.5' and should be fixed.
  2. [Section 3.3 and Appendix D] The 'warmup phase' used to estimate η̂²_cal is never defined precisely; please state its length in steps and how the single calibration measurement is obtained.
  3. [Appendix B.3, Lemma 11] The coverage condition q is stated as a lower bound on the fraction of anchor pairs per score level, but the anchor-pair distribution changes as the policy improves; please clarify whether q is assumed to hold uniformly over time and how it could be checked in practice.
  4. [Appendix C] The paper transparently describes the manual train/test split and the exclusions (audio tasks, billion-word-imputation, 15 DSBench tasks). I would still like the main text to state that the held-out sets are a filtered subset, since the abstract's 'held-out benchmarks' phrasing may overstate the generality.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WMRL's convergence theorems are derived from stated assumptions and its empirical comparisons use external benchmarks.

full rationale

The paper's derivation chain is self-contained rather than circular. Theorem 3 and Theorem 4 are proved in Appendix B from Assumptions 2 and 6 together with auxiliary lemmas (Lemmas 9-12), and the bounds are obtained by explicit algebra and concentration arguments, not by assuming the conclusion. The Online Debiasing mechanism fits an isotonic map on anchor score pairs, and Lemma 11 supplies a genuine generalization bound (with a 1/sqrt(t) rate) showing the calibrated score's systematic error shrinks; this is a nontrivial statistical result, not a restatement of the fitting objective. The Inverse-Variance Denoising rule is justified by the standard optimal-linear-fusion Lemma 12, whose harmonic-variance conclusion is derived, not assumed. The plug-in fusion weight uses a warmup measurement (c := 1/eta_hat^2_cal), but this is a calibrated constant and Remark 8 explicitly shows the fused variance is second-order insensitive to its error; the convergence theorem itself is stated with exact variances and does not reduce to the fitted value. The anchor stream supplies independent ground-truth labels, and the main empirical claims are measured against held-out MLE-Dojo, DSBench, and LIBERO tasks, not against the training objective. The only self-citation (reference [52], co-authored by Z. Liao) appears in a related-work contrast and is not load-bearing. The main weakness is that the contractive-bias guarantee depends on Assumption 6 (monotone, stationary score-scale bias) and the coverage/separation conditions of Appendix B.3, which are not empirically validated; that is a correctness and robustness risk, not a circularity, because the theorem does not assume the bias term contracts, it proves contraction under explicit conditions.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on four modeling assumptions about the world model and the RL objective: regularity of the score landscape, monotone distortion of the world model bias, finite separated score levels with coverage, and stationarity of the bias. The free parameters are the anchor rate, the calibrated variance-ratio constant, and the anchor weight bound, which are hand-set or estimated from one run. No new physical or mathematical entities are introduced.

free parameters (3)
  • Anchor group fraction = about 10%
    Section 3.3 chooses about 10% of groups to be graded by real execution. It sets the cost/performance tradeoff and no sensitivity analysis is given.
  • c = 1/η^2_cal (variance-ratio scale) = 0.96 in one run
    Section 3.3 and Appendix D calibrate the conversion constant from a single warmup residual; the fusion weight ρ depends on it, despite the text claiming no free parameter.
  • w_max (anchor weight bound) = 4
    Appendix D clips the anchor weight to [1,4] for stability; this is a hand-set regularization of the fused gradient.
assumptions (5)
  • domain assumption The score objective J is L-smooth and satisfies gradient domination, and the policy-gradient norm is bounded by M (Assumption 2).
    Section 4.1 states this as the regularity needed for all convergence bounds; it is not verified for AutoResearch reward landscapes.
  • ad hoc to paper The world model bias is a monotone distortion of the true score and the noise is bounded and independent (Assumption 6).
    Section 4.3 and Appendix B.3 impose this so isotonic regression can learn the bias. If false, Online Debiasing fails.
  • ad hoc to paper The benchmark scores form finitely many ordered levels with positive coverage q and positive separation Δφ (Lemma 11).
    Appendix B.4 needs these for the parametric recalibration rate; they are not guaranteed for continuous leaderboard percentiles.
  • domain assumption The bias and noise statistics are stationary enough that cumulative anchor pairs track the current distortion.
    The contraction proof accumulates pairs over time under a fixed φ; the paper does not model policy-induced distribution shift changing the distortion.
  • standard math Policy gradient identity E[r(τ)∇log πθ(τ)] = ∇J(θ) and independence of trajectories within a GRPO group.
    Used in Lemma 10(i) and the GRPO estimator; standard but not proven in full.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Automatic Research Agents via World Models." pith.science (2026). https://pith.science/paper/ESSK5AHQ

@misc{pith2026260812564,
  author       = {Pith},
  title        = {Pith review of: Scaling Automatic Research Agents via World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESSK5AHQ}},
  note         = {Machine review of arXiv:2608.12564}
}
read the original abstract

Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, which demonstrates the generalizability of our method.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 32 canonical work pages

  1. [1]

    The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    arXiv 2024

  2. [2]

    Towards an AI co-scientist.arXiv preprint arXiv:2502.18864, 2025

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an AI co-scientist.arXiv preprint arXiv:2502.18864, 2025

    arXiv 2025

  3. [3]

    Agent laboratory: Using LLM agents as research assistants.arXiv preprint arXiv:2501.04227, 2025

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants.arXiv preprint arXiv:2501.04227, 2025

    arXiv 2025

  4. [4]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  5. [5]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  6. [6]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  7. [7]

    Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    2023

  8. [8]

    Augmenting large language models with chemistry tools.Nature Machine Intelligence, 6(5):525–535, 2024

    Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools.Nature Machine Intelligence, 6(5):525–535, 2024

    work page 2024

Show all 68 references
  1. [9]

    An autonomous laboratory for the accelerated synthesis of novel inorganic materials.Nature, 624:86–91, 2023

    Nathan J Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E Kumar, Tanjin He, David Milsted, Matthew J McDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, et al. An autonomous laboratory for the accelerated synthesis of novel inorganic materials.Nature, 624:86–91, 2023

    2023

  2. [10]

    AIDE: AI-driven exploration in the space of code.arXiv preprint arXiv:2502.13138, 2025

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: AI-driven exploration in the space of code.arXiv preprint arXiv:2502.13138, 2025

    2025 arXiv

  3. [11]

    DS-Agent: Automated data science by empowering large language models with case-based reasoning

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-Agent: Automated data science by empowering large language models with case-based reasoning. InInternational Conference on Machine Learning, 2024

    2024

  4. [12]

    MLAgentBench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. InInternational Conference on Machine Learning, 2024

    2024

  5. [13]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  6. [14]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 12 Scaling Automatic Research Agents via World Models

    2024 arXiv

  7. [15]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  8. [16]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DayaGuo, DejianYang,HaoweiZhang,JunxiaoSong, RuoyuZhang, RunxinXu, QihaoZhu, ShirongMa, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, 2023

    2023

  10. [18]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information P...

    2024

  11. [19]

    MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering.arXiv preprint arXiv:2505.07782, 2025

    Rushi Qiang, Yuchen Zhuang, Yinghao Li, Dingu Sagar V K, et al. MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering.arXiv preprint arXiv:2505.07782, 2025

    2025 arXiv

  12. [20]

    SWE-World: Building software engineering agents in docker-free environments.arXiv preprint arXiv:2602.03419, 2026

    Shuang Sun, Huatong Song, Lisheng Huang, Jinhao Jiang, Ran Le, Zhihao Lv, Zongchao Chen, Yiwen Hu, WenyangLuo, WayneXinZhao, YangSong, HongtengXu, TaoZhang, andJi-RongWen. SWE-World: Building software engineering agents in docker-free environments.arXiv preprint arXiv:2602.03419, 2026

    2026

  13. [21]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  14. [22]

    Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    2025

  15. [23]

    Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131, 2025

    Alexander Novikov, Ngân V˜u, Marvin Eisenberger, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131, 2025

    2025 arXiv

  16. [24]

    The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025

    2025 arXiv

  17. [25]

    Autodata: An agentic data scientist to create high quality synthetic data.arXiv preprint arXiv:2606.25996, 2026

    Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, and Jason Weston. Autodata: An agentic data scientist to create high...

    2026 arXiv

  18. [26]

    Many ai analysts, one dataset: Navigating the agentic data science multiverse.Proceedings of the National Academy of Sciences, 123(29):e2606495123, 2026

    Martin Bertran, Riccardo Fogliato, and Zhiwei Steven Wu. Many ai analysts, one dataset: Navigating the agentic data science multiverse.Proceedings of the National Academy of Sciences, 123(29):e2606495123, 2026

    2026

  19. [27]

    Frontis-MA1: Training an AI4AI model towards recursive self-improvement in machine learning engineering.arXiv preprint arXiv:2607.28568, 2026

    Junlin Yang, Che Jiang, Yu Fu, Tianwei Luo, Can Ren, Weizhi Wang, Kaikai Zhao, Hongyi Liu, et al. Frontis-MA1: Training an AI4AI model towards recursive self-improvement in machine learning engineering.arXiv preprint arXiv:2607.28568, 2026

    2026 arXiv

  20. [28]

    First steps toward automated AI research.https://www.recursive.com/articles/ first-steps-toward-automated-ai-research, 2026

    Recursive. First steps toward automated AI research.https://www.recursive.com/articles/ first-steps-toward-automated-ai-research, 2026. 13 Scaling Automatic Research Agents via World Models

    2026

  21. [29]

    MLE-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024

    arXiv

  22. [30]

    DSBench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703, 2024

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, et al. DSBench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703, 2024

    2024 arXiv

  23. [31]

    MLGym: A new framework and benchmark for advancing AI research agents.arXiv preprint arXiv:2502.14499, 2025

    Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, et al. MLGym: A new framework and benchmark for advancing AI research agents.arXiv preprint arXiv:2502.14499, 2025

    2025 arXiv

  24. [32]

    SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, 2024

    2024

  25. [33]

    SWE-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, 2024

    2024

  26. [34]

    OpenHands: An open platform for AI software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. OpenHands: An open platform for AI software developers as generalist agents. InInternational Conference on Learning Representations, 2025

    2025

  27. [35]

    Training software engineering agents and verifiers with SWE-Gym

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. InInternational Conference on Machine Learning, 2025

    2025

  28. [36]

    HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, et al. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  29. [37]

    QUEST: Training frontier deep research agents with fully synthetic tasks.arXiv preprint arXiv:2605.24218, 2026

    Jian Xie, Tianhe Lin, Zilu Wang, Yuting Ning, Yuekun Yao, Tianci Xue, Zhehao Zhang, Zhongyang Li, Kai Zhang, Yufan Wu, Shijie Chen, Boyu Gou, Mingzhe Han, Yifei Wang, Vint Lee, Xinpeng Wei, Xiangjun Wang, Yu Su, and Huan Sun. QUEST: Training frontier deep research agents with ...

    2026 arXiv

  30. [38]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  31. [39]

    POPE: Learning to reason on hard problems via privileged on-policy exploration.arXiv preprint arXiv:2601.18779, 2026

    Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. POPE: Learning to reason on hard problems via privileged on-policy exploration.arXiv preprint arXiv:2601.18779, 2026

    2026

  32. [40]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  33. [41]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as- a-judge with MT-bench and chatbot arena. InAdvances in Neural Information Process...

    2023

  34. [42]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InInternational Conference on Machine Learning, 2024

    2024

  35. [43]

    Cosmos world foundation model platform for physical AI.arXiv preprint arXiv:2501.03575, 2025

    NVIDIA. Cosmos world foundation model platform for physical AI.arXiv preprint arXiv:2501.03575, 2025

    2025 arXiv

  36. [44]

    Generating code world models with large language models guided by monte carlo tree search

    Nicola Dainese, Matteo Merler, Minttu Alakuijala, and Pekka Marttinen. Generating code world models with large language models guided by monte carlo tree search. InAdvances in Neural Information Processing Systems, 2024

    2024

  37. [45]

    Worldcoder, a model-based LLM agent: Building world models by writing code and interacting with the environment

    Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based LLM agent: Building world models by writing code and interacting with the environment. InAdvances in Neural Information Processing Systems, 2024

    2024

  38. [46]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 10835–10866, 2023

    2023

  39. [47]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. InInternational Conference on Learning Representations, 2024

    2024

  40. [48]

    Ahmad Ajalloeian and Sebastian U. Stich. On the convergence of SGD with biased gradients.arXiv preprint arXiv:2008.00051, 2020

    2008 arXiv

  41. [49]

    Transforming classifier scores into accurate multiclass probability estimates

    Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002

    2002

  42. [50]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321–1330, 2017

    2017

  43. [51]

    DualDICE: Behavior-agnostic estimation of dis- counted stationary distribution corrections

    Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of dis- counted stationary distribution corrections. InAdvances in Neural Information Processing Systems, 2019

    2019

  44. [52]

    Off-policy reinforcement learning with optimistic exploration and distribution correction

    Jiachen Li, Shuo Cheng, Zhenyu Liao, Huayan Wang, William Yang Wang, and Qinxun Bai. Off-policy reinforcement learning with optimistic exploration and distribution correction. InDeep Reinforcement Learning Workshop, NeurIPS, 2022

    2022

  45. [53]

    ML-Agent: ReinforcingLLMagentsforautonomousmachinelearningengineering.arXivpreprintarXiv:2505.23723, 2025

    Zexi Liu, Jingyi Chai, Xinyu Zhu, Shuo Tang, Rui Ye, Bo Zhang, Lei Bai, and Siheng Chen. ML-Agent: ReinforcingLLMagentsforautonomousmachinelearningengineering.arXivpreprintarXiv:2505.23723, 2025

    2025 arXiv

  46. [54]

    AceGRPO: Adaptive curriculum enhanced group relative policy optimization for autonomous machine learning engineering

    Yuzhu Cai, Zexi Liu, Xinyu Zhu, Cheng Wang, Yanfeng Wang, and Siheng Chen. AceGRPO: Adaptive curriculum enhanced group relative policy optimization for autonomous machine learning engineering. arXiv preprint arXiv:2602.07906, 2026

    2026 arXiv

  47. [55]

    Synthetic sandbox for training machine learning engineering agents.arXiv preprint arXiv:2604.04872, 2026

    Yuhang Zhou, Lizhu Zhang, Yifan Wu, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao, and Hong Yan. Synthetic sandbox for training machine learning engineering agents.arXiv preprint arXiv:2604.04872, 2026. 15 Scaling Automatic Research Agents via World Models

    2026 arXiv

  48. [56]

    LIBERO: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. InAdvances in Neural Information Processing Systems, 2023

    2023

  49. [57]

    Qwen3.5.https://qwen.ai/blog?id=qwen3.5, 2026

    Qwen Team. Qwen3.5.https://qwen.ai/blog?id=qwen3.5, 2026

    2026

  50. [58]

    MiniVLA: A better VLA with a smaller footprint.https://github

    Suneel Belkhale and Dorsa Sadigh. MiniVLA: A better VLA with a smaller footprint.https://github. com/Stanford-ILIAD/openvla-mini, 2024

    2024

  51. [59]

    Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang

    Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang. Robometer: Scaling general-purpose rob...

    2026 arXiv

  52. [60]

    OpenVLA: An open-source vision-language- action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language- action model. InConference on Robot Learning, 2024

    2024

  53. [61]

    Physical Intelligence, Kevin Black, Noah Brown, et al.π0.5: a vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    2025 arXiv

  54. [62]

    Latent reasoning VLA: Latent thinking and prediction for vision-language-action models

    Shuanghao Bai, Jing Lyu, Wanqi Zhou, Zhe Li, Dakai Wang, Lei Xing, Xiaoguang Zhao, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Badong Chen, and Shanghang Zhang. Latent reasoning VLA: Latent thinking and prediction for vision-language-action models. InInternational Conference on M...

    2026

  55. [63]

    Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47:235–256, 2002

    Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47:235–256, 2002

    2002

  56. [64]

    Information-theoretic considerations in batch reinforcement learning

    Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, 2019

    2019

  57. [65]

    Risk bounds in isotonic regression.The Annals of Statistics, 30(2):528–555, 2002

    Cun-Hui Zhang. Risk bounds in isotonic regression.The Annals of Statistics, 30(2):528–555, 2002

    2002

  58. [66]

    MLE-bench: Evaluating machine learning agents on machine learning engineering.https: //openai.com/index/mle-bench/, 2024

    OpenAI. MLE-bench: Evaluating machine learning agents on machine learning engineering.https: //openai.com/index/mle-bench/, 2024. 16 Scaling Automatic Research Agents via World Models Appendix A Notation 18 B Proofs 18 B.1 Warm-up: convergence of standard RL. . . . . . . . . ....

    2024

  59. [67]

    ANALYSIS (2-4 short sentences of plain text, FIRST): name exactly what went wrong and the ONE change that fixes it, or the ONE change you will make to raise the score. [...]

  60. [68]

    reason":

    CODE: a SINGLE fenced Python block –- a complete, self-contained script. Feedback loop. After your code runs, the environment returns your print() output and –- if a valid submission was written –- your leaderboard Position and Raw Score; otherwise the full error traceback. [....

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.