Pith. sign in

REVIEW 4 major objections 4 minor 7 cited by

A small model trained with reinforcement learning beats much larger prompted models at ML engineering tasks by 22% on average.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

RL-trained Qwen2.5-3B outperforms prompted Claude-3.5-Sonnet and GPT-4o on 12 MLEBench tasks by an average of 22% and 24%, using two targeted RL modifications.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Worth a serious referee despite a real selection-bias problem: the RL machinery is novel and useful, but the 22% headline only holds on the 12-task feasible subset. the 4 major comments →

arxiv 2509.01684 v1 pith:DPRS2Y47 submitted 2025-09-01 cs.LG cs.AI

Reinforcement Learning for Machine Learning Engineering Agents

Sherry Yang , Joy He-Yueya , Percy Liang This is my paper
classification cs.LG cs.AI
keywords reinforcement learninglanguage model agentsML engineeringMLEBenchduration-aware gradient updatesenvironment instrumentationpartial credit rewardsself-improvement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing agents that do machine-learning engineering rely on prompting the largest available language model, and a prompted agent never improves with the experience it collects. This paper claims that a much smaller model which trains on its own experience through reinforcement learning can beat those static large models: a 3-billion-parameter model tuned per task outperformed prompted frontier models on 8 of 12 Kaggle-style MLEBench tasks, by an average of 22% over prompting Claude-3.5-Sonnet with an agent scaffold. Two obstacles had to be removed for RL to work here. First, agent actions take variable amounts of wall-clock time, so standard distributed RL silently favors fast but mediocre solutions; the paper reweights policy-gradient updates by action duration. Second, test accuracy alone is an all-or-nothing reward, so the paper has a frozen copy of the model instrument the generated code with progress-reporting print statements, giving partial credit for steps like loading data or training a model. If the finding holds, compute spent on gradient updates can substitute for a much larger static model in this setting.

Core claim

The central claim is that a policy which learns from its own executions can outperform a policy that merely samples from a much larger model, even when the learner starts from a far weaker base. Concretely, Qwen2.5-3B trained with RL on each of 12 MLEBench tasks reaches an average 22% improvement over prompting Claude-3.5-Sonnet with the AIDE agent scaffold and 24% over prompt-based GPT-4o, winning 8 of 12 tasks, and beats GPT-4o run under three different scaffolds on 9 of 12 tasks. The claim rests on two mechanisms. Duration-aware gradient updates multiply each policy-gradient term by the executing action's duration, canceling the sampling bias that otherwise starves slow, expensive solutio

What carries the argument

The argument runs on two modifications inside the RL training loop, plus a third, lighter ingredient. (1) Duration-aware gradient updates: in a fixed wall-clock period, an action of duration Δt gets sampled about T/Δt times, so its total gradient contribution carries a hidden 1/Δt factor that favors fast actions; weighting each update by Δt cancels this, yielding the update rule ∇θJ = E[Δt·∇θ log πθ(a|s)·Â(s,a)] (with Δt rescaled by the batch average), so slow high-reward actions keep their share of the gradient. (2) Environment instrumentation: a frozen second copy of the same small model inserts progress print statements ('loaded data', 'trained model', ...) into the agent's code; regex-ma

Load-bearing premise

The 12 tasks were chosen because the base 3B model could already produce a valid first solution in a batch of 128 samples; if that feasibility filter selects tasks that are easier or more RL-friendly, the average 22% advantage over prompted large models may not extend to the full 75-task benchmark.

What would settle it

Run the identical RL pipeline on all 75 MLEBench tasks, in particular the ones where the base model produces no valid initial solution, and recompute the average advantage over Claude-3.5-Sonnet with AIDE on the subset where the baseline itself produced valid submissions (several baseline cells are N/A in the paper's Table 1). If the 22% shrinks or reverses on the previously infeasible tasks, the claim rests on the feasibility filter rather than on RL itself; a further check would train a frontier-scale model with the same RL loop and budget, which isolates whether the driver is the gradient u

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • An ML-engineering agent does not need a frontier-scale model: a 3B open-weight model that is allowed to train on its own code-execution experience beats static frontier models on most of the tasks tested.
  • Gradient updates use compute better than extended prompting in this regime: the paper notes that running a prompted scaffold for days, or 100 hours in the case of GPT-4o, barely moves the best solution, while RL training improves it steadily.
  • Distributed RL systems with variable-duration actions need the duration weighting: without it, the policy collapses toward fast-but-suboptimal actions such as linear logistic regression instead of gradient boosting.
  • Dense, execution-based partial credit makes RL training converge faster and less likely to get stuck with no valid solution, compared with sparse test-score rewards.
  • The same RL loop improves both generating solutions from scratch and improving a previous solution; asking the agent to revise its own work yields about an 8% average gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison is run on a feasibility-filtered subset: the paper selected the 12 tasks where the base 3B model already writes a valid first solution in one batch of 128 samples. A natural next experiment is the same recipe on tasks the base model initially fails, to see whether RL plus instrumentation opens those up or whether the advantage is confined to tasks the model can already appr
  • The duration-weighting principle is not specific to Kaggle-style tasks: any agentic RL setting with variable wall-clock execution (browser automation, robotics, scientific computing) should exhibit the same sampling bias toward fast actions and could take the same correction.
  • The comparison is against static prompted models, not against a large model that is itself RL-trained; if gradient updates are the real driver of the gains, an RL-trained frontier model is the stiffer test the paper leaves for future work.
  • Print-statement instrumentation is a cheap, execution-verifiable reward mechanism that could generalize to other executable formats (shell scripts, SQL, notebooks) and could complement learned reward models rather than replace them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an RL training framework for small language models used as ML engineering agents on MLEBench. It introduces two mechanisms: duration-aware gradient updates to correct the sampling bias in asynchronous distributed RL when actions take variable amounts of time, and environment instrumentation, in which a frozen LM inserts print statements into the agent's code to provide partial credit for intermediate execution steps. The authors train Qwen2.5-3B on 12 selected MLEBench tasks and report that it outperforms prompting Claude-3.5-Sonnet with the AIDE scaffold by 22% on average and outperforms GPT-4o with several scaffolds on 9 of 12 tasks.

Significance. If the results hold, this is a useful empirical contribution. The paper demonstrates a complete pipeline for RL training of a small open-weight model on agentic ML engineering tasks, and the two proposed ideas—duration weighting and print-based partial credit—are simple, plausible, and potentially transferable. The paper includes ablations for each mechanism, detailed hyperparameters, and a comparison against multiple agent scaffolds. However, the headline claim is not currently established because of the task-selection criterion and because the RL reward is the final test-split score. The central contribution depends on whether these issues can be resolved; with the current evidence, the 22% number is an upper envelope over a feasibility-filtered subset and may partly reflect direct optimization on the test split.

major comments (4)
  1. [Section 4.1, Table 1] The 12-task evaluation set is not a random sample from MLEBench. Section 4.1 defines it as the set of tasks where Qwen2.5-3B 'could generate a valid initial solution in a batch of 128 samples with temperature 0.7.' This filters out tasks where the small model cannot bootstrap, which is likely correlated with task difficulty and with the effectiveness of RL. The reported 22% average improvement is therefore an upper envelope over a feasible subset, not an estimate for MLEBench as a whole. Additionally, Table 1 contains N/A entries for several baselines, and the manuscript does not state how the average is computed over an uneven task set. Provide per-task comparisons, report results on the full MLEBench suite or at least on a pre-specified random subset, and state explicitly how N/A baselines are handled in the average.
  2. [Sections 2 and 3.2] The RL reward is the performance on the test split: Section 2 defines R as 'performance on the test split,' and Section 3.2 states that if a program runs without error, 'the submission (label for the test split) is graded by the grader of the environment, and true task performance is used as reward.' Thus the policy is trained with scalar feedback derived from the test labels, while the baseline prompted agents receive no such gradient updates. This conflates test-time optimization with generalization and may account for a substantial part of the reported improvement. I ask the authors to use a validation split (or a leaderboard with limited access) for the RL reward and evaluate on the test split only once at the end, or to clearly argue why direct test-split reward is an acceptable comparison in this benchmark.
  3. [Section 3.2, Figure 5] Environment instrumentation can be gamed by the agent itself. The partial-credit reward is obtained by regex-matching terminal output for strings such as 'loaded data.' The agent generates the program, and the static instrumentation LM only inserts additional print statements; it does not remove or sanitize print statements already present in the agent's code. An agent can therefore write `print("loaded data")` before actually loading data and collect +0.1 per marker without performing the corresponding step. The paper's argument that using a separate static LM prevents this is insufficient. Demonstrate robustness, for example by stripping all pre-existing print statements from the agent's code before instrumentation, by verifying that the marked operation actually occurs between markers, or by showing empirically that the learned policy does not exploit spurious prints.
  4. [Section 3.1, Eq. (3)] The duration-aware gradient correction is derived from a deterministic frequency model n_x ≈ π(x|s)·T/Δt_x. This assumes that, in asynchronous distributed RL, action samples arrive at a rate strictly proportional to the policy probability divided by execution time. In practice, actor scheduling, timeouts, stale policies, and limited queue sizes can break this proportionality. The paper should derive Eq. (3) from the actual asynchronous sampling distribution, or at least measure the discrepancy between the assumed and observed sampling frequencies. I also ask for a sensitivity analysis of the final scores to the magnitude of the duration weighting, especially since the manuscript rescales Δt by the batch average.
minor comments (4)
  1. [Table 1] Column header 'Claude3.5-Sonn' is truncated. Also, the column labeled 'GPT-4o-100hrs' is referred to as 'GPT-4o 100 hours' in the text; unify the naming. The row 'leaf-classification' has a very large standard error for Llama3.1-405B (6.747 ± 5.398); consider verifying or annotating this outlier.
  2. [Section 4.2] The statement 'average improvement of 17.7% over the best scaffold for each tasks' is ambiguous. Define how the average is computed across tasks when scores have different scales, directions, and N/A values.
  3. [Appendix D.1] The appendix says 'across 5 seeds per task,' but the main text and Table 1 report mean and standard error across 3 runs. Clarify whether the number of RL runs is 3 or 5, and use consistent language.
  4. [Algorithm 1, Appendix A.1] The algorithm samples m·B prompts but waits for B executions to complete. Clarify the relationship between the sampling multiplier m, the rollout batch, and the PPO mini-batch size reported in Appendix B.1.

Circularity Check

0 steps flagged

No significant circularity: the central RL-vs-prompting comparison is empirical, uses an external MLEBench grader, and the 12-task selection is a sampling concern rather than a definitional reduction.

full rationale

The paper's central empirical claim — that RL-trained Qwen2.5-3B outperforms prompted Claude-3.5-Sonnet on 12 MLEBench tasks — is not derived from the target result. The RL reward uses the independent MLEBench grader for test performance plus a hand-specified partial-credit scheme (regex-matched print statements awarded 0.1 each); no parameter is fitted to the headline 22% improvement. The duration-aware gradient update (Eq. 3) is a proposed correction for sampling bias from variable-duration actions, derived from the stated frequency model, not an input assumed to produce the result. Environment instrumentation uses a separate static copy of the Qwen model, and the paper explicitly notes the instrumentation must be a separate copy to prevent reward hacking. Baselines are taken from MLEBench (an external benchmark) and MLAgentBench; the only self-citation (MLAgentBench, involving author Percy Liang) is used as a baseline scaffold, not as the justification for the RL improvement. The 12-task subset is selected by whether the base model can produce a valid initial solution, which is a selection-bias/correctness concern about generalization, not a logical circularity: the selected tasks are not defined in terms of the measured outcome, and the grader is external. No self-definitional, fitted-input-as-prediction, or self-citation-chain circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No new physical entities are introduced. The central claim depends on the hand-set partial-credit weights, the self-improvement prompt frequency, the choice of benchmark subset, and the three stated assumptions about the sampling process, reward validity, and instrumenter reliability.

free parameters (4)
  • partial_credit_invalid_penalty = -10
    Hand-chosen reward assigned to invalid solutions in environment instrumentation (Section 3.2).
  • partial_credit_per_print = +0.1
    Hand-chosen per-print reward for each regex match in terminal output (Section 3.2).
  • self_improve_prompt_probability = 0.5
    Probability of prompting the agent to improve a previous solution during RL training (Section 3.3).
  • task_selection_criterion = 12 of 75 tasks
    Only tasks where Qwen2.5-3B produces a valid initial solution in 128 samples are included; this selection is a free choice that determines the reported average improvement (Section 4.1).
axioms (3)
  • domain assumption In asynchronous RL, samples of an action arrive at a rate proportional to the policy probability divided by the action's execution time.
    Used to show that fast actions are overrepresented and to derive the duration-aware weighting correction (Section 3.1, Eq. 3).
  • domain assumption Test-split performance from the MLEBench grader is a valid and sufficient reward signal.
    The RL objective and all comparisons use the MLEBench grader score (Section 4.1).
  • domain assumption A static copy of Qwen2.5-3B can accurately insert print statements that reflect real progress stages such as data loading, model definition, training, and prediction.
    Partial credit is parsed from these prints via regex; incorrect or easily gameable insertion would corrupt the reward signal (Section 3.2).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Machine Learning Engineering Agents." pith.science (2026). https://pith.science/paper/DPRS2Y47

@misc{pith2026250901684,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Machine Learning Engineering Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DPRS2Y47}},
  note         = {Machine review of arXiv:2509.01684}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Existing agents for solving tasks such as ML engineering rely on prompting powerful language models. As a result, these agents do not improve with more experience. In this paper, we show that agents backed by weaker models that improve via reinforcement learning (RL) can outperform agents backed by much larger, but static models. We identify two major challenges with RL in this setting. First, actions can take a variable amount of time (e.g., executing code for different solutions), which leads to asynchronous policy gradient updates that favor faster but suboptimal solutions. To tackle variable-duration actions, we propose duration-aware gradient updates in a distributed asynchronous RL framework to amplify high-cost but high-reward actions. Second, using only test split performance as a reward provides limited feedback. A program that is nearly correct is treated the same as one that fails entirely. To address this, we propose environment instrumentation to offer partial credit, distinguishing almost-correct programs from those that fail early (e.g., during data loading). Environment instrumentation uses a separate static language model to insert print statement to an existing program to log the agent's experimental progress, from which partial credit can be extracted as reward signals for learning. Our experimental results on MLEBench suggest that performing gradient updates on a much smaller model (Qwen2.5-3B) trained with RL outperforms prompting a much larger model (Claude-3.5-Sonnet) with agent scaffolds, by an average of 22% across 12 Kaggle tasks.

Figures

Figures reproduced from arXiv: 2509.01684 by Joy He-Yueya, Percy Liang, Sherry Yang.

Figure 1
Figure 1. Figure 1: Performing gradient update with RL on Qwen2.5-3B ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Suboptimal convergence due to limited feedback. In a task of extract￾ing sentiment-relevant phrases from tweets (tweet-sentiment-extraction), the agent converged to a suboptimal solution of directly coding the Jaccard similarity and search for the best phrase in the test input, bypassing ML completely. This demonstrates how sparse rewards can lead to an agent exploiting evaluation metrics rather than learn… view at source ↗
Figure 4
Figure 4. Figure 4: Proposed framework overview. Duration-aware gradient updates re-weights the policy gradient updates according to the execution duration of an action. Environment instrumentation inserts print statements using a static LM, the execution output can then be extracted for partial credit. The agent can be further asked to improve a previous solution, the response can further be enforced with RL. 3.1 Duration-Aw… view at source ↗
Figure 5
Figure 5. Figure 5: Environment instrumentation overview. Another copy of the small LM (Qwen2.5-3B) is prompted to insert print statement into the code generated by the agent. After code execution, output from the terminal is then parsed to assign partial credit by regex matching. Environment instrumentation. We propose to introduce partial credit so that generated programs that fail in the beginning (e.g., during data loadin… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative examples of improvements proposed by the agent during RL training. [Top] earlier improvement proposed by the agent using difference between response length as features for preference prediction. [Bottom] later improvements proposed by the agent using additional features such as word count and average word length difference as features [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The best scores achieved by the agent across time comparing prompting a large model to RL training of a small model. A small model running RL starts off with low scores for many tasks, but eventually outperforms prompting a large model. No DAG Duration Aware Gradient Execution time of generated programs (s) Score = 0.62 More expensive Score = 0.67 steps [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Duration-aware gradient enables the agent to explore more expensive but high-return actions by coming up with more expensive solutions such as gradient boosting, which achieves higher score than linear logistic regression. The RL agent still tends to find faster executing solutions over time. whereas without duration-aware gradient, the agent quickly converges to fast but suboptimal solution (e.g., linear … view at source ↗
Figure 9
Figure 9. Figure 9: Environment instrumentation ablation. Plots show the mean task scores (excluding the partial credit from environment instrumentation) across 128 samples for 3 example tasks across RL training steps. Environment instrumentation improves RL training and enables faster convergence. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Average scores achieved during RL training for different tasks. Scores are -10 for invalid solutions and the actual score from the grader in MLEBench [5] if scores are valid. RL consistently improves average scores across tasks and across 5 seeds per task. D.2 Full Results for Ablating Self-Improvement Prompt [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example code before environment instrumentation for solving the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example code after environment instrumentation for solving the [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: An example of high-performance (score 0.66) high cost (115 seconds) solution for [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: An example of high-performance (score 0.73) high cost (281 seconds) solution using gradient [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards

    cs.LG 2026-07 conditional novelty 7.0

    RLVP post-trains one LLM across eight PDE families with hybrid validity-plus-continuous physics rewards, improving solver accuracy and enabling selective compositional transfer to held-out PDEs.

  2. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 unverdicted novelty 7.0

    MLS-Bench is a benchmark with 140 tasks that evaluates AI agents on inventing generalizable and scalable ML methods, finding they lag human performance especially in insight-driven invention rather than tuning.

  3. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 accept novelty 7.0

    Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.

  4. Revisiting DAgger in the Era of LLM-Agents

    cs.LG 2026-05 conditional novelty 6.0

    DAgger-style training with turn-level policy interpolation raises 4B and 8B LLM agents to 27.3% and 29.8% on SWE-bench Verified, beating several larger published systems.

  5. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 unverdicted novelty 6.0

    MLS-Bench shows that current AI agents fall short of reliably inventing generalizable ML methods, with engineering tuning easier than genuine invention.

  6. Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering

    cs.AI 2026-07 conditional novelty 5.0

    Matryoshka Agent’s orchestrator–sub-agent hierarchy plus tree-ranked RL raises MLE-Dojo HumanRank, letting a 4B orchestrator approach o4-mini and giving a 30B coder up to 36.7% relative gain.

  7. PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents

    cs.LG 2026-05 unverdicted novelty 5.0

    PACEvolve++ uses a phase-adaptive reinforcement learning advisor to decouple hypothesis selection from execution in LLM-driven evolutionary search, delivering faster convergence than prior frameworks on load balancing...

Reference graph

Works this paper leans on

62 extracted references · 13 canonical work pages · cited by 5 Pith papers · 3 internal anchors

  1. [1]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  3. [3]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Reinforcement learning: An introduction, volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  9. [9]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  11. [11]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229–256, 1992

    work page 1992

  12. [12]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  13. [13]

    Efficient exploration in reinforcement learning

    Sebastian B Thrun. Efficient exploration in reinforcement learning. Carnegie Mellon University, 1992

    work page 1992

  14. [14]

    On the sample complexity of reinforcement learning

    Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003

    work page 2003

  15. [15]

    Rllib: Abstractions for distributed reinforcement learning

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053–3062. PMLR, 2018

    work page 2018

  16. [16]

    Acme: A research framework for distributed reinforcement learning

    Matthew W Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Mom- chev, Danila Sinopalnikov, Piotr Sta´nczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020

    Pith/arXiv arXiv 2006

  17. [17]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. 10

    Pith/arXiv arXiv 2024

  18. [18]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024

    2024

  19. [19]

    LangChain, October 2022

    Harrison Chase. LangChain, October 2022

    work page 2022

  20. [20]

    Significant Gravitas. AutoGPT

  21. [21]

    Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code

    Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint arXiv:2311.09835, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Mlrc-bench: Can language agents solve machine learning research challenges? arXiv preprint arXiv:2504.09702, 2025

    Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Lo- geswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Mlrc-bench: Can language agents solve machine learning research challenges? arXiv preprint arXiv:2504.09702, 2025

    arXiv 2025

  24. [24]

    Large language models orchestrating structured reasoning achieve kaggle grandmaster level

    Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprint arXiv:2411.03562, 2024

    arXiv 2024

  25. [25]

    Exploring llm agents for cleaning tabular machine learning datasets

    Tommaso Bendinelli, Artur Dox, and Christian Holz. Exploring llm agents for cleaning tabular machine learning datasets. arXiv preprint arXiv:2503.06664, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI

    Tidor-Vlad Pricope. Hardml: A benchmark for evaluating data science and machine learning knowledge and reasoning in ai. arXiv preprint arXiv:2501.15627, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Automl-gpt: automatic machine learning with gpt (2023)

    Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl-gpt: automatic machine learning with gpt (2023). URL https://arxiv. org/abs/2305.02499

    Pith/arXiv arXiv 2023

  28. [28]

    Large language model agent for hyper-parameter optimization

    Siyi Liu, Chen Gao, and Yong Li. Large language model agent for hyper-parameter optimization. arXiv preprint arXiv:2402.01881, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Large language models for constructing and optimizing machine learning workflows: A survey

    Yang Gu, Hengyu You, Jian Cao, Muran Yu, Haoran Fan, and Shiyou Qian. Large language models for constructing and optimizing machine learning workflows: A survey. arXiv preprint arXiv:2411.10478, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Aide: Ai-driven exploration in the space of code

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Ja- cenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    I-mcts: Enhancing agentic automl via introspective monte carlo tree search

    Zujie Liang, Feng Wei, Wujiang Xu, Lin Chen, Yuxi Qian, and Xinhui Wu. I-mcts: Enhancing agentic automl via introspective monte carlo tree search. arXiv preprint arXiv:2502.14693, 2025

    arXiv 2025

  32. [32]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering

    Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406.12334, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Policy gradient meth- ods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. 11

    work page 1999

  35. [35]

    A natural policy gradient

    Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001

    2001

  36. [36]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015

    2015

  37. [37]

    Q-learning

    Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279–292, 1992

    1992

  38. [38]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  39. [39]

    The arcade learning environment: An evaluation platform for general agents

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research, 47:253–279, 2013

    2013

  40. [40]

    Tensorflow agents: Efficient batched reinforcement learning in tensorflow

    Danijar Hafner, James Davidson, and Vincent Vanhoucke. Tensorflow agents: Efficient batched reinforcement learning in tensorflow. arXiv preprint arXiv:1709.02878, 2017

    Pith/arXiv arXiv 2017

  41. [41]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  42. [42]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  43. [43]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023

    2023

  44. [44]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020

    2020

  45. [45]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023

    2023

  46. [46]

    Reinforcement learning for reasoning in small llms: What works and what doesn’t

    Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn’t. arXiv preprint arXiv:2503.16219, 2025

    arXiv 2025

  47. [47]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Agentbench: Evaluating llms as agents, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang añd Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2023

    work page 2023

  50. [50]

    Context-Aware Language Modeling for Goal-Oriented Dialogue Systems

    Charlie Snell, Mengjiao Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204.10198, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  51. [51]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461–12495, 2024. 12

    work page 2024

  52. [52]

    Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning

    Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08426, 2022

    Pith/arXiv arXiv 2022

  53. [53]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried and Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried and Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024

    work page 2024

  54. [54]

    Language Understanding for Text-based Games Using Deep Reinforcement Learning

    Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  55. [55]

    Offline rl for natural language generation with implicit language q learning

    Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871, 2022

    Pith/arXiv arXiv 2022

  56. [56]

    Process reward models for llm agents: Practical framework and directions

    Sanjiban Choudhury. Process reward models for llm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.arXiv preprint arXiv:2410.12832, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Au- tonomous evaluation and refinement of digital agents

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Code as reward: Empowering reinforcement learning with vlms

    David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Code as reward: Empowering reinforcement learning with vlms. arXiv preprint arXiv:2402.04764, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    Llm-as-a-judge & reward model: What they can and cannot do

    Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    ‘ re- quest_id,requester_received_pizza t3_i8iy4,0 t3_1mfqi0,0 etc “‘ • Data snippet: -> /workdir/random-acts-of-pizza/prepared/public/test.json: [

    Shivam Singhal, Cassidy Laidlaw, and Anca Dragan. Reliability-aware preference learning for llm reward models. 13 Appendix In this appendix, we provide additional details on the method (Appendix A), additional details on the experimental setups (Appendix B), prompts to LLMs (Appendix C), and additional experimental results (Appendix D). A Additional Metho...

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.