Pith. sign in

REVIEW 3 major objections 4 minor 15 cited by

Reinforcement Learning for Long-Horizon Interactive LLM Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that a 32-billion-parameter interactive agent trained with LOOP, a PPO variant using leave-one-out advantages and per-token importance weights, outperforms o1 on AppWorld after training on just 24 scenarios.

desk verdict LOOP is a credible empirical result—a 32B model trained on 24 AppWorld scenarios with per-token PPO beats o1 and GPT-4o—but the unreported o1 baseline protocol and missing artifacts keep the SOTA claim from being fully verifiable. read the letter →

arxiv 2502.01600 v3 pith:M3PBBCOO submitted 2025-02-03 cs.LG cs.AI

Kevin Chen , Marco Cusumano-Towner , Brody Huval , Aleksei Petrenko , Jackson Hamburger , Vladlen Koltun , Philipp Krähenbühl This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearninginteractivedigitalagentsproximalpolicyoptimizationleave-one-outadvantageAppWorldlong-horizontaskslargelanguagemodelsgradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that reinforcement learning can train interactive digital agents inside the stateful environments they are meant to operate in, rather than relying on instruction-tuned models that have never practiced in those environments. It introduces LOOP, a proximal policy optimization variant that estimates advantages by leaving one rollout out of a group and uses per-token importance weights to allow off-policy updates, all without a value network or extra model copies. Trained on 24 AppWorld scenarios with a simple unit-test reward, a 32-billion-parameter agent reaches 71.3 task goal completion on the test-normal split and 45.7 on test-challenge, surpassing the much larger o1 reasoning model by 9 percentage points. The result matters because prior methods completed fewer than half of AppWorld tasks, and the paper shows that RL with tiny data and a task-completion reward can teach agents to consult API documentation, avoid assumptions, and recover from failed calls.

What carries the argument

The load-bearing object is the LOOP update, which pairs the leave-one-out advantage estimator of Eq. (3) with the per-token PPO objective of Eq. (5). The leave-one-out baseline gives an unbiased, low-variance Monte Carlo advantage without a value network, while per-token importance weights keep off-policy gradient updates stable by clipping each token's probability ratio independently. This combination lets the algorithm reuse K=6 rollouts per task across multiple gradient epochs, making it sample-efficient and memory-efficient, since only one LLM copy is in memory.

What would settle it

Run the same LOOP training on a version of AppWorld in which API responses are nondeterministic (e.g., random delays, shuffled query results, or transient errors) while rewards and tasks stay identical; if task goal completion drops sharply or training diverges compared with the deterministic environment, the i.i.d.-rollout assumption is the cause. A cheaper check is to compare LOOP against an oracle on-policy advantage estimator in the standard environment and look for a systematic gap that grows as the policy drifts off-policy.

Watch

Extended reading notes

Core claim

LOOP combines PPO with a leave-one-out advantage estimate: for each task, K rollouts are sampled i.i.d., and each rollout's advantage is its return minus the average return of the other K-1 rollouts. The PPO objective is applied with per-token importance weights, so the policy can be updated multiple times over reused rollouts while staying inside a trust region, and no critic network is needed. The paper formalizes the agent-environment interaction as a POMDP and assumes the environment response is a deterministic function of initial state and agent tokens, which makes the trajectory distribution factor into policy token probabilities. Under this model, LOOP trains Qwen2.5-32B-Instruct with LoRA on 24 scenarios and 6 rollouts per task, maximizing the fraction of unit tests passed, and achieves state-of-the-art results on both AppWorld splits. The paper also reports that per-token importance weighting outperforms per-turn and per-trajectory variants and that normalizing rewards by group standard deviation hurts performance.

Load-bearing premise

The leave-one-out baseline is unbiased only if the K rollouts for a task are independent samples from the same policy, which requires the environment's responses to be deterministic and unaffected by which policy produced the tokens.

Editorial extensions

If this is right

  • RL with a task-completion reward can be a practical training signal for long-horizon interactive agents, even with only a few dozen training scenarios.
  • LOOP's memory footprint equals fine-tuning a single LLM, so the method scales to agents where a separate critic or reference model would be prohibitive.
  • The trained agent generalizes beyond training scenarios, and its behavior shifts toward reading API docs, avoiding assumptions, and persisting after failed calls.
  • Per-token importance weighting is the key design choice; per-turn and per-trajectory variants are less stable and clip more often.
  • Reward normalization by group standard deviation, as used in GRPO, is not beneficial in this setting and lowers performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the authors leave implicit is that LOOP should transfer to other executable environments with unit-test-style rewards, such as tool-use benchmarks or code repair, whenever rollouts are cheap enough to sample K per task.
  • The method's reliance on i.i.d. rollouts implies that stochastic environments would require an environment-aware baseline or a learned critic, potentially reintroducing the memory cost LOOP avoids.
  • The observed rollout diversity (many distinct solution strategies per task) suggests a testable extension: deliberately encouraging diversity, e.g., via temperature schedules or entropy bonuses, may improve generalization further.
  • One could quantify how much of the gain comes from the leave-one-out baseline versus the per-token clipping by ablating each component independently on the same training runs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents LOOP, a proximal-policy-optimization variant for fine-tuning interactive digital agents with LLMs in a stateful API environment. The method formalizes agent-environment interaction as a POMDP, replaces the learned critic with a leave-one-out Monte Carlo baseline, and uses per-token importance weights and clipping to allow off-policy reuse. Training a 32B Qwen2.5-Instruct model with LOOP on 24 AppWorld scenarios with a unit-test-fraction reward is reported to achieve 71.3 TGC on test-normal and 45.7 TGC on test-challenge, outperforming OpenAI o1 by 9 percentage points and all examined fine-tuning and RL baselines. The paper also provides behavioral analyses and full rollouts in the appendix.

Significance. If the results hold, this is a meaningful advance: it is the first reported RL training of an IDA in a stateful, multi-domain, multi-app environment via direct API calls, and it shows that a simple, memory-efficient policy-gradient method can substantially improve long-horizon interactive behavior from only 24 training scenarios. The paper's strengths include controlled comparisons with shared hyperparameters across RL methods, multiple training seeds and evaluation runs with reported standard deviations, full rollout transcripts, and an external unit-test reward without fitted constants. The main load-bearing weakness is the under-specified OpenAI o1 baseline protocol, which directly supports the headline SOTA claim, plus a few reproducibility gaps that should be addressed.

major comments (3)
  1. [Section 5.3, Table 1] The evaluation protocol for the OpenAI o1 baseline is not specified: the paper lists o1 only as an NFT baseline, with no API model version, sampling temperature, number of evaluation runs, interaction budget, or example transcript, and it does not explicitly confirm that o1 received the same ReAct prompt with the same in-context example, the same API-doc access, the same 50-interaction cap, and the same code-cell parsing as LOOP. Since the abstract's central claim is a 9 pp / 15% improvement over o1, this comparison is load-bearing and currently not independently verifiable. Please provide the complete o1 evaluation protocol and, ideally, release the evaluation harness so that the comparison can be reproduced.
  2. [Algorithm 1, Appendix D] The algorithm specification in Algorithm 1 (line 4) assumes exactly K i.i.d. rollouts per task, but Appendix D states that rollout collection is early-stopped once at least 4 rollouts per task and 90% of the total rollouts have been collected. The paper does not state whether the leave-one-out baseline in Eq. 3 is computed with the actual collected group size or with the nominal K=6; if the nominal K is used after early stopping, the baseline is no longer the mean of the other rollouts and the stated unbiasedness of Eq. 3 no longer holds. Please clarify how the effective group size is used in the advantage computation and, if necessary, correct the estimator or the algorithm description.
  3. [Eqs. 6-8, Section 6] The derivation of the importance-weight cancellation in Eq. 8 assumes that the environment is deterministic and policy-independent, formalized by the indicator I(s0,x) in Eq. 6. This matches AppWorld and is acknowledged in Section 6 as a missing feature, but the paper presents the POMDP formulation and LOOP derivation in a general way. The text should state explicitly that in stochastic or policy-dependent environments the leave-one-out baseline in Eq. 3 is biased with respect to the sampling policy and that Eq. 8 no longer holds; currently this limitation is only implicit in a footnote and in the discussion of AppWorld's scope.
minor comments (4)
  1. [Section 5.2, Appendix D.1] Section 5.2 says '24 out of 30 scenarios' while Section 5.1 reports a train split of 35 scenarios; please clarify whether 30 refers to the number of available difficulty-1/2 scenarios after filtering or whether one of these numbers is a typo.
  2. [Table 1 vs. Table 2, Section 5.4] The headline 71.3 TGC and 9 pp o1 comparison come from the best checkpoint of the best training run, while the multi-run average for LOOP (token) is 66.4 TGC on Test-N. The tables do disclose this selection protocol, but the abstract and Section 5.4 should state it more prominently so that the best-run result is not read as expected performance.
  3. [Appendix G] There are minor typographical issues in the appendix, including 'at at gradient step' in Appendix G.3 and the missing space in 'K = 6rollouts' in Section 5.2; these should be corrected in a final pass.
  4. [Reproducibility] No code or checkpoints are released. Given the closed-source nature of the o1 baseline and the complexity of the AppWorld harness, releasing the evaluation and training code, or at least the exact o1 evaluation script, would substantially strengthen the reproducibility of the central claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LOOP's empirical results are benchmarked externally on AppWorld; the only self-citations are implementation components and do not constrain the claims.

full rationale

The paper's central claim is an empirical result: LOOP, a PPO variant with leave-one-out advantages and per-token importance weights, trained with a task-completion reward on 24 AppWorld scenarios, achieves 71.3 TGC on test-normal and 45.7 TGC on test-challenge, outperforming OpenAI o1. This result is measured against the external AppWorld benchmark, with the reward defined by the benchmark's unit tests, and no algorithm parameter is fitted to the test splits. The leave-one-out advantage in Eq. 3 and the per-token importance weighting in Eq. 5 are standard policy-gradient estimators with established external literature provenance, and the derivation from Eq. 6 to Eq. 8 is a genuine cancellation of the environment indicator I(s0,x) rather than an encoding of the result. The only self-citations are implementation-level components: Cut Cross-Entropy from Wijmans et al. (2024) for memory efficiency and rollout early-stopping similar to Wijmans et al. (2020); neither is load-bearing for the scientific claim. Checkpoint selection on the dev set is ordinary model selection and does not constitute a fitted input being renamed as a prediction. The under-specified OpenAI o1 baseline protocol is a legitimate evaluation-quality concern, but it is not circularity: no equation, fitted constant, or self-citation chain reduces the o1 comparison to the paper's own outputs. The derivation chain is therefore self-contained, and no circular step is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim is empirical, so no scientific constants are fitted to data. The listed free parameters are training hyperparameters, mostly shared across compared RL methods. The axioms are standard RL background plus domain assumptions about AppWorld determinism and reward validity. The per-token PPO variant is introduced by the paper and justified empirically rather than formally. No new entities are posited.

free parameters (5)
  • Rollouts per task K = 6
    Chosen for all leave-one-out methods; controls baseline variance and compute; same for LOOP, RLOO, and GRPO.
  • Learning rate = 5e-5
    Constant for all fine-tuning; reported in Appendix D.
  • LoRA rank and alpha = r=16, alpha=32
    Memory-efficient low-rank adaptation; kept fixed across methods.
  • Advantage pruning threshold = 0.01
    Rollouts with near-zero advantage are discarded before gradient updates (Appendix D); a design choice.
  • Interaction budget = 40 train, 50 eval
    Episode length cap; affects success rates and comparison.
assumptions (6)
  • standard math Policy gradient theorem and PPO trust-region update (Eqs. 2, 4, 5).
    LOOP's gradient and clipping derive from standard PPO; no proof is repeated, but this is accepted background.
  • standard math Leave-one-out baseline is unbiased for i.i.d. rollouts from the sampling policy (Eq. 3).
    The advantage estimate uses K-1 other rollouts per task; unbiasedness requires samples to be i.i.d. draws from p_psi.
  • domain assumption AppWorld's REPL environment is deterministic and policy-independent (footnote 1, Eq. 6).
    The trajectory distribution factors as policy token probabilities times an indicator I(s0,x); if the environment were stochastic, the leave-one-out baseline would be biased.
  • domain assumption Reward as fraction of unit tests passed is a valid task-completion signal (Section 5.2).
    The reward is the external benchmark's unit-test pass rate; it assumes unit tests cover all required and forbidden state changes.
  • ad hoc to paper Per-token importance weighting with a trajectory-level advantage is a stable PPO variant (Eq. 10).
    The paper asserts per-token weights are generally more stable than per-turn or per-trajectory weights and demonstrates this empirically, but provides no formal stability guarantee.
  • domain assumption Training on difficulty 1/2 scenarios generalizes to held-out difficulty 3 and new-app scenarios (Section 5.4).
    The central SOTA claim depends on this generalization; the paper provides evidence but no mechanism beyond rollout diversity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Long-Horizon Interactive LLM Agents." pith.science (2026). https://pith.science/paper/M3PBBCOO

@misc{pith2026250201600,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Long-Horizon Interactive LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M3PBBCOO}},
  note         = {Machine review of arXiv:2502.01600}
}
read the original abstract

Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.

Figures

Figures reproduced from arXiv: 2502.01600 by the authors.

Figure 1
Figure 1. An interactive digital agent receives a user request and iteratively interacts with APIs through a Python read-eval-print loop (REPL) to accomplish the task. of information-gathering and state-changing actions, and potential replanning in response to information that is ob￾tained along the way. The best open-weights models (Yang et al., 2024) have a success rate below 40% in these kinds of tasks, while top-of-the li… view at source ↗
Figure 2
Figure 2. Top: modeling interactive agents as a POMDP. States take the form [s0, c, x1:t], where s0 is the hidden state (containing the initial state of the coding environment), c is the initial context for the task (green), and x1:t combines tokens emitted by the LLM (blue) with tokens emitted by the environment (red). Bottom: PPO with a per-token, -turn, or -trajectory importance weights. Policy Optimization with a Leave-On… view at source ↗
Figure 3
Figure 3. (a) Aggregate changes in agent behavior between the base model and LOOP, averaged over three i.i.d. rollouts per dev (validation) task. (b) Rollouts of the base and LOOP agents on a single dev task. The agent must identify the user’s roommates (via phone.search contacts) and pending requests to them (via venmo.show sent payment requests), and then call venmo.remind payment request on each. The base agent assumes the… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A visualization of 100 i.i.d. rollouts of an agent on the same task after LOOP training. Each column represents a rollout, containing 12 to 33 color-coded API calls. The majority of rollouts (98 out of 100) successfully complete the task, yet 94 out of 98 exhibit uniqu…
Figure 5
Figure 5. Figure 5: Changes in behaviors, aggregated over all 72 tasks used in reinforcement learning training (3 i.i.d. rollouts per task). Showing that the emergent behaviors observed on dev tasks ( [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Summaries of three i.i.d. rollouts of task 2a163ab 1 from near the beginning of training (step 38, left column) and three i.i.d. rollouts of the same task near the best checkpoint for this model (step 530, right column). First episode from early and late conditions are…
Figure 7
Figure 7. Figure 7: Training curves for the best-performing training run with LOOP. In this experiment, training took place over 42 hours on two NVIDIA H100 8-GPU nodes. Strictly Normalized Test Normal (Test-N) Test Challenge (Test-C) Type Algorithm Action on-policy reward TGC SGC TGC SGC…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

    cs.AI 2026-08 conditional novelty 7.0 of 10

    DiDPO splits code diffs into similar sub-diff anchors across rollouts and uses those groups to compute fine-grained token-level advantages, outperforming GRPO and GiGPO on coding benchmarks.

  2. Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Gated-BEPO derives step-level Bellman advantages from empirical rollout graphs and uses a confidence gate to mix them with episode-level credit, improving LLM agent success on WebShop, ALFWorld, and Sokoban.

  3. WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A benchmark and Docker sandbox that tests whether owner-scoped AI agents can collaborate on real tasks without being manipulated into privacy leaks, poisoned evidence, or invalid approvals.

  4. Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

    cs.AI 2026-07 conditional novelty 7.0 of 10

    A self-speculating agent trained by alternating agent and speculator RL raises its own next-tool-call Hit@1 from 44 to 61 (Qwen3-4B) and from 49 to 66 (Qwen3.5-4B) without task-success loss.

  5. Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A bidirectional self-distillation signal, built from an augmented Meta-Skill view and a pruned skill view, rescales GRPO advantages and improves skill utilization for LLM agents on ALFWorld and WebShop.

  6. CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

    cs.LG 2026-08 conditional novelty 6.0 of 10

    AI agents that use tools remember each past step as cached tokens; CommitKV deletes a chunk only when its influence drops from high before a tool call to low after the observation returns.

  7. ADIAS: Automated Design of Interactive Agentic Systems

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Persistently tracking repair issues instead of only candidate performance improves automated full-code AI agent design by 25.2% on average over five benchmarks.

  8. TCPO: Turn-Level Credit Policy Optimization

    cs.AI 2026-08 conditional novelty 6.0 of 10

    TCPO improves multi-turn LLM RL by converting verifier scores into turn-level credit via three reference-based comparisons, beating baseline methods on math, code, and agent tasks.

  9. STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Normalized entropy locates trajectory-neglect steps; STAPO selectively optimizes them with a trajectory-aware reward and trajectory-independent penalty, beating group-based RL baselines on agent tasks.

  10. Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

    cs.AI 2026-07 conditional novelty 6.0 of 10

    HCRA turns collaborative recommendation into a stochastic game whose termination condition is driven by human-calibrated acceptance probability and linguistic self-reflection, yielding higher success rates than uncali...

  11. SOD: Step-wise On-policy Distillation for Small Language Model Agents

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    SOD reweights on-policy distillation strength step-by-step using divergence to stabilize tool use in small language model agents, yielding up to 20.86% gains and 26.13% on AIME 2025 for a 0.6B model.

  12. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  13. Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning

    cs.LG 2026-08 conditional novelty 5.0 of 10

    EFCA combines immediate environment feedback and recent state-history signals into a return reweighting that improves step-level credit assignment for LLM agents in ALFWorld and WebShop.

  14. Agentic Reinforcement Learning with Self-Distilled Reward Shaping

    cs.LG 2026-08 conditional novelty 5.0 of 10

    ADRS converts a frozen model's privileged skill-conditioned token scores into calibrated, return-gated reward shaping for agentic reinforcement learning, reporting consistent long-horizon gains.

  15. Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

    cs.AI 2026-02 conditional novelty 5.0 of 10

    BAO, a behavior-enhanced SFT plus regularized RL pipeline, improves proactive agents' task performance while lowering user-involvement rate, beating UserRL baselines on three UserRL gym tasks.

Reference graph

Works this paper leans on

50 extracted references · 30 canonical work pages · cited by 15 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLMs

    Ahmadian, A., Cremer, C., Gall \' e , M., Fadaee, M., Kreutzer, J., Pietquin, O., \" U st \" u n, A., and Hooker, S. Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLMs . In ACL, 2024

    work page 2024

  3. [3]

    Thinking fast and slow with deep learning and tree search

    Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. In NeurIPS, 2017

    work page 2017

  4. [4]

    DigiRL : Training in-the-wild device-control agents with autonomous reinforcement learning

    Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. DigiRL : Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv:2406.11896, 2024

    arXiv 2024

  5. [5]

    Grounding large language models in interactive environments with online reinforcement learning

    Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In ICML, 2023

    work page 2023

  6. [6]

    FireAct : Toward language agent fine-tuning

    Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. FireAct : Toward language agent fine-tuning. arXiv:2310.05915, 2023

    arXiv 2023

  7. [7]

    DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI . DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948, 2025

    arXiv 2025

  8. [8]

    The Llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv:2407.21783, 2024

    arXiv 2024

Show all 50 references
  1. [9]

    D., Oosterhuis, H., de Rijke, M., and Shukla, S

    Gupta, S., Ahuja, C., Lin, T.-Y., Roy, S. D., Oosterhuis, H., de Rijke, M., and Shukla, S. N. A simple and effective reinforcement learning method for text-to-image diffusion fine-tuning. arXiv:2503.00897, 2025

    2025

  2. [10]

    C., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R

    Havrilla, A., Du, Y., Raparthy, S. C., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R. Teaching large language models to reason with reinforcement learning. arXiv:2403.04642, 2024

    2024 arXiv

  3. [11]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In ICLR, 2022

    2022

  4. [12]

    P., Littman, M

    Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101 0 (1-2), 1998

    1998

  5. [13]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In ICML, 2002

    2002

  6. [14]

    Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. VinePPO : Unlocking RL potential for LLM reasoning through refined credit assignment. arXiv:2410.01679, 2024

    2024 arXiv

  7. [15]

    Language models can solve computer tasks

    Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. In NeurIPS, 2024

    2024

  8. [16]

    Buy 4 reinforce samples, get a baseline for free! In ICLR 2019 Workshops, 2019

    Kool, W., van Hoof, H., and Welling, M. Buy 4 reinforce samples, get a baseline for free! In ICLR 2019 Workshops, 2019

    2019

  9. [17]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention . In Symposium on Operating Systems Principles, 2023

    2023

  10. [18]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv:2411.15124, 2024

    2024 arXiv

  11. [19]

    AgentInstruct : Toward generative teaching with agentic flows

    Mitra, A., Del Corro, L., Zheng, G., Mahajan, S., Rouhana, D., Codas, A., Lu, Y., Chen, W.-g., Vrousgos, O., Rosset, C., et al. AgentInstruct : Toward generative teaching with agentic flows. arXiv:2407.03502, 2024

    2024 arXiv

  12. [20]

    WebGPT : Browser-assisted question-answering with human feedback

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. WebGPT : Browser-assisted question-answering with human feedback. arXiv:2112.09332, 2021

    2021 arXiv

  13. [21]

    D., and Barzilay, R

    Narasimhan, K., Kulkarni, T. D., and Barzilay, R. Language understanding for text-based games using deep reinforcement learning. In EMNLP, 2015

    2015

  14. [22]

    Introducing OpenAI o1, 2024

    OpenAI. Introducing OpenAI o1, 2024. URL https://openai.com/o1/

    2024

  15. [23]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurIPS, 2022

    2022

  16. [24]

    Agent Q : Advanced reasoning and learning for autonomous AI agents

    Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., and Rafailov, R. Agent Q : Advanced reasoning and learning for autonomous AI agents. arXiv:2408.07199, 2024

    2024 arXiv

  17. [25]

    ToolLLM : Facilitating large language models to master 16000+ real-world APIs

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. ToolLLM : Facilitating large language models to master 16000+ real-world APIs . In ICLR, 2024

    2024

  18. [26]

    Toolformer: Language models can teach themselves to use tools

    Schick, T., Dwivedi-Yu, J., Dess \` , R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. NeurIPS, 2023

    2023

  19. [27]

    I., and Abbeel, P

    Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016

    2016

  20. [28]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv:1707.06347, 2017

    2017 arXiv

  21. [29]

    DeepSeekMath : Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024

    2024 arXiv

  22. [30]

    Direct multi-turn preference optimization for language agents

    Shi, W., Yuan, M., Wu, J., Wang, Q., and Feng, F. Direct multi-turn preference optimization for language agents. In EMNLP, 2024

    2024

  23. [31]

    Reflexion: Language agents with verbal reinforcement learning

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. NeurIPS, 2024

    2024

  24. [32]

    D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P

    Singh, A., Co - Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem-solving with language models. TMLR, 2024

    2024

  25. [33]

    M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback. In NeurIPS, 2020

    2020

  26. [34]

    torchtune: PyTorch's finetuning library, April 2024

    torchtune maintainers and contributors. torchtune: PyTorch's finetuning library, April 2024. URL https//github.com/pytorch/torchtune

    2024

  27. [35]

    A pp W orld: A controllable world of apps and people for benchmarking interactive coding agents

    Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian, N. A pp W orld: A controllable world of apps and people for benchmarking interactive coding agents. In ACL, 2024

    2024

  28. [36]

    Executable code actions elicit better LLM agents

    Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better LLM agents. In ICML, 2024

    2024

  29. [37]

    DD-PPO : L earning near-perfect PointGoal navigators from 2.5 billion frames

    Wijmans, E., Kadian, A., Morcos, A., Lee, S., Essa, I., Parikh, D., Savva, M., and Batra, D. DD-PPO : L earning near-perfect PointGoal navigators from 2.5 billion frames. In ICLR, 2020

    2020

  30. [38]

    a henb \

    Wijmans, E., Huval, B., Hertzberg, A., Koltun, V., and Kr \" a henb \" u hl, P. Cut your losses in large-vocabulary language models. arXiv:2411.09009, 2024

    2024 arXiv

  31. [39]

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 1992

    1992

  32. [40]

    Qwen2.5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv:2412.15115, 2024

    2024 arXiv

  33. [41]

    Intercode: standardizing and benchmarking interactive coding with execution feedback

    Yang, J., Prabhakar, A., Narasimhan, K., and Yao, S. Intercode: standardizing and benchmarking interactive coding with execution feedback. In NeurIPS, 2023

    2023

  34. [42]

    Keep CALM and explore: Language models for action generation in text-based games

    Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K. Keep CALM and explore: Language models for action generation in text-based games. In EMNLP, 2020

    2020

  35. [43]

    WebShop : Towards scalable real-world web interaction with grounded language agents

    Yao, S., Chen, H., Yang, J., and Narasimhan, K. WebShop : Towards scalable real-world web interaction with grounded language agents. In NeurIPS, 2022

    2022

  36. [44]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. ReAct : Synergizing reasoning and acting in language models. In ICLR, 2023

    2023

  37. [45]

    Scaling relationship on learning mathematical reasoning with large language models

    Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models. arXiv:2308.01825, 2023

    2023 arXiv

  38. [46]

    ST ar: Bootstrapping reasoning with reasoning

    Zelikman, E., Wu, Y., Mu, J., and Goodman, N. ST ar: Bootstrapping reasoning with reasoning. In NeurIPS, 2022

    2022

  39. [47]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning

    Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In NeurIPS, 2024

    2024

  40. [48]

    PyTorch FSDP : Experiences on scaling fully sharded data parallel

    Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. PyTorch FSDP : Experiences on scaling fully sharded data parallel...

    2023

  41. [49]

    and Zanette, A

    Zhou, Y. and Zanette, A. ArCHer : training language model agents via hierarchical multi-turn RL . In ICML, 2024

    2024

  42. [50]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.