Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Branching Policy Optimization argues that agent RL should exploit snapshottable sandboxes: forking at high-entropy steps and comparing sibling rollouts yields unbiased, lower-variance policy gradients that outperform GRPO and RLOO by 3.6–6.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An RL algorithm for language agents that snapshots sandboxes at high-entropy steps and shares branches across common prefixes, reducing advantage variance and achieving 3.6–6.1 point gains over GRPO/RLOO-style baselines.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Sibling-baseline variance reduction is a good idea with a clean local theorem, but the global unbiasedness claim is overstated and the empirics need artifacts. the 4 major comments →

arxiv 2607.14171 v1 pith:2BLPPV5Z submitted 2026-07-15 cs.LG cs.CL

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

Bowei He , Yankai Chen , Xiaokun Zhang , Xue Liu This is my paper
classification cs.LG cs.CL
keywords reinforcement learningLLM agentssandboxsnapshot/restorevariance reductionpolicy gradientsibling baselineGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the standard rollout topology for agent RL — N independent trajectories per prompt with a baseline computed only at the initial state — throws away information that executable sandboxes were built to provide. Because sandboxes can be snapshotted, restored, and resumed, Branching Policy Optimization (BPO) instead rolls out one backbone trajectory, picks the steps where the policy is most uncertain, forks several alternative actions there, and compares the returns of the sibling rollouts that share the prefix. The paper proves this sibling-baseline advantage is unbiased and has strictly lower variance than the GRPO/RLOO trajectory-level baseline, with the reduction exactly the prefix-explained portion of return variance, and demonstrates on WebShop, ALFWorld, and SWE-bench that it improves success by 3.6–6.1 absolute points at matched compute while reaching the best baseline's final score in 38% fewer gradient steps. A sympathetic reader would care because high return variance is the main obstacle to stable policy-gradient training on long-horizon agent tasks, and the paper turns an overlooked property of the environment into a reusable variance-reduction primitive.

Core claim

The central discovery: in a Markov decision process with a faithful snapshot/restore operator, the leave-one-out advantage formed from K rollouts that share a prefix τ0:t is unbiased and has variance K/(K−1)·E[Var(R|s_t)] — the GRPO/RLOO variance minus the prefix-explained term Var(Vπ(s_t)|s_0). This follows from the law of total variance. BPO operationalizes the identity with a backbone rollout, entropy-selected branch points, K−1 sibling forks per branch, and λ-discounted propagation to earlier steps. On WebShop, ALFWorld, and SWE-bench Verified, at matched compute, it improves success by 3.6–6.1 absolute points over PPO, RLOO, GRPO, and VinePPO, halves gradient-norm variance, and reaches

What carries the argument

The load-bearing object is the rollout tree built by snapshot/restore: a single backbone trajectory plus K−1 sibling sub-trajectories forked at selected branch points, so all K leaves share the prefix up to the branch. The advantage estimator is the sibling baseline, the leave-one-out difference between a sibling's return-to-go and the average of the other K−1 siblings' return-to-go at the same state; it is unbiased because the other siblings are independent of the chosen action given the state. The variance-reduction identity, Var(A_BPO)=Var(A_GRPO)−K/(K−1)·Var(Vπ(s_t)|s_0), is what carries the argument, and the entropy-driven branch scheduler (top-M decision boundaries by token-level Shann

Load-bearing premise

The load-bearing premise is that restoring a sandbox from a snapshot reproduces exactly the same future every time—if a live page, an external API, or a network response has moved on between restores, the rollout branches forked from the same state no longer share a true conditional baseline, and the variance-reduction guarantee shrinks or reverses.

What would settle it

Compare sibling-return variance after restore to variance across independent trajectories in a live network environment (e.g., a real e-commerce site where page loads change between restores). Under the theorem, siblings conditioned on the same state should be uncorrelated with variance near E[Var(R|s_t)], so Var_BPO/Var_GRPO should equal E[Var(R|s_t)]/Var(R|s_0) and fall well below 1; if the ratio stays near 1 or sibling returns show extra covariance, snapshot fidelity is violated and the unbiasedness/variance-reduction claims do not transfer.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Algorithms in the baseline-only family (GRPO, RLOO) can be improved by replacing independent rollouts with a shared-prefix tree whenever the environment has faithful snapshot/restore; the prefix-explained return variance is removed from the gradient estimator.
  • Deeper branch points yield strictly larger variance reduction, so the benefit concentrates on long-horizon tasks where a single early decision determines the outcome.
  • At matched compute, BPO reaches the best baseline's final success in 0.62× the gradient steps and maintains roughly half the gradient-norm variance across training.
  • Branching at the policy's highest-entropy decision boundaries is worth more than uniform, random, or lowest-entropy placement, and gains saturate at moderate widths (K=4–8), where snapshot overhead stays near 4% of rollout cost.
  • The tree-weighted objective increases the fraction of gradient steps with a non-degenerate advantage signal (from 71% to 94% on SWE-bench), meaning a larger effective batch of informative updates per step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable prediction of Theorem 2 is that the empirical gradient-variance ratio Var_BPO/Var_GRPO should track E[Var(R|s_t)]/Var(R|s_0) and rise as training progresses; the paper's reported ratio moving from 0.42 to 0.58 is consistent, but this specific diagnostic is left implicit.
  • The entropy scheduler may miss states where the policy is confidently wrong but outcomes still diverge; the paper's own oracle ablation (value-disparity beats entropy by 0.6 points) suggests a cheap critic used only for branch selection is a natural untested extension.
  • The proof is environment-agnostic, so the same sibling-baseline recipe transfers to any deterministic, resumable simulator beyond LLM agents — a consequence the paper states only as future work.
  • For environments with imperfect restores, sibling returns will gain extra covariance after forking; measuring that covariance and subtracting it would yield a partial-crediting estimator, a natural robustness extension the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Branching Policy Optimization (BPO), an RL algorithm for LLM agents that exploits deterministic, snapshottable sandboxes to replace N independent rollouts with a single tree: a backbone trajectory is branched at high-entropy action boundaries; K−1 sibling rollouts per branch point are generated by restore-snapshot; per-step advantages are computed as leave-one-out sibling baselines and propagated upstream with a discount λ. The authors claim the estimator is unbiased (Theorem 1) and has strictly lower variance than GRPO/RLOO trajectory-level baselines (Theorem 2), with the reduction equal to the prefix-explained portion of return variance. Empirically, BPO is reported to outperform PPO, RLOO, GRPO, and VinePPO by 3.6–6.1 absolute points on WebShop, ALFWorld, and SWE-bench Verified with Qwen2.5-7B and Llama-3.1-8B at matched compute, and to reach baseline performance in 38% fewer policy updates.

Significance. The central idea—treating snapshot/restore as a first-class RL primitive to create correlated rollouts that share prefixes—is timely and, if correct, would be a meaningful advance for sandbox-based agent training. The variance-reduction calculation in Theorem 2 is transparent, parameter-free, and follows cleanly from the law of total variance; the explicit assumption of snapshot fidelity (Assumption 1) is honestly stated. However, the paper overclaims by extending local unbiasedness to the full propagated estimator, which is biased for pre-branch actions, and the on-policy state distribution is distorted by entropy-based branching. The empirical section also suffers from per-seed hyperparameter selection and an unclear compute-matching protocol, and no code or data is released. These issues are addressable but currently leave the central theoretical claim unsupported.

major comments (4)
  1. [§4.4, Eq. (3)–(5), Theorem 1] Theorem 1 proves unbiasedness only for the local sibling advantage at a single branch point (Eq. 3). The 'Consequently' clause refers to Eq. (5), which also includes pre-branch steps via Eq. (4). For t' < t, E[λ^{t−t'} Â_t | s_{t'}, a_{t'}] = λ^{t−t'} E[Q(s_t,a_t)−V(s_t) | s_{t'},a_{t'}], which is not A^π(s_{t'},a_{t'}) because it omits the immediate reward r_{t'} and uses a baseline that depends on a_{t'} through the future state. In a sparse-reward MDP where a_{t'} determines the terminal outcome and the branch point is terminal, the propagated advantage is zero in expectation for every pre-branch action while the true advantage is nonzero. Thus the unbiasedness theorem does not apply to the actual BPO gradient estimator; the same issue affects the variance-reduction guarantee, which is proved for the local estimator only.
  2. [§4.2, Eq. (5)–(6)] Even if local advantages were unbiased, the gradient sum in Eq. (5) is over a tree whose branch points are selected by entropy, not drawn from the on-policy occupancy measure. The policy gradient theorem requires expectation over the on-policy state distribution; the entropy-biased tree overweights high-entropy states and their successors, and no importance-weight correction is applied. Consequently, the estimator is biased for ∇J(θ) even with λ=0. The authors should either state the actual objective being optimized and analyze its bias, or add a correction that restores the on-policy measure.
  3. [§5.1, hyperparameter selection] The protocol 'pick the best per (method, environment, seed) configuration' and then report mean ± std over three seeds is statistically questionable. Selecting hyperparameters per seed on the test task invalidates the reported standard deviations as measures of variability and can materially inflate the reported gains. A proper protocol would select hyperparameters on a validation set (or at least per method/environment, not per seed) and report the corresponding test performance. Additionally, no code or data is released, which limits reproducibility of Tables 1–4 and Figs. 2–3.
  4. [§5.1, Table 2; compute matching] The compute-matching description is ambiguous. BPO with M=4, K=4 yields 13 rollouts, yet GRPO is said to use N=8; the statement 'we then sub-sample to match' is not a well-defined procedure, since dropping individual rollouts from a sibling set changes the leave-one-out baseline and its variance. Table 2's K=1 row lists M=12, which contradicts the stated budget N=1+M(K−1)=1; this appears to be a different computational budget and undermines the 'compute-matched' interpretation of the ablation.
minor comments (5)
  1. [Eq. (5)] The displayed equation is typeset incorrectly: the PPO-clip objective should read min(ρ(s,a)Â, clip(ρ(s,a),1−ε,1+ε)Â); the current text has duplicated 'bA' and misplaced parentheses.
  2. [Author affiliations] There is a typo: 'McGill Univeristy' should be 'McGill University'.
  3. [Bibliography [11]] Reference [11] ('Buy 4 reinforce samples, get a baseline for free!') is incomplete; please provide the full author list, venue, and year.
  4. [Algorithm 1] The loss in Eq. (5) includes a KL term β KL(πθ || πref), but Algorithm 1 does not show how the reference-policy KL is computed or weighted. Please clarify the implementation.
  5. [§4.2, Eq. (6)] The minimum spacing Δ_min is specified in tokens, but the entropy is computed at 'decision boundaries' (end of an action). Please clarify whether the spacing is measured in tokens or in agent steps, and whether the entropy is computed over the full action distribution or the first-token distribution.

Circularity Check

0 steps flagged

No significant circularity: the variance-reduction and unbiasedness theorems are self-contained derivations from standard policy-gradient identities, and the empirical gains are measured against external baselines rather than fitted into the theory.

full rationale

I walked the paper's derivation chain and found no step where a claimed prediction or first-principles result is equivalent to its inputs by construction. Theorem 1 is a standard leave-one-out baseline unbiasedness argument: conditional on the branch state s_t, the sibling returns are i.i.d. estimates of Q^π(s_t, a) for the chosen action and V^π(s_t) for the others, so the estimator's conditional expectation is exactly the advantage. This does not assume the conclusion; it follows from the definitions of Q and V. Theorem 2 is also a parameter-free law-of-total-variance computation comparing the leave-one-out variance of K i.i.d. returns from s_0 with K siblings sharing prefix τ_{0:t}; the algebra yields Var(ABPO) = K/(K−1) E[Var(G|s_t)] and Var(AGRPO) = K/(K−1)(E[Var(G|s_t)] + Var(V^π(s_t))), so the claimed reduction is derived, not fitted. The empirical results (Table 1) are compared with PPO, RLOO, GRPO, and VinePPO at matched sample budgets, and the variance plots are separate measurements; they are not used to define the estimator. The entropy-based branch scheduler is an empirical design choice, and the paper ablates it against uniform, random, equally spaced, low-entropy, and oracle schedules (Table 3), so its benefit is tested rather than assumed. I found no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. Assumption 1 (snapshot fidelity) is an explicitly stated structural assumption; the theoretical results are conditional on it, which is a limitation but not circularity. The correctness concern about Eq. (4)'s propagation to pre-branch actions—if valid—would be a proof-bias flaw, not a circular-input flaw, and therefore outside this pass's scoring rubric. Overall, the derivation chain is self-contained and the central claims do not reduce to their inputs.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 1 invented entities

The central derivation (Theorems 1–2) rests on standard math plus the snapshot-fidelity assumption. The empirical headline is supported by fitted hyperparameters (M, K, delta_min, lambda, beta, LR) and an entropy heuristic. The tree structure is an algorithmic artifact, not an invented entity in the graviton sense. Overall the paper's theoretical contribution is relatively lean in free parameters; its empirical conclusions are only partially supported because the hyperparameters are tuned per experiment.

free parameters (7)
  • M (branch count) = 4 (default); 12 for K=1; 0.8–1.7 for large K
    Per-prompt branching budget; chosen by sweep in Tables 2–3. Entropy schedule outperforms random/equal/lowest-entropy, but M is hand-tuned per environment.
  • K (branch width) = 4 (default); 2, 8, 16 in ablations
    Number of siblings per branch point; K=4–8 near-optimal in Table 2. Theoretical Proposition 1 suggests K=2 but is idealised.
  • delta_min (min branch spacing) = 64 tokens
    Introduced ad hoc to avoid clustering branch points; no theory or ablation for this value.
  • lambda (propagation discount) = 0.95
    Temporal decay for credit propagation in Eq. (4); paper reports a broad plateau in [0.9, 0.99], but the exact default is hand-picked.
  • epsilon (PPO clip) = 0.2
    Standard PPO clip; introduced as a fixed hyperparameter, not fitted.
  • beta (KL coefficient) = 0.05
    KL weight in Eq. (5); swept over {0.01, 0.05} in setup, best chosen.
  • learning rate = 2e-6
    Swept over {1,2,5}e-6 per (method, env, seed); best chosen.
axioms (6)
  • domain assumption Assumption 1: Snapshot fidelity — rest(snap(s)) recovers identical transition distribution for every on-policy state.
    Load-bearing for Theorem 2's 'conditional on s_t' variance comparison and for the algorithmic procedure itself. Required in §3, listed as Assumption 1.
  • domain assumption Markov property of the sandbox MDP: state s is the full view of the sandbox (shell history, DOM, etc.).
    Standard MDP modeling; stated in §3. Needed for Q^pi and V^pi to be well-defined.
  • standard math The policy gradient theorem and the unbiasedness of replacing A with an unbiased advantage estimator.
    Invoked in Eq. (1) and Theorem 1 proof; standard result, not proved in the paper.
  • standard math Law of total variance and i.i.d. draws from pi_theta given s_t.
    Used in Theorem 2 and Corollary 1; standard.
  • ad hoc to paper The policy entropy H_t is a good proxy for which branching points maximally reduce return variance.
    The whole scheduler (§4.2) relies on this. It is validated empirically in Table 3 but never derived; the oracle value-disparity schedule still outperforms it.
  • domain assumption Return samples are exchangeable within a branch point and across branch points when computing gradient variance in Proposition 1.
    The proof sketch treats M branch-point advantages as approximately independent; this is an approximation, not a theorem.
invented entities (1)
  • Rollout tree T_x with backbone + K-1 sibling sub-trajectories per branch point independent evidence
    purpose: Computational structure for variance reduction in policy gradient; no new physical entity.
    The tree is an algorithm construct, not a hypothetical physical entity; its utility is evidenced by the empirical gains and the variance bound.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/2BLPPV5Z

@misc{pith2026260714171,
  author       = {Pith},
  title        = {Pith review of: Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BLPPV5Z}},
  note         = {Machine review of arXiv:2607.14171}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes. State-of-the-art algorithms such as PPO, RLOO, and GRPO inherit their rollout topology from RLHF: for each prompt, N independent trajectories are sampled from the initial state, and an advantage is computed by subtracting a group baseline. This design ignores a defining property of agent sandboxes. They are deterministic, snapshottable, and resumable from any intermediate state. We argue that this property enables a fundamentally different rollout topology: rather than N independent trees of depth T, one can construct a single tree of N leaves whose siblings share prefixes, and therefore share variance. We instantiate this idea as Branching Policy Optimization (BPO), a sandbox-native RL algorithm that (i) adaptively snapshots the sandbox at high-entropy decision points along a backbone trajectory, (ii) forks K alternative actions per branch point and rolls out each to termination, and (iii) computes per-step advantages from sibling returns rather than from independent prompts. We prove this estimator is unbiased and has strictly lower variance than the trajectory-level baseline, with the reduction equal to the prefix-explained portion of return variance. On WebShop, ALFWorld, and SWE-bench Verified with Qwen2.5-7B and Llama-3.1-8B backbones, BPO improves success by 3.6--6.1 absolute points over GRPO and RLOO at matched compute, halves gradient-norm variance, and matches the best baseline using 38% fewer policy updates.

Figures

Figures reproduced from arXiv: 2607.14171 by Bowei He, Xiaokun Zhang, Xue Liu, Yankai Chen.

Figure 1
Figure 1. Figure 1: Schematic of BPO. The backbone trajectory (black) is sampled greedily￾ish from πθ; high-entropy steps (red dots) trigger sandbox snapshots and K−1 sibling forks (grey). The sibling-baseline advantage is computed locally at each branch point and propagated to the shared prefix. 4.1 Rollout trees and the sibling-baseline advantage For each prompt x, BPO constructs a rooted tree Tx whose root is s0(x). The tr… view at source ↗
Figure 2
Figure 2. Figure 2: Training curves on three sandbox benchmarks. Shaded regions denote [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Empirical gradient-norm variance over training. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 9 linked inside Pith

  1. [1]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Ahmadian, A., Cremer, C., Gall´ e, M., Fadaee, M., Kreutzer, J., Pietquin, O., ¨Ust¨ un, A., Hooker, S.: Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12248–12267 (2024)

    2024

  2. [2]

    arXiv preprint arXiv:2212.08073 (2022)

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022)

    Pith/arXiv arXiv 2022

  3. [3]

    Advances in neural information processing systems30(2017)

    Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in neural information processing systems30(2017)

    2017

  4. [4]

    Transactions on Machine Learning Research (2023)

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., Zhang, T.: Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research (2023)

    2023

  5. [5]

    arXiv preprint arXiv:2407.21783 (2024)

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  6. [6]

    arXiv preprint arXiv:2308.08998 (2023)

    Gulcehre, C., Paine, T.L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al.: Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998 (2023)

    Pith/arXiv arXiv 2023

  7. [7]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  8. [8]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., Hu, Z.: Reasoning with language model is planning with world model. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 8154–8173 (2023)

    2023

  9. [9]

    Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.: Swe-bench: Can language models resolve real-world github issues? In: International Conference on Learning Representations. vol. 2024, pp. 54107– 54157 (2024)

    2024

  10. [10]

    arXiv preprint arXiv:2410.01679 (2024)

    Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Roux, N.L.: Vineppo: Refining credit assignment in rl training of llms. arXiv preprint arXiv:2410.01679 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Kool, W., van Hoof, H., Welling, M.: Buy 4 reinforce samples, get a baseline for free!

  12. [12]

    In: International Conference on Learning Representations

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K.: Let’s verify step by step. In: International Conference on Learning Representations. vol. 2024, pp. 39578–39601 (2024) 14 Bowei He et al

    2024

  13. [13]

    In: International Conference on Learning Representations (2018)

    Liu, H., Feng, Y., Mao, Y., Zhou, D., Peng, J., Liu, Q.: Action-dependent control variates for policy optimization via stein identity. In: International Conference on Learning Representations (2018)

    2018

  14. [14]

    Advances in neural information processing systems35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)

    2022

  15. [15]

    In: International Conference on Learning Representations

    Qi, Z., Liu, X., Iong, I.L., Lai, H., Sun, X., Sun, J., Yang, X., Yang, Y., Yao, S., Xu, W., et al.: Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In: International Conference on Learning Representations. vol. 2025, pp. 79791–79821 (2025)

    2025

  16. [16]

    In: International Conference on Learning Representations

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al.: Toolllm: Facilitating large language models to master 16000+ real-world apis. In: International Conference on Learning Representations. vol. 2024, pp. 9695–9717 (2024)

    2024

  17. [17]

    Advances in neural information processing systems36, 53728–53741 (2023)

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems36, 53728–53741 (2023)

    2023

  18. [18]

    In: Bengio, Y., LeCun, Y

    Schulman, J., Moritz, P., Levine, S., Jordan, M.I., Abbeel, P.: High- dimensional continuous control using generalized advantage estimation. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Rep- resentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016)

    2016

  19. [19]

    arXiv preprint arXiv:1707.06347 (2017)

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  20. [20]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Advances in neural information processing systems36, 8634–8652 (2023)

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems36, 8634–8652 (2023)

    2023

  22. [22]

    In: International Conference on Learning Representations (2021)

    Shridhar, M., Yuan, X., Cote, M.A., Bisk, Y., Trischler, A., Hausknecht, M.: Alfworld: Aligning text and embodied environments for interactive learning. In: International Conference on Learning Representations (2021)

    2021

  23. [23]

    nature550(7676), 354–359 (2017)

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.: Mastering the game of go without human knowledge. nature550(7676), 354–359 (2017)

    2017

  24. [24]

    Advances in neural information processing systems12(1999)

    Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems12(1999)

    1999

  25. [25]

    In: International Conference on Machine Learning

    Wan, Z., Feng, X., Wen, M., Mcaleer, S.M., Wen, Y., Zhang, W., Wang, J.: Alphazero-like tree-search can guide large language model decoding and training. In: International Conference on Machine Learning. pp. 49890–49920. PMLR (2024) Sandbox-Native Language Agent Reinforcement Learning 15

    2024

  26. [26]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., Sui, Z.: Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9426–9439 (2024)

    2024

  27. [27]

    arXiv preprint arXiv:2504.20073 (2025)

    Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M.N., Liu, L., et al.: Ragen: Understanding self- evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Advances in Neural Information Processing Systems38, 78500–78525 (2026)

    Wei, Y., Duchenne, O., Copet, J., Carbonneaux, Q., Zhang, L., Fried, D., Synnaeve, G., Singh, R., Wang, S.: Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. Advances in Neural Information Processing Systems38, 78500–78525 (2026)

    2026

  29. [29]

    Machine learning8(3), 229–256 (1992)

    Williams, R.J.: Simple statistical gradient-following algorithms for connec- tionist reinforcement learning. Machine learning8(3), 229–256 (1992)

    1992

  30. [30]

    In: The First Workshop on System-2 Reasoning at Scale, NeurIPS’24

    Xie, Y., Goyal, A., Zheng, W., Kan, M.Y., Lillicrap, T.P., Kawaguchi, K., Shieh, M.: Monte carlo tree search boosts reasoning via iterative prefer- ence learning. In: The First Workshop on System-2 Reasoning at Scale, NeurIPS’24

  31. [31]

    arXiv preprint arXiv:2412.15115 (2024)

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Advances in Neural Information Processing Systems37, 50528– 50652 (2024)

    Yang, J., Jimenez, C.E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O.: Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems37, 50528– 50652 (2024)

    2024

  33. [33]

    Advances in Neural Information Processing Systems35, 20744–20757 (2022)

    Yao, S., Chen, H., Yang, J., Narasimhan, K.: Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems35, 20744–20757 (2022)

    2022

  34. [34]

    Advances in neural information processing systems36, 11809–11822 (2023)

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems36, 11809–11822 (2023)

    2023

  35. [35]

    In: International Conference on Learning Representations (ICLR) (2023)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: International Conference on Learning Representations (ICLR) (2023)

    2023

  36. [36]

    Advances in Neural Information Processing Systems35, 15476–15488 (2022)

    Zelikman, E., Wu, Y., Mu, J., Goodman, N.: Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems35, 15476–15488 (2022)

    2022

  37. [37]

    Advances in Neural Information Processing Systems37, 64735–64772 (2024)

    Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., Tang, J.: Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems37, 64735–64772 (2024)

    2024

  38. [38]

    In: International Conference on Learning Representations

    Zhou, S., Xu, F.F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al.: Webarena: A realistic web environment for building autonomous agents. In: International Conference on Learning Representations. vol. 2024, pp. 15585–15606 (2024)

    2024

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.