Pith. sign in

REVIEW 5 major objections 5 minor 27 references

RRO: LLM Agent Optimization Through Rising Reward Trajectories

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reward-rising stopping rule makes LLM agent training cost a fraction of standard exploration, the paper claims.

desk verdict A plausible new stopping heuristic for process-supervision data collection, but the theoretical grounding and experiments do not yet support the efficiency claim. read the letter →

arxiv 2505.20737 v1 pith:EY5EIUT4 submitted 2025-05-27 cs.AI

Zilong Wang , Jingfeng Yang , Sreyashi Nag , Samarth Varshney , Xianfeng Tang , Haoming Jiang , Jingbo Shang , Sheikh Muhammad Sarwar This is my paper
classification cs.AI
keywords LLMagentsprocessrewardmodelsrisingoptimizationdynamicexplorationdirectpreferenceagentplanningWebShopInterCode-SQL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM agents can be trained more efficiently by noticing when the process reward goes up along a trajectory, instead of exploring many candidate actions at every step. The authors propose stopping the sampling of next actions at the first candidate whose estimated reward is at least as high as the previous step's reward, then building DPO preference pairs from that sample. They claim this 'reward rising' criterion gives better task performance with far fewer sampled trajectories, and report WebShop reward 62.91 with 1.86 average samples and InterCode-SQL reward 55.08 with 1.64 average samples, beating fixed-size exploration and IPR baselines.

What carries the argument

The central object is the reward-rising stopping criterion, a deterministic rule for ending next-action exploration. The process reward $r_{PRM}(s_t,a_t)$ is computed as the average outcome reward of $m$ Monte Carlo rollouts following the prefix $e_{1:t}$, i.e. $r_{PRM}(s_t,a_t)=\frac{1}{m}\sum_j r_{ORM}(u,e_{1:t}\oplus \hat{e}^{(j)}_{t+1:n})$. The machinery works by comparing this estimate to the preceding step's estimate, terminating exploration at the first candidate with $r_{PRM}(s_t,a_t)\geq r_{PRM}(s_{t-1},a_{t-1})$, and using that pair as DPO training data.

What would settle it

A direct test is to measure the variance of the Monte Carlo process reward for a fixed step across repeated rollouts. If independent re-draws of the $m$ rollouts frequently flip the sign of the reward difference between adjacent steps, then the rising-reward candidate is largely noise, and RRO's preference pairs lose their meaning.

Watch

Extended reading notes

Core claim

The central claim is that the relative trend of process rewards between neighboring reasoning steps—whether the reward is rising—is a sufficient signal for collecting preference data for LLM agent optimization. Instead of sampling a fixed number of next-action candidates and picking the best, RRO samples action candidates one by one and stops at the first candidate whose Monte Carlo process reward estimate $r_{PRM}(s_t, a_t)$ is at least the preceding step's reward $r_{PRM}(s_{t-1}, a_{t-1})$. The chosen action is the reward-rising candidate; the rejected action is the lowest-reward candidate seen during that short exploration. The paper grounds this in an averaging argument: because the process reward at step $t$ is the expectation over next actions of the process reward at step $t+1$, at least one next action must have process reward at least as large, so a rising step exists and the stopping rule can terminate.

Load-bearing premise

The main assumption is that the average outcome reward from a small number of Monte Carlo rollouts honestly reflects how good a step really is, so that the first step where this noisy estimate goes up is a genuinely better action and not just a lucky sample.

Editorial extensions

If this is right

  • If correct, process-supervision data for agent training can be collected without enumerating wide candidate sets, lowering the compute barrier for scaling PRM-style training.
  • The fixed-cost trade-off of exploration—more samples giving stronger preference pairs but at higher cost—is replaced by a per-step adaptive stopping rule that spends computation only until a relative improvement is found.
  • Because the stopping rule is policy-agnostic and the reward signal is the Monte Carlo outcome average, the method should transfer to other agent benchmarks and base models without changing the pipeline.
  • The observed shift in rising-reward proportions (e.g., InterCode-SQL Final-stage 40.87% to 45.29%) suggests the training signal concentrates on steps where the model is improving, which may be the mechanism behind the efficiency gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's existence theorem guarantees a rising action exists in expectation, but not that the first sampled candidate with a rising estimate is the best or even a strong candidate; the practical success depends on the quality of the $m$-rollout estimate, which the paper does not characterize.
  • The stopping rule resembles a one-step lookahead in a stochastic process whose reward is a Monte Carlo average; if the variance of that average is high relative to the per-step reward difference, the rule could stop on noise, a scenario the theory does not rule out.
  • Since the rejected action is taken from a small set of sampled candidates, the preference signal is relative to the local sampling budget rather than the true action space; on benchmarks with high-quality first samples, a larger budget would likely reduce gains.
  • The method is compatible with any outcome reward that can be averaged, so it could be tested immediately on multi-step code generation or math verification tasks where per-step rewards are unavailable or expensive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Reward Rising Optimization (RRO), a method for training LLM agents in which the next-action exploration is expanded dynamically until the Monte Carlo estimate of the process reward for a candidate action is at least as large as the estimate for the preceding action. The identified candidate is then used as the preferred action in a DPO preference pair, and the method is evaluated on WebShop and InterCode-SQL, reporting higher rewards with substantially fewer sampled trajectories than Fixed-sized Exploration and IPR baselines. The authors also derive a mathematical argument in Section 3.3 intended to justify the existence of an action with a rising reward, and they analyze the proportion of rising-reward actions in Section 4.6.

Significance. If the mechanism worked as described, RRO would address a real scalability problem in process reward modeling, and the idea of using local reward trends as a stopping rule is interesting. However, the paper's central claim is not supported by the current evidence: the mathematical derivation equates a finite-sample Monte Carlo estimate with a true conditional probability, making the existence argument invalid; Table 2 evaluates the method using the very signal it selects on, which is circular; and Table 1 reports single-point numbers with no error bars, seeds, or significance tests. The empirical improvements are small and may well be noise. I see no machine-checked proofs, reproducible code, or parameter-free derivations in the manuscript, and the reported results are not sufficient to establish the performance or efficiency claims.

major comments (5)
  1. [§3.3, equations defining rPRM] The derivation writes rPRM(st,at) = (1/m) Σ rORM(u,e1:t ⊕ ê(j)t+1:n) and then asserts that this equals P(1|e1:t) for the current state-action pair. This equality does not hold for a finite Monte Carlo sample; it holds only in expectation over rollouts. The law-of-total-probability expansion and the weighted-average argument that follow therefore apply to the true success probability, not to the empirical average defined in the paper. Consequently, the claimed existence of a sampled action with rPRM(st+1,a) ≥ rPRM(st,at) does not follow from the equations as written.
  2. [§3.2, Algorithm 1 lines 15-20, and §3.3] Even if the existence statement in §3.3 were correct, it quantifies over the support of P(at+1|e1:t), whereas Algorithm 1 samples candidates from the policy πθ(·|e1:t-1). Since πθ and P(at+1|e1:t) can be very different, there is no guarantee that the repeat-until loop terminates quickly or at all, and the paper provides no bound on the expected number of samples. In addition, the stopping comparison compares two noisy Monte Carlo estimates, so a positive differential can simply be an upward fluctuation of the estimator. The DPO preference pairs constructed in lines 21-22 are then unreliable because the chosen action may not be better than the rejected action in any ground-truth sense.
  3. [§4.6, Table 2] Table 2 reports the proportion of actions with rising rewards as evidence that RRO improves reward alignment. However, this is exactly the signal used by Algorithm 1 to select preferred actions and to decide when to stop sampling. The comparison therefore measures whether the optimized policy produces the local trend that the optimization procedure was trained to increase, not whether that trend corresponds to actual action quality or task success. As a validation of the reward-rising mechanism, the table is circular.
  4. [§4.4, Table 1] The central empirical claim is based on single-point comparisons: WebShop 62.91 versus 61.39 for IPR and 61.20 for Fixed-sized Exploration, and InterCode-SQL 55.08 versus 54.68 for Fixed-sized Exploration. The paper provides no error bars, no multiple-seed results, and no significance tests, so differences of 1.5 and 0.4 points are not shown to be real. The efficiency claim is likewise unsupported because the average sample counts (1.86 and 1.64) are reported without variance or a precise account of how they are averaged across tasks and steps.
  5. [§3.1-§3.4] The paper describes 'mathematical groundings' and a 'Bellman equation' alignment for RRO, but the formulation has no explicit assumptions about the policy distribution, the Monte Carlo sample size m, or the relationship between the finite-sample estimate and the true probability. Without such assumptions, no theorem about the stopping rule can be derived, and the statements in §3.3 do not meet the standard of a mathematical grounding.
minor comments (5)
  1. [§4.2 and §4.3] The base model is referred to as Gemma-22B in §4.2 and in Table 1, but §4.3 says 'Gemma-2 2B-base'; please clarify which model was actually used, as this is necessary for reproducibility.
  2. [Algorithm 1, lines 21-22] There is a typographical error in the algorithm: 'rPPM,t' appears twice where 'rPRM,t' is meant.
  3. [References] Lightman et al. appears twice with the same title and venue (2023 and 2024 entries); this duplicate should be consolidated.
  4. [§4.3 and §4.4] The number of DPO iterations and the size of the SFT dataset are not specified, which makes it difficult to compare the sample counts across methods or to reproduce the training pipeline.
  5. [Figure 2] The x-axis label 'Number of Required Sampled Trajectories (RRO vs IPR)' is ambiguous: for IPR, the horizontal axis appears to be a round index, whereas for RRO it is an average per decision step; this conflation should be explained or relabeled.

Circularity Check

2 steps flagged · score 4.0 of 10

Mathematical 'grounding' for reward-rising stopping is a definitional property of expectations, and the rising-reward analysis is self-referential, while final benchmark results remain external.

  1. self definitional [Section 3.3, 'Math Grounding' (process-reward definition and existence claim; used to justify Algorithm 1's stopping rule)]
    "rPRM(st,a t) = ∑ at+1 rPRM(st+1,a t+1)P(a t+1|e1:t). Since P(a t+1|e1:t)∈[ 0, 1], there must exist at least one sampled action a(τ) t+1 such that its corresponding process reward satisfies: rPRM(st+1,a (τ) t+1)≥r PRM(st,a t)."

    The claimed existence result is a direct algebraic consequence of writing rPRM(st,at) as a weighted average, i.e., an expectation, of rPRM(st+1,·); every finite weighted average has an element at least as large as the average. No property of the policy, the sampling procedure, or the Monte Carlo rollouts is used. The paper also identifies the finite-m sample average rPRM with the true probability P(1|e1:t), silently replacing a noisy estimator with a population expectation. The 'grounding' thus restates the definition of expectation and cannot independently justify the stopping rule, which in Algorithm 1 decides on the pair of noisy estimates rPRM,t and rPRM,t−1 for the candidates actually sampled.

  2. other [Section 4.6, 'Rising Rewards Analysis' and Table 2; compare with Algorithm 1 stopping criterion]
    "We analyze the reward tendency in the trajectories generated by the supervised fine-tuned model πSFT and the reward rising optimized model πRRO. Specifically, we check if the neighboring actions demonstrate a rising tendency and calculate the proportion of actions with the rising rewards."

    This analysis measures the proportion of actions satisfying the same 'rising reward' condition (rPRM,t ≥ rPRM,t−1) that Algorithm 1 uses as its stopping/selection criterion while curating RRO's DPO training data. Reporting that πRRO has a higher proportion of such actions than πSFT is therefore partly checking whether the optimized policy reproduces the definition used to build its own training signal, not an independent validation of the mechanism. The external task rewards and average sample counts in Table 1 are the independent evidence; the rising-reward proportion in Table 2 is a self-referential diagnostic.

full rationale

The paper's headline empirical claims (WebShop reward 62.91 with 1.86 average samples; InterCode-SQL reward 55.08 with 1.64 average samples) are benchmark results against external baselines and are not circular: they depend on actual environment rewards and on the measured number of rollouts under the stopping rule, not on restating the method's definitions. The circularity is localized to the paper's theoretical and analytical self-justification. In Section 3.3 the paper defines rPRM as an average of outcome rewards, rewrites it using the law of total probability, and concludes that some action must have rPRM at least the current value. That conclusion is an immediate property of any weighted average or expectation, so the 'math grounding' for the reward-rising stopping rule is a definitional tautology rather than an independent feasibility result; it also silently equates a finite-m Monte Carlo estimate with the population probability P(1|e1:t). In Section 4.6, the 'rising rewards analysis' measures the proportion of actions satisfying the very rising-reward condition used in Algorithm 1's stopping rule, making that analysis partly self-referential rather than an independent test. No load-bearing self-citation or imported uniqueness theorem appears. The Monte Carlo variance and small-sample concerns raised about rPRM are correctness risks, not additional circularity. Overall, the circularity is partial and localized: the stated mathematical grounding and the rising-reward validation reduce to definitions, but the central empirical comparison against baselines remains external and independently meaningful.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces a new stopping rule but no new physical or conceptual entity; all quantities are standard RL/LLM objects. The ledger therefore lists the unreported hyperparameters and the unproven statistical assumptions that the central claims rest on.

free parameters (3)
  • m: number of Monte-Carlo rollouts per process-reward estimate
    Chosen by hand and never reported; it controls the accuracy of every rPRM value and the real cost of each candidate sample.
  • DPO beta (temperature)
    Standard DPO hyperparameter not reported in the paper; central to the optimization step in Section 3.4.
  • number of DPO rounds / SFT epochs
    Training schedule not reported; the '1.86 samples' average depends on when training stops.
assumptions (4)
  • domain assumption The Monte Carlo average of outcome rewards over m rollouts equals the true success probability P(1|e_{1:t}).
    Section 3.3 writes rPRM(s_t,a_t) = P(1|e_{1:t}) without accounting for finite-sample error or the dependence of rollouts on the policy.
  • domain assumption The action distribution has enough mass on rising-reward actions that the repeat-until loop in Algorithm 1 terminates and identifies a useful preferred action.
    Section 3.3 only proves existence of some action with r >= previous in the support; it does not prove the sampling process will draw such an action quickly, nor that the first such draw is a strong 'chosen' action.
  • ad hoc to paper A single-step increase in the noisy process-reward estimate is a reliable signal of action quality.
    This is the core heuristic of RRO (Section 3.2); no variance or false-positive analysis is provided.
  • domain assumption DPO applied to these automatically generated preference pairs improves final task reward.
    Section 3.4 relies on DPO's standard assumptions; the paper does not validate that the pairs satisfy them, such as the chosen action being consistently preferred.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RRO: LLM Agent Optimization Through Rising Reward Trajectories." pith.science (2026). https://pith.science/paper/EY5EIUT4

@misc{pith2026250520737,
  author       = {Pith},
  title        = {Pith review of: RRO: LLM Agent Optimization Through Rising Reward Trajectories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EY5EIUT4}},
  note         = {Machine review of arXiv:2505.20737}
}
read the original abstract

Large language models (LLMs) have exhibited extraordinary performance in a variety of tasks while it remains challenging for them to solve complex multi-step tasks as agents. In practice, agents sensitive to the outcome of certain key steps which makes them likely to fail the task because of a subtle mistake in the planning trajectory. Recent approaches resort to calibrating the reasoning process through reinforcement learning. They reward or penalize every reasoning step with process supervision, as known as Process Reward Models (PRMs). However, PRMs are difficult and costly to scale up with a large number of next action candidates since they require extensive computations to acquire the training data through the per-step trajectory exploration. To mitigate this issue, we focus on the relative reward trend across successive reasoning steps and propose maintaining an increasing reward in the collected trajectories for process supervision, which we term Reward Rising Optimization (RRO). Specifically, we incrementally augment the process supervision until identifying a step exhibiting positive reward differentials, i.e. rising rewards, relative to its preceding iteration. This method dynamically expands the search space for the next action candidates, efficiently capturing high-quality data. We provide mathematical groundings and empirical results on the WebShop and InterCode-SQL benchmarks, showing that our proposed RRO achieves superior performance while requiring much less exploration cost.

Figures

Figures reproduced from arXiv: 2505.20737 by the authors.

Figure 1
Figure 1. Reward Rising Optimization: We dynamically adjust the scope of the next action exploration and prioritize action steps that exhibit a “rising reward trend” compared to their predecessors, avoiding the over-exploration or under-exploration. (a) The illustration of Direct Preference Optimization (Rafailov et al., 2024) where a pair of the preference data is used to optimize the LLM agent. (b) The process reward estima… view at source ↗
Figure 2
Figure 2. Sampling efficiency of RRO and IPR on WebShop and InterCode-SQL. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 5 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [4]

    Fireact: Toward language agent fine-tuning

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023 a

    arXiv 2023

  4. [5]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023 b

    work page 2023

  5. [6]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [7]

    Complexity-based prompting for multi-step reasoning

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2022

    work page 2022

  7. [8]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  8. [9]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

Show all 27 references
  1. [10]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024

    2024

  2. [11]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  3. [12]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  4. [13]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [14]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  6. [15]

    Trial and error: Exploration-based trajectory optimization of LLM agents

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computatio...

    2024 doi

  7. [16]

    Math-shepherd: Verify and reinforce LLM s step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLM s step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meetin...

    2024 doi

  8. [17]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024

  9. [18]

    Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision

    Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658, 2024 c

    2024 arXiv

  10. [19]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  11. [20]

    Watch every step! llm agent learning via iterative step-level process refinement

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! llm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406.11176, 2024

    2024 arXiv

  12. [21]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36: 0 23826--23854, 2023

    2023

  13. [22]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volum...

    2022

  14. [23]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  15. [24]

    Debug like a human: A large language model debugger via verifying runtime execution step by step

    Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step by step. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, T...

    2024 doi

  16. [25]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning R...

    2023

  17. [26]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  18. [27]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  19. [28]

    Math-Shepherd: Verify and Reinforce LLM s Step-by-step without Human Annotations

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1992 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.