Pith. sign in

REVIEW 3 major objections 4 minor 40 references

For long-horizon agents, a frozen model's answer confidence becomes a per-tool reward

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:06 UTC pith:VA36OPPE

load-bearing objection TRACE is a plausible dense-credit recipe for agentic RL, but single-run evidence and a private-reasoning reward-hacking channel mean the headline gains should be treated as provisional, not established. the 3 major comments →

arxiv 2607.13988 v1 pith:VA36OPPE submitted 2026-07-15 cs.LG

TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

Leitian Tao , Baolin Peng , Wenlin Yao , Tao Ge , Hao Cheng , Mike Hang Wang , Jianfeng Gao , Sharon Li This is my paper
classification cs.LG
keywords credit assignmenttemporal differencereinforcement learningLLM agentstool usedense rewardlog-probabilitylong-horizon search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that turn-level credit for long-horizon agents can be computed from a frozen reference model alone, with no critic, process labels, or strong judge. The key idea is that a trajectory prefix makes the gold answer more predictable as it gathers evidence, so the temporal-difference change in the gold answer's log-probability is a valid dense reward for each tool call. Combined with an outcome-level advantage, this signal lets pure RL train competitive web-search agents: on the closed-web BrowseComp-Plus benchmark, a 4B model rises from 7.2 to 35.6 and a 30B-A3B model from 8.4 to 42.6. The paper argues this matters because outcome-only training cannot distinguish useful actions inside a failed rollout from the eventual mistake, and TRACE supplies that distinction without extra supervision.

Core claim

On the paper's own terms, the central discovery is that a frozen reference model's gold-answer log-probability, transformed into a log-ratio state value, gives a stable and telescope-friendly progress signal for agentic reinforcement learning. TRACE scores each prefix by the average gold-answer log-probability, defines the remaining gap with a small offset, sets the state value to the log ratio of the initial gap to the current gap, and rewards each tool call by the one-step temporal-difference change in that value, with a K-step backup and a terminal outcome anchor. Because the one-step changes telescope, redundant tool calls cannot inflate cumulative credit. The paper reports that this den

What carries the argument

The central object is the log-ratio state value V(S_k) = log(d_0 / d_k), where d_k is the remaining gap in the frozen reference model's average gold-answer log-probability for prefix k. The defining identity is telescoping: summing per-turn TD changes delta_k = V(S_{k+1}) - V(S_k) yields only V(S_T) - V(S_0), so total credit is endpoint-determined and trajectory padding cannot inflate it. This value plays the role a learned critic would play, but it is computed from a frozen model and combined with an outcome-level advantage.

Load-bearing premise

The load-bearing premise is that a transcript with relevant evidence makes the gold answer easier for a fixed reference model to predict; if that monotonicity fails, the dense TD rewards are misaligned.

What would settle it

Construct a rollout in which the policy copies the gold answer into its private reasoning before any search and check whether the frozen reference model's gold-answer log-probability spikes; if it does, the monotonicity premise fails. Alternatively, run the method on a long-horizon task with long or open-ended outputs, which the paper identifies as outside its current scope, and see whether turn credit tracks real progress.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Outcome-only RL for long-horizon agents can be made dense without a critic, step labels, or a strong judge, making verifier-plus-progress training practical at scale.
  • Base chat models already contain useful search primitives; a dense credit signal can organize them into effective multi-turn behavior without cold-start SFT or agentic mid-training.
  • The learned search strategy transfers across corpora, retrieval engines, and languages, suggesting the dense signal is not merely overfitting a closed corpus.
  • Training curves start improving earlier and converge faster, indicating turn-level credit accelerates long-horizon learning rather than only rescaling final rewards.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same log-ratio TD credit should apply to any task with a compact verifiable answer, such as grounded single-answer tool use; the paper only tests search, so the breadth is open.
  • Inference: the method's main risk is reward hacking by answer leakage into private reasoning; a controlled test of reference-model sensitivity to that leakage would tell whether the monotonicity premise holds.
  • Inference: the telescoping property doubles as a diagnostic tool for post-hoc analysis, pinpointing the exact tool call that destroyed a secured answer, which the paper's qualitative examples illustrate without formalizing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TRACE, a dense turn-level reward mechanism for long-horizon agentic RL. At each tool-call boundary, a frozen reference model scores the prefix by the gold answer's average log-probability; these scores are converted into log-ratio state values, and Temporal-Difference changes in those values supply per-turn credit. The turn-level signal is combined with a standard GRPO outcome advantage. On closed-web BrowseComp-Plus, the authors report gains from 7.2 to 35.6 (Qwen3-4B) and 8.4 to 42.6 (Qwen3-30B-A3B), with transfer to open-web benchmarks, all without cold-start SFT or live-web training. The paper includes controlled comparisons against GRPO, GSPO, and GiGRPO, ablations of the credit format and hyperparameters, and qualitative trajectory analyses.

Significance. If the reported gains are robust, TRACE is a practically significant contribution: it provides dense, critic-free, label-free turn credit for long-horizon tool-use agents while keeping the verifiable outcome reward as the final anchor. The controlled setup is a strength: the RL baselines share the same backbone, browser interface, training data, and terminal reward, and the ablations directly target the design choices. The qualitative examples in Appendix A.5 concretely illustrate the intended credit-localization behavior. However, the central empirical claim rests on single training runs, and the mechanism's core assumption—that gold-answer log-probability on the full prefix is a monotone progress signal—is exposed to a concrete reward-hacking failure mode through the policy's private reasoning. These issues need to be addressed before the headline claims can be accepted at face value.

major comments (3)
  1. [§3.2, Eq. (5), Algorithm 1, §A.3] The central premise is that a transcript containing relevant evidence makes the gold answer easier for the frozen reference model. However, the reference scorer computes log π_ref(y*_t | S_k, y*_<t) on prefixes that include the policy's private reasoning (Section A.3 explicitly counts reasoning and formatting tokens as part of the assistant turn), and Algorithm 1 performs no masking of reasoning tokens. A policy can therefore raise V(S_k) and obtain positive δ_k = V(S_{k+1}) − V(S_k) by writing y* or a high-likelihood paraphrase into its chain of thought before any evidence-adding tool observation, while still failing the final verifier. The paper provides no countermeasure, no quantification of how often y* appears in reasoning before the supporting evidence, and no sensitivity analysis. Since the headline gains are presented as evidence for credit assignment, this failure mode needs to
  2. [§4.1, Tables 1–2, Figures 3–5] The manuscript states 'controlled ablations are single training runs, so small differences should be read as directional rather than as variance-adjusted conclusions.' This caveat covers all controlled RL runs, not just ablations. In particular, the Table 2 comparison between 'remaining gap' (34.6) and 'log-ratio' (35.5) is a 0.9-point difference from one seed, and Figure 5(a) shows 35.6 vs 34.5 across coefficient settings. Without repeated seeds or error bars, the central ordering of methods cannot be distinguished from noise. At minimum, the primary TRACE-vs-GRPO comparison and the credit-format ablation should be run with multiple seeds and reported with variance.
  3. [Related Work, Table 1] The contribution is empirical, yet no controlled comparison is made to the closest prior methods that derive process rewards from log-likelihood ratios or return decomposition (e.g., Yuan et al. 2024; Arjona-Medina et al. 2019). Without an adapted baseline, the observed gains cannot be attributed specifically to the TD/log-ratio formulation rather than to any dense frozen-model signal. A controlled variant implementing a log-ratio process-reward baseline on the 4B model would substantially strengthen the attribution of the improvement.
minor comments (4)
  1. [Abstract, §3.3] The abstract says the 'one-step log-ratio TD component telescopes across redundant tool calls,' but the deployed reward uses the K-step backup (K=3) and terminal fill, which intentionally break the endpoint-only telescoping property. The paper acknowledges this in §3.3, but the abstract is potentially misleading.
  2. [Figure 5(b)] The x-axis label says 'step k' but the ablation varies the look-ahead horizon K; please rename the axis to avoid confusion with the turn index k.
  3. [Table 4] The 'Legacy transform stabilizer' parameter is not used in the reported log-ratio runs but is listed in the TRACE configuration table. Please mark it clearly as a legacy/disabled parameter or remove it to avoid ambiguity.
  4. [§2.1, Eq. (2)] The notion of 'final-answer opener' used to score answer tokens is introduced informally and not reflected in the notation of S_k. Please state explicitly how the opener is constructed and whether it is included in every prefix used in Eq. (5).

Circularity Check

0 steps flagged

No significant circularity: the dense turn reward is a by-construction definition of answer-likelihood progress, not a fitted prediction; the headline gains are external empirical outcomes.

full rationale

TRACE's derivation chain is self-contained: Eq. 5 defines the reference-model gold-answer log-probability, Eq. 6 defines the log-ratio state value, and Eq. 7 defines the turn credit as a TD difference. These are definitions, not predictions derived from the benchmarks; the only assumption is the monotone-progress premise stated in Sec. 3.2, which is a correctness condition rather than a circular step. The terminal verifier remains the final anchor (Alg. 1, Eqs. 10-12), and the reported gains (7.2->35.6, 8.4->42.6) are measured on held-out BrowseComp-Plus plus transfer benchmarks, not fitted. Ablations in Fig. 5 and Table 2 select hyperparameters and reward forms, but do not fit a benchmark-derived constant. Related-work citations, including any with overlapping authors, are contextual and not load-bearing. The paper's own Sec. 6 limitation (the proxy may be unreliable for long/structured outputs) is acknowledged; the separate risk that the policy could copy the gold answer into private reasoning to inflate Eq. 5 is a reward-hacking/robustness concern, not a circular derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method's added value rests on one hard assumption (frozen-model answer log-probability equals progress) and several hand-tuned reward hyperparameters. No new physical or ontological entities are introduced.

free parameters (6)
  • epsilon_train (gap offset) = 0.1
    Offset in d_k = -l_k + epsilon; stabilizes the log when l_k is near zero; chosen by hand and no ablation is reported.
  • K (TD look-ahead horizon) = 3
    K=0 disabled gives 30.0, K=3 gives 35.6, K=7 drops to 28.9 on BrowseComp-Plus (Figure 5b), so the headline result depends on this choice.
  • gamma_td (TD discount) = 0.8
    Discount for delayed evidence in the K-step backup; no ablation is shown.
  • lambda_term (terminal-outcome scale) = 2.0
    Controls the strength of the terminal-outcome fill; no ablation is shown.
  • alpha_turn (turn-reward weight) = 0.2 in Table 4; ablation best at coefficient '3' in Figure 5a
    Mixing weight for turn-level credit; ablation shows performance varies from 31.1 to 35.6, and the reported value is inconsistent with the ablation optimum unless the labels mean something else.
  • alpha_out (outcome weight) = 1.0
    Outcome anchor weight; fixed at the default value.
axioms (5)
  • domain assumption Gold-answer log-probability under the frozen reference model is a valid proxy for progress toward the correct answer.
    Stated in Section 3.2: 'a transcript that has collected relevant evidence should make that answer easier for a fixed language model to conditionally predict.' If false, the turn-level TD credit is misaligned.
  • domain assumption The exact-match verifier on synthetic answers correctly identifies success and failure.
    The outcome reward and the terminal anchor both rely on normalized exact-match correctness (Appendix A.3). Verifier errors propagate into both the outcome and turn rewards.
  • ad hoc to paper The deployed K-step reward still provides useful credit even though it intentionally breaks the one-step telescoping property.
    Section 3.3: 'The endpoint telescoping property ... applies exactly to the one-step component delta_k; the K-step backup and terminal fill intentionally trade this exact endpoint-only form.' The main runs use K=3, so the advertised non-inflation property is not active in the reported configuration.
  • domain assumption Assistant private reasoning does not leak the gold answer into the prefix in a way that inflates reference-model log-probability.
    Eq. 5 scores every prefix including private reasoning tokens; no masking or countermeasure is described in Appendix A.3. If the policy writes the gold answer into its reasoning, turn credit can be gamed.
  • standard math Standard TD and potential-based shaping identities hold.
    The method relies on the TD error identity (Eq. 3), the telescoping sum, and the potential-based shaping framework cited in Section 5 (Sutton 1988; Ng et al. 1999).

pith-pipeline@v1.3.0-alltime-deepseek · 25425 in / 17813 out tokens · 175293 ms · 2026-08-02T03:06:00.694197+00:00 · methodology

0 comments
read the original abstract

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

Figures

Figures reproduced from arXiv: 2607.13988 by Baolin Peng, Hao Cheng, Jianfeng Gao, Leitian Tao, Mike Hang Wang, Sharon Li, Tao Ge, Wenlin Yao.

Figure 1
Figure 1. Figure 1: Credit assignment at tool-call boundaries in a search trajectory. The figure illustrates a long-horizon search rollout decomposed into tool-call turns. Early search and open actions can add task-relevant evidence to the transcript even when a later branch leads to an incorrect final answer. Outcome-reward training attaches one trajectory-level advantage to all actions in the rollout, whereas TRACE computes… view at source ↗
Figure 2
Figure 2. Figure 2: TRACE reward construction at tool-call boundaries. The figure shows a long-horizon rollout segmented by tool calls and observations. Each prefix state is scored by a frozen reference model, transformed into a log-ratio value, and compared with adjacent prefix values to obtain turn-level TD credit. The turn-level advantage is then combined with the trajectory-level outcome advantage for policy optimization.… view at source ↗
Figure 3
Figure 3. Figure 3: Learning dynamics on BrowseComp-Plus. (a) Training reward on Qwen3-4B. (b) Evaluation accuracy on Qwen3-4B. (c) Training reward on Qwen3-30B-A3B. (d) Evaluation accuracy on Qwen3-30B-A3B. data, so the improvement suggests that the base backbones already contain useful reasoning and tool-use primitives, but need a denser credit signal to organize them into effective long-horizon behavior. The external-agent… view at source ↗
Figure 5
Figure 5. Figure 5: Ablations on BrowseComp-Plus with Qwen3-4B. (a) Turn-level reward coefficient. (b) TD look-ahead horizon K. (c) Reference checkpoint for answer-readiness scoring. assigning turn-level TD credit when an interaction makes the gold answer more predictable under the reference model, while still keeping the outcome reward in the objective to enforce final correctness. 0 25 50 75 100 125 150 175 Training step 2.… view at source ↗
Figure 4
Figure 4. Figure 4: Trajectory-scale dynamics on BrowseComp-Plus with Qwen3-4B. The TRACE curve increases earlier and faster than GRPO, indicating that the two training objectives expand interaction length at different rates during the early training stage. TRACE scales up interaction length faster in the beginning stage [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 31 linked inside Pith

  1. [3]

    K. Chen, Y . Ren, Y . Liu, X. Hu, H. Tian, T. Xie, F. Liu, H. Zhang, H. Liu, Y . Gong, et al. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations.arXiv preprint arXiv:2506.13651, 2025a. M. Chen, T. Li, H. Sun, Y . Zhou, C. Zhu, F. Yang, Z. Zhou, W. Chen, H. Wang, J. Z. Pan, W. Zhang, and H. Chen. ReSearch: Learning ...

    Pith/arXiv arXiv

  2. [5]

    L. Feng, Z. Xue, T. Liu, and B. An. Group-in-group policy optimization for LLM agent training. arXiv preprint arXiv:2505.10978,

    Pith/arXiv arXiv

  3. [6]

    13 TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents J. Gao, W. Fu, M. Xie, S. Xu, C. He, Z. Mei, B. Zhu, and Y . Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous RL.arXiv preprint arXiv:2508.07976,

    arXiv

  4. [7]

    doi: 10.1038/s41586-025-09422-z. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. InInternational Conference on Learning Representations,

    doi:10.1038/s41586-025-09422-z

  5. [8]

    N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica. R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents.arXiv preprint arXiv:2504.07164,

    Pith/arXiv arXiv

  6. [9]

    B. Jin, J. Yoon, P. Kargupta, S. O. Arik, and J. Han. An empirical study on reinforcement learning for reasoning-search interleaved LLM agents.arXiv preprint arXiv:2505.15117, 2025a. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han. Search-R1: Train- ing LLMs to reason and leverage search engines with reinforcement learning.arXiv ...

    Pith/arXiv arXiv

  7. [10]

    K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. WebSailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025a. M. Li, Y . Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y . Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs.arXiv preprint arXiv:2304.08244,

    Pith/arXiv arXiv

  8. [11]

    X. Li, G. Dong, J. Jin, Y . Zhang, Y . Zhou, Y . Zhu, P. Zhang, and Z. Dou. Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366, 2025b. X. Li, J. Jin, G. Dong, H. Qian, Y . Wu, J.-R. Wen, Y . Zhu, and Z. Dou. WebThinker: Empowering large reasoning models with deep research capability.arXiv preprint arXiv:2504.21776, 2...

    Pith/arXiv arXiv

  9. [13]

    Nakano, J

    14 TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders, et al. WebGPT: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332,

    Pith/arXiv arXiv

  10. [14]

    Y . Pan, D. Kong, S. Zhou, C. Cui, Y . Leng, B. Jiang, H. Liu, Y . Shang, S. Zhou, T. Wu, and Z. Wu. WebCanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373,

    Pith/arXiv arXiv

  11. [15]

    doi: 10.18653/v1/2024.acl-long.810. Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, et al. ToolLLM: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789,

    Pith/arXiv arXiv 2024

  12. [16]

    Y . Qu, Y . Jiang, B. Wang, Y . Mao, C. Wang, C. Liu, and X. Ji. Latent reward: LLM-empowered credit assignment in episodic reinforcement learning.arXiv preprint arXiv:2412.11120,

    Pith/arXiv arXiv

  13. [17]

    Rawles, S

    C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573,

    Pith/arXiv arXiv

  14. [18]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  15. [19]

    Setlur, C

    A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning.arXiv preprint arXiv:2410.08146,

    Pith/arXiv arXiv

  16. [20]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  17. [21]

    Shinn, F

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366,

    Pith/arXiv arXiv

  18. [22]

    H. Song, J. Jiang, Y . Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J.-R. Wen. R1-Searcher: Incen- tivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592,

    Pith/arXiv arXiv

  19. [23]

    H. Sun, Z. Qiao, J. Guo, X. Fan, Y . Hou, Y . Jiang, P. Xie, F. Huang, and Y . Zhang. ZeroSearch: Incentivize the search capability of LLMs without searching.arXiv preprint arXiv:2505.04588,

    Pith/arXiv arXiv

  20. [25]

    Tongyi DeepResearch Team, B. Li, B. Zhang, D. Zhang, F. Huang, G. Li, G. Chen, H. Yin, J. Wu, J. Zhou, et al. Tongyi DeepResearch technical report.arXiv preprint arXiv:2510.24701,

    Pith/arXiv arXiv

  21. [26]

    Uesato, N

    J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Pith/arXiv arXiv

  22. [28]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, et al. OSWorld: Bench- marking multimodal agents for open-ended tasks in real computer environments.arXiv preprint arXiv:2404.07972,

    Pith/arXiv arXiv

  23. [29]

    S. Yan, A. Bahloul, E. Nie, S. Schwarzmann, R. Trivisonno, V . Tresp, and Y . Ma. Memory- R2: Fair credit assignment for long-horizon memory-augmented LLM agents.arXiv preprint arXiv:2605.21768,

    Pith/arXiv arXiv

  24. [30]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. ...

    Pith/arXiv arXiv

  25. [31]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE-agent: Agent- computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793,

    Pith/arXiv arXiv

  26. [32]

    J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y . Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang. SWE-smith: Scaling data for software engineering agents. InAdvances in Neural Information Processing Systems, 2025b. Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explai...

    2018

  27. [33]

    C. Ye, Z. Yu, Z. Zhang, H. Chen, N. Sadagopan, J. Huang, T. Zhang, and A. Beniwal. Beyond correctness: Harmonizing process and outcome rewards through RL training.arXiv preprint arXiv:2509.03403,

    Pith/arXiv arXiv

  28. [34]

    L. Yuan, W. Li, H. Chen, G. Cui, N. Ding, K. Zhang, B. Zhou, Z. Liu, and H. Peng. Free process rewards without process labels.arXiv preprint arXiv:2412.01981,

    Pith/arXiv arXiv

  29. [35]

    A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y . Dong, and J. Tang. AgentTuning: Enabling generalized agent abilities for LLMs.arXiv preprint arXiv:2310.12823,

    Pith/arXiv arXiv

  30. [36]

    Zhang, Z

    C. Zhang, Z. Yang, J. Liu, Y . Han, X. Chen, Z. Huang, B. Fu, and G. Yu. AppAgent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771,

    Pith/arXiv arXiv

  31. [37]

    Zhang, X

    H. Zhang, X. Liu, B. Lv, X. Sun, B. Jing, I. L. Iong, Z. Hou, Z. Qi, H. Lai, Y . Xu, et al. AgentRL: Scaling agentic reinforcement learning with a multi-turn, multi-task framework.arXiv preprint arXiv:2510.04206, 2025a. K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin. CodeAgent: Enhancing code generation with tool- integrated agent systems for real-world repo-...

    arXiv 2024

  32. [38]

    Zheng, D

    Y . Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, P. Liu, et al. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160, 2025b. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A realistic web environment for build...

    Pith/arXiv arXiv

  33. [39]

    at the Jakarta Expo Boxing Hall

    For a fixed rollout, write ¯ℓk for the raw average gold-answer log-probability and dk =− ¯ℓk +ϵ for the remaining answer-likelihood gap. The raw log-probability difference ∆ℓ k = ¯ℓk+1 − ¯ℓk telescopes to ¯ℓT − ¯ℓ0 and therefore preserves the ordering of two rollouts from the same prompt when they share the same initial state. However, raw differences mea...

    2023

  34. [40]

    influxdata company name change from early 2010s to later name

    browser.search(query="influxdata company name change from early 2010s to later name") Turn credit: δ5 = +5.44. After four stalled probes, this reformulated query returns “Errplane became InfluxData in 2015,” localizing nearly all progress in the eight-turn trajectory to this rescue query. A.5.2 NEGATIVETRAJECTORIES: LOCALIZING THETOOLCALLTHATLOSES Every f...

    2015

  35. [2018]

    Tan, X.-W

    15 TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents H.-Z. Tan, X.-W. Yang, H. Chen, J.-J. Shao, Y . Wen, Y . Shen, W. Luo, X. Du, L.-Z. Guo, and Y .-F. Li. Hindsight credit assignment for long-horizon LLM agents.arXiv preprint arXiv:2603.08754,

    arXiv

  36. [2022]

    G. Wang, S. Dai, G. Ye, Z. Gan, W. Yao, Y . Deng, X. Wu, and Z. Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn search agents. InInternational Conference on Learning Representations, 2026a. H. Wang, Z. Hou, Y . Wei, J. Tang, and Y . Dong. SWE-Dev: Building software engineering agents with training and infer...

    Pith/arXiv arXiv 2025

  37. [2023]

    G. Dong, Y . Chen, X. Li, J. Jin, H. Qian, Y . Zhu, H. Mao, G. Zhou, Z. Dou, and J.-R. Wen. Tool- Star: Empowering LLM-brained multi-tool reasoner via reinforcement learning.arXiv preprint arXiv:2505.16410, 2025a. G. Dong, H. Mao, K. Ma, L. Bao, Y . Chen, et al. Agentic reinforced policy optimization.arXiv preprint arXiv:2507.19849, 2025b. A. Drouin, M. G...

    Pith/arXiv arXiv

  38. [2024]

    doi: 10.1145/3643757. I. Bouzenia, P. Devanbu, and M. Pradel. RepairAgent: An autonomous, LLM-based agent for program repair. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering, pages 2188–2200,

    doi:10.1145/3643757

  39. [2025]

    doi: 10.1109/ICSE55347.2025.00157. J. Chai, G. Yin, Z. Xu, C. Yue, Y . Jia, S. Xia, X. Wang, J. Jiang, X. Li, C. Dong, H. He, and W. Lin. RLFactory: A plug-and-play reinforcement learning post-training framework for LLM multi-turn tool-use.arXiv preprint arXiv:2509.06980,

    arXiv 2025

  40. [2026]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Pith/arXiv arXiv