Pith. sign in

REVIEW 2 major objections 6 minor 58 references

Expert trajectories alone can train LLM agents to prefer correct actions over their own mistakes at each state, without online environment rollouts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:32 UTC pith:JBM4CAXL

load-bearing objection Clean offline agent recipe: same-state one-step student negatives + PPA, multi-benchmark evidence that can match GRPO cost-wise without env rollouts. the 2 major comments →

arxiv 2607.10601 v1 pith:JBM4CAXL submitted 2026-07-12 cs.AI

Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

Yixiong Chen , Alan Yuille This is my paper
classification cs.AI
keywords LLM agentspreference optimizationDPOoffline policy optimizationexpert trajectoriestool usesupervised fine-tuningPolicy-Preserving Augmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agents are usually trained by imitating expert trajectories as ordinary text. That only teaches what the expert did, not how to choose the right action when the model would otherwise make a mistake. This paper argues that the same expert logs can be turned into state-level preferences: at each expert step, sample a one-step action from the current student, treat a plausible wrong action as the negative, and contrast it with the expert action under a DPO-style loss. Two stabilizers keep the signal on the latent decision rather than on surface formatting—an SFT anchor plus Policy-Preserving Augmentation that re-renders the same decision under multiple schemas and recovery contexts. Across tool use, long-horizon retail interaction, and web GUI grounding, the method improves agents beyond imitation and, for a 9B model on τ-bench retail, reaches 41.4% success versus 21.7% for SFT and 40.0% for online GRPO, without environment interaction during gradient steps.

Core claim

Expert trajectories can support low-cost agentic policy optimization when each expert action is treated as the preferred choice at its state and contrasted with a one-step student-sampled negative under a DPO-style objective, provided the contrast is stabilized so learning targets the latent decision rather than schema-specific rendering.

What carries the argument

Agentic-DPO: at each expert state it samples one-step student candidates, keeps the hardest different latent action as the negative, and optimizes a length-scaled DPO loss against the expert action, with an SFT anchor and Policy-Preserving Augmentation that re-renders the same latent pair under multiple schemas and recovery contexts.

Load-bearing premise

The important decision states are already present in the expert trajectories, so one-step student mistakes sampled only at those states are enough and the method need not discover new long-horizon states the student would reach on its own.

What would settle it

On a held-out task where success requires visiting states far outside the expert support, train Agentic-DPO and online GRPO on the same backbone and expert pool; if Agentic-DPO then lags online RL by a large margin while still matching it on expert-covered tasks, the offline-coverage premise fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents can be improved beyond pure imitation without reward models or full multi-turn environment rollouts during training.
  • On long-horizon tool–user tasks, step-level offline student negatives can reach success rates comparable to online grouped RL under the same backbone.
  • Rendering the same latent trajectory under multiple schemas reduces schema-dominated preference gradients and shrinks the canonical–perturbation accuracy gap.
  • Smaller models benefit from more negative-refresh rounds; larger models often peak after a single refresh.
  • Per-step training cost stays near SFT (about 1.6×) rather than online RL (about 13.6× in the reported τ-bench setup).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • When expert coverage is dense, offline one-step hard negatives may recover much of the self-mistake signal usually sought from online RL.
  • The same convert-demos-to-state-preferences recipe is a natural fit for other sequential settings where full rollouts are expensive but expert logs are plentiful.
  • Failure modes should concentrate on states experts never visit, so hybrid pipelines that add light online exploration only for under-covered states are a direct next test.
  • Schema-preserving multi-view rendering may stabilize any preference method that scores rendered action strings rather than latent actions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes Agentic-DPO, an offline method that converts expert agent trajectories into state-conditioned preference pairs: at each expert state it samples one-step actions from the current student, filters expert-equivalent candidates, and contrasts the expert action with a hard student negative under a length-scaled DPO objective. Training is stabilized by an SFT warm-up/anchor and Policy-Preserving Augmentation (PPA), which re-renders the same latent decision under multiple schemas and recovery-style histories. The method requires no environment execution during gradient steps, no reward model, and no full-trajectory student exploration. Experiments on StableToolBench, τ-bench retail, and Mind2Web across Qwen3.5-2B/4B/9B (plus a Gemma ablation) show consistent gains over SFT and strong preference/RL baselines; on τ-bench retail a 9B model rises from 21.7% (SFT) to 41.4%, matching GRPO (40.0%) at substantially lower per-step cost. Code is released.

Significance. If the results hold, the work offers a practical middle ground between pure imitation and online agent RL: expert trajectories alone can supply a useful self-mistake signal when preferences are constructed at the action level and schema effects are controlled. Strengths that support this assessment include multi-benchmark, multi-scale evaluation with three seeds and reported variance; ablations isolating SFT warm-up, the SFT anchor, and both PPA families (Table 2); held-out perturbation and OOD BFCL-v3 checks (Tables 3–4); K/data scaling and refresh-round dynamics (Figs. 2–3); a direct wall-time cost comparison versus GRPO (§5); and a short local logit-space analysis of the anchor and hard-negative rule (Appendix B) that is appropriately scoped. The offline coverage limit is stated explicitly rather than oversold. The contribution is incremental relative to DPO/Step-DPO/ETO-style preference work, but the agent-specific construction (one-step student negatives at expert states + PPA) and the empirical cost–performance tradeoff are useful for the field.

major comments (2)
  1. §4.1 states a single R=5 negative-refresh schedule for all backbones, yet Fig. 3 shows clear scale dependence: the 2B model peaks near R=3 and then declines, while the 9B model is essentially best after R=1. The main Table 1 numbers are therefore not necessarily the best Agentic-DPO configuration per scale. Either report the R-selected (or early-stopped) numbers used for the primary comparison, or justify why a fixed R=5 is the fair protocol against GRPO/ETO and show that the ranking is unchanged under per-scale R.
  2. Table 1’s “Average” column aggregates StableToolBench, τ-bench, and Mind2Web, but GRPO has no Mind2Web entry (explained by missing step-level rewards). Averaging methods with unequal benchmark coverage can overstate relative standing. Prefer reporting per-benchmark winners only, or define Average over the intersection of methods that run on all three tasks, and keep the GRPO comparison as a separate two-benchmark cost–accuracy statement (§5).
minor comments (6)
  1. §3.2 / Eq. (1): β_eff uses max(|a+|,|a−|)^α with α=0.5 fixed; a one-sentence intuition for α=0.5 (vs 0 or 1) would help readers who must reimplement without a sweep.
  2. Table 5 lists identical training-pool sizes (3,786) for StableToolBench and τ-bench retail; a brief note that this is coincidental (or not) would avoid a false impression of shared data.
  3. Figure 1 caption and §1: “student” is introduced late relative to its first use in the figure; define student = current policy being optimized at first mention.
  4. Appendix D recovery-context example is clear; a short note on how often recovery-context vs action-rendering views are sampled in practice (uniform over Φ) would make the PPA mixture fully reproducible from the main text alone.
  5. Typo/style: abstract and §1 use both “tau-bench” and “τ-bench”; standardize on τ-bench throughout.
  6. §4.4 Table 4 (BFCL-v3): report whether decoding/tool-calling hyperparameters match the ToolACE training setup, so the SFT regression on Live is interpretable.

Circularity Check

0 steps flagged

No significant circularity: empirical agent-training method with external benchmarks and non-circular local gradient analysis.

full rationale

Agentic-DPO is an offline preference-optimization recipe for LLM agents. Its central claim—that expert trajectories can be converted into state-conditioned action preferences (expert action vs one-step student negative under a DPO-style loss, stabilized by an SFT anchor and PPA) and thereby improve agents beyond SFT—is evaluated on external held-out metrics (StableToolBench canonical/perturbed accuracy, τ-bench task success, Mind2Web step success, BFCL-v3 OOD). Success is not redefined by the training objective. The short theory (Prop. 1 / Cor. 2 / Prop. 3 in §3.4 and App. B) only shows that at initialization the SFT and contrastive gradients are collinear in logit space and that a small anchored step decreases expert KL; that is a local consequence of the softmax/DPO algebra under stated assumptions, not a fit renamed as a prediction and not a self-citation uniqueness chain. No load-bearing self-citation, no uniqueness theorem imported from the authors, no ansatz smuggled via prior work by the same authors, and no renaming of a known empirical law as a derived result. The paper’s own offline-coverage limit (§5) is a scope boundary, not circularity. Honest finding: score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard preference-optimization math, the assumption that expert traces cover the states that matter, and several hand-chosen training coefficients. No new physical entities are postulated; the invented constructs are methodological (Agentic-DPO preference construction and PPA views). Free parameters are the usual ML hyperparameters that control preference strength, anchoring, negative sampling, and refresh rounds.

free parameters (5)
  • DPO temperature β
    Fixed at 0.008 across all main runs; scales the preference margin and, with length normalization, the relative strength of the contrastive term versus the SFT anchor.
  • SFT anchor coefficient λ
    Fixed at 0.5; controls how strongly expert likelihood is preserved while suppressing student negatives.
  • Length-scaling exponent α
    Set to 0.5 in β_eff = β / max(|a+|,|a-|)^α to prevent long actions from dominating short ones.
  • Negatives per state K
    Default K=4 student samples; scaling curve shows gains saturate around K=2–4.
  • Negative-refresh rounds R
    Default R=5; best R depends on model scale (1 for 9B, more for 2B), so the reported peaks partly reflect this schedule choice.
axioms (5)
  • domain assumption DPO-style Bradley–Terry preference model on rendered action strings is a valid surrogate for improving agent decision quality.
    Inherited from DPO literature and used as the training objective in Eq. (1) without a learned reward model.
  • domain assumption Expert trajectories provide the preferred latent action at each observed state, and latent-action equivalence after parsing/normalization is well-defined.
    Required to build (s, u+, u−) triples and to drop expert-equivalent candidates in Algorithm 1.
  • ad hoc to paper One-step student samples at expert states, without environment execution, yield informative hard negatives for multi-turn agent policies.
    Core methodological bet of Agentic-DPO; contrasted with full-trajectory RL/ETO in §2 and limited in §5.
  • standard math Softmax logit-space abstraction of latent actions is adequate for local gradient alignment analysis.
    Appendix B analyzes SFT/DPO collinearity under finite action sets and softmax policies.
  • ad hoc to paper Policy-preserving schema rewrites leave the latent decision fixed while changing only surface form/history.
    Assumption underlying PPA families (action-rendering and recovery-context) in §3.3 and Appendix D.
invented entities (2)
  • Agentic-DPO preference construction no independent evidence
    purpose: Turn each expert step into a state-conditioned preferred/rejected action pair using one-step student hard negatives.
    Methodological construct, not a physical entity; evaluated only through downstream agent metrics.
  • Policy-Preserving Augmentation (PPA) no independent evidence
    purpose: Render the same latent trajectory under multiple schemas so preference gradients track decisions rather than formatting.
    Introduced to stabilize action-level DPO; supported by ablations and held-out perturbation operators, but defined by the paper’s rewrite families.

pith-pipeline@v1.1.0-grok45 · 24548 in / 3628 out tokens · 35695 ms · 2026-07-14T10:32:48.577915+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state. Existing methods to mitigate this problem include preference learning or reinforcement learning, but they usually need high-cost environment rollouts and reward models. We propose Agentic-DPO, a lightweight offline agent policy optimization method that turns expert trajectories into state-conditioned preference supervision. At each expert action state, Agentic-DPO samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective. To avoid mixing both policy and schema in preference learning, we introduce Policy-Preserving Augmentation (PPA), which renders the same latent trajectory under multiple schemas while keeping the expert policy fixed. Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration. We conduct experiments across StableToolBench, tau-bench retail, and Mind2Web, where Agentic-DPO consistently improves agents at different model scales beyond imitation. In particular, it raises tau-bench accuracy from 21.7% (SFT) to 41.4% for a 9B model, matching online GRPO under the same backbone with only step-level rollouts and without environment interaction during gradient steps. The results suggest that expert trajectories can support low-cost agentic policy optimization when converted from demonstrations into state-level action preferences. Code for Agentic-DPO is released at https://github.com/Schuture/Agentic-DPO.

Figures

Figures reproduced from arXiv: 2607.10601 by Alan Yuille, Yixiong Chen.

Figure 1
Figure 1. Figure 1: Illustration of the difference between SFT and Agentic-DPO for agent training. SFT treats [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scaling behavior of Agentic-DPO on StableToolBench with Qwen3.5-2B. Shaded bands [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of negative-refresh rounds R on StableToolBench for Qwen3.5-2B/4B/9B. Lines show per-round medians over seeds, and shaded bands show min–max. R = 0 is PPA+SFT only. Smaller models benefit from more refreshes, while the 9B model reaches its best performance after one refresh round. 5 Discussion Training cost. Agentic-DPO is more expensive than SFT, but much cheaper than online RL. On τ -bench with Qw… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 30 linked inside Pith

  1. [1]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InThe twelfth international conference on learning representations, 2024

    2024

  2. [2]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024

    2024

  3. [3]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461–12495, 2024

    2024

  4. [4]

    Fireact: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915, 2023

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Agent-FLAN: Designing data and methods of effective agent tuning for large language models.arXiv preprint arXiv:2403.12881, 2024

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-FLAN: Designing data and methods of effective agent tuning for large language models.arXiv preprint arXiv:2403.12881, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    ATLaS: Agent tuning via learning critical steps.arXiv preprint arXiv:2503.02197, 2025

    Zhixun Chen, Ming Li, Yuxuan Huang, Yali Du, Meng Fang, and Tianyi Zhou. ATLaS: Agent tuning via learning critical steps.arXiv preprint arXiv:2503.02197, 2025. Findings of ACL 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. InICML, 2024

    2024

  8. [8]

    Rethinking DPO: The role of rejected responses in preference misalignment.arXiv preprint arXiv:2506.12725, 2025

    Jay Hyeon Cho, JunHyeok Oh, Myunsoo Kim, and Byung-Jun Lee. Rethinking DPO: The role of rejected responses in preference misalignment.arXiv preprint arXiv:2506.12725, 2025. Proposes Bounded-DPO (BDPO)

    Pith/arXiv arXiv 2025

  9. [9]

    Agent-rlvr: Training software engineering agents via guidance and environment rewards.arXiv preprint arXiv:2506.11425, 2025

    Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards.arXiv preprint arXiv:2506.11425, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Mind2Web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. InAdvances in Neural Information Processing Systems, 2023

    2023

  11. [11]

    KTO: Model alignment as prospect theoretic optimization.International Conference on Machine Learning (ICML), 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization.International Conference on Machine Learning (ICML), 2024. arXiv:2402.01306

    Pith/arXiv arXiv 2024

  12. [12]

    AgentRefine: Enhancing agent generalization through refinement tuning.arXiv preprint arXiv:2501.01702, 2025

    Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. AgentRefine: Enhancing agent generalization through refinement tuning.arXiv preprint arXiv:2501.01702, 2025. ICLR 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Solving the granularity mismatch: Hierarchical preference learning for long-horizon llm agents

    Heyang Gao, Zexu Sun, Erxue Min, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Xu Chen. Solving the granularity mismatch: Hierarchical preference learning for long-horizon llm agents. arXiv preprint arXiv:2510.03253, 2025

    arXiv 2025

  14. [14]

    Gemma 3 technical report

    Gemma Team. Gemma 3 technical report. Technical report, Google DeepMind, 2025. arXiv:2503.19786

    Pith/arXiv arXiv 2025

  15. [15]

    StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models. InFindings of ACL, 2024. arXiv:2403.07714

    Pith/arXiv arXiv 2024

  16. [16]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. 10

    Pith/arXiv arXiv 2024

  17. [17]

    DiaTool-DPO: Multi-turn direct preference optimization for tool-augmented large language models.Annual Meeting of the Association for Computational Linguistics (ACL), 2025

    Sunghee Jung, Donghun Park, Shinbok Jeon, Juyoung Lee, Eunhwa Yang, and Pilsung Kang. DiaTool-DPO: Multi-turn direct preference optimization for tool-augmented large language models.Annual Meeting of the Association for Computational Linguistics (ACL), 2025. arXiv:2504.02882

    Pith/arXiv arXiv 2025

  18. [18]

    Step- dpo: Step-wise preference optimization for long-chain reasoning of llms.arXiv preprint arXiv:2406.18629, 2024

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step- dpo: Step-wise preference optimization for long-chain reasoning of llms.arXiv preprint arXiv:2406.18629, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    RLAIF vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with ai feedback.International Conference on Machine Learning (ICML), 2024. arXiv:2309.00267

    Pith/arXiv arXiv 2024

  20. [20]

    Hammer: Robust function- calling for on-device language models via function masking.arXiv preprint arXiv:2410.04587, 2024

    Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, and Weinan Zhang. Hammer: Robust function- calling for on-device language models via function masking.arXiv preprint arXiv:2410.04587, 2024

    arXiv 2024

  21. [21]

    ToolACE: Winning the points of LLM function calling

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. InICLR, 2025. arXiv:2409.00920

    Pith/arXiv arXiv 2025

  22. [22]

    Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  23. [23]

    Gui agents: A survey

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. InFindings of the Association for Computational Linguistics: ACL 2025, pages 22522–22538, 2025

    2025

  24. [24]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- bac...

    Pith/arXiv arXiv 2022

  25. [25]

    Smaug: Fixing failure modes of preference optimisation with DPO-positive.arXiv preprint arXiv:2402.13228, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-positive.arXiv preprint arXiv:2402.13228, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. InICML, 2025

    2025

  27. [27]

    WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning. InICLR, 2025. arXiv:2411.02337

    Pith/arXiv arXiv 2025

  28. [28]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. InICLR, 2024. Spotlight. arXiv:2307.16789

    Pith/arXiv arXiv 2024

  29. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  30. [30]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. 11

    Pith/arXiv arXiv 2017

  31. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    ScribeAgent: Towards specialized web agents using production-scale workflow data.arXiv preprint arXiv:2411.15004, 2024

    Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. ScribeAgent: Towards specialized web agents using production-scale workflow data.arXiv preprint arXiv:2411.15004, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Direct multi-turn preference optimization for language agents

    Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. Direct multi-turn preference optimization for language agents. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2024. arXiv:2406.14868

    Pith/arXiv arXiv 2024

  34. [34]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    2023

  35. [35]

    ALFWorld: Aligning text and embodied environments for inter- active learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for inter- active learning. InInternational Conference on Learning Representations (ICLR), 2021. arXiv:2010.03768

    Pith/arXiv arXiv 2021

  36. [36]

    Trial and error: Exploration-based trajectory optimization for LLM agents.arXiv preprint arXiv:2403.02502, 2024

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for LLM agents.arXiv preprint arXiv:2403.02502, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving.arXiv preprint arXiv:2601.01426, 2026

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, et al. Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving.arXiv preprint arXiv:2601.01426, 2026

    arXiv 2026

  38. [38]

    Triplets better than pairs: Towards stable and effective self-play fine-tuning for LLMs

    Yibo Wang, Hai-Long Sun, Guangda Huzhang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Triplets better than pairs: Towards stable and effective self-play fine-tuning for LLMs. InNeurIPS, 2025

    2025

  39. [39]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning

    Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7920–7939, 2025

    2025

  41. [41]

    On the generalization of SFT: A reinforcement learning perspective with reward rectification.arXiv preprint arXiv:2508.05629, 2025

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of SFT: A reinforcement learning perspective with reward rectification.arXiv preprint arXiv:2508.05629, 2025. ICLR

    arXiv 2025

  42. [42]

    Referred to in text as DFT (Dynamic Fine-Tuning)

  43. [43]

    Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755, 2025

    Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, et al. Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning.arXiv preprint arXiv:2509.08755, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    WebShop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2207.01206

    Pith/arXiv arXiv 2022

  46. [46]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023. 12

    2023

  47. [47]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Pivotrl: High accuracy agentic post-training at low compute cost.arXiv preprint arXiv:2603.21383, 2026

    Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta, Oleksii Kuchaiev, Jiantao Jiao, et al. Pivotrl: High accuracy agentic post-training at low compute cost.arXiv preprint arXiv:2603.21383, 2026

    arXiv 2026

  49. [49]

    Self-rewarding language models.International Conference on Machine Learning (ICML), 2024

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.International Conference on Machine Learning (ICML), 2024. arXiv:2401.10020

    Pith/arXiv arXiv 2024

  50. [50]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. InFindings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024

    2024

  51. [51]

    ToolACE-R: Model-aware iterative training and adaptive refinement for tool learning.arXiv preprint arXiv:2504.01400, 2025

    Xingshan Zeng, Weiwen Liu, Xu Huang, Zezhong Wang, Lingzhi Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruiming Tang, and Qun Liu. ToolACE-R: Model-aware iterative training and adaptive refinement for tool learning.arXiv preprint arXiv:2504.01400, 2025

    arXiv 2025

  52. [52]

    first_name

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. SeeAct: Gpt-4v(ision) is a generalist web agent, if grounded. InInternational Conference on Machine Learning (ICML), 2024. A Expert Trajectory Sources Each benchmark’s expert traces and the resulting Agentic-DPO training pool are summarized in Table 5. Table 5: Expert trajectory sources and the res...

    2024

  53. [55]

    user_id":

    assistant: Action: get_order Action Input: {"user_id": "123"} After applying recovery-context PPA at index 4, the trajectory becomes:

  54. [56]

    name": "Alice

    assistant: Action: find_user Action Input: {"name": "Alice"}

  55. [57]

    user_id":

    user: Observation: {"user_id": "123"}

  56. [58]

    user_id":

    assistant: Thought: Let me try using get_product... [INJECTED] Action: get_product Action Input: {"user_id": "123"}

  57. [59]

    error":

    user: Observation: {"error": "Tool returned no results"} [INJECTED]

  58. [60]

    user_id":

    assistant: Thought: The previous tool call failed. Let me try a different approach. [RECOVERY] Action: get_order Action Input: {"user_id": "123"} Indices 4 and 5 are the injected mistake and its observation; index 6 is the original expert action with a prepended recovery thought, now serving as the expert positive at this step. The latent decision (callge...