Pith. sign in

REVIEW 5 major objections 5 minor 53 references

WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rule-based RL without reasoning annotations lets a 14B open-source web agent beat gpt-4o on WorkArena.

desk verdict A solid, reproducible recipe paper on R1-style RL for web agents, with an honest central result and fixable framing issues. read the letter →

arxiv 2505.22942 v2 pith:BRBLUY7I submitted 2025-05-28 cs.CL cs.AI

Yuchen Zhuang , Di Jin , Jiaao Chen , Wenqi Shi , Hanrui Wang , Chao Zhang This is my paper
classification cs.CLcs.AI
keywords webagentsreinforcementlearningrule-basedrewardGRPOWorkArenaLLMreasoningsingle-stepplanninghacking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a simple, rule-based reinforcement-learning recipe can teach open-source language-model web agents to reason their way through workplace web tasks without any human-written reasoning traces. It converts multi-turn web navigation into single-step decisions and rewards only format compliance plus exact action matches, letting intermediate reasoning emerge as a by-product. On WorkArena the trained 14B model reaches 46.79% success, outperforming the SFT baseline by 16.59 points and the gpt-4o reference by about 4 points. If this holds, it shifts the cost of building capable web agents from expensive proprietary APIs and expert demonstrations to a training signal that can be generated automatically.

What carries the argument

The load-bearing mechanism is a trajectory decomposition: every multi-step BrowserGym/WorkArena trajectory is cut at each action into an independent prompt containing the goal, current HTML/AXTree observation, and action history, with the next ground-truth action as target. On top of that sits a thinking template that forces responses into '<think>...</think>' and '<action>...</action>' blocks, and a progressive reward $R = R_f + R_s + R_p$, where $R_f = 0.1$ rewards valid format, $R_s = 1$ rewards exact action type and parameters, $0.1$ rewards type-only matches, and $R_p = -0.9$ penalizes stray tokens after the closing action tag. The policy is updated with GRPO, a group-relative policy-gradient method that scores each sampled action against other samples in the same group, with clipping and KL regularization. An SFT warm-up on the same oracle trajectories supplies the starting policy, and the sparse, discrete reward structure is credited with preventing the reward hacking that dense similarity rewards induce.

What would settle it

Run the same trained checkpoints on enterprise-style web tasks built on a different platform than ServiceNow, with human-verified optimal actions instead of heuristic oracles; if the RL gains over the SFT baseline disappear or the '<think>' traces stop tracking the true next step, the claim that rule-based RL alone produces transferable web reasoning fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that decomposing a multi-turn web navigation trajectory into independent single-step decisions makes the R1-style rule-based reinforcement-learning recipe work for interactive web agents. Trained first by behavior cloning on 1,000 heuristic-generated trajectories, then optimized with GRPO under a reward that adds a 0.1 format bonus, 1.0 for an exact action-type-plus-parameter match, 0.1 for a type-only match, and a -0.9 penalty for tokens after the closing action tag, the 3B/7B/14B Qwen checkpoints reach 36.85/39.56/46.79 success on WorkArena, versus 26.59/27.32/30.20 for their SFT-only counterparts and 42.65 for the gpt-4o reference. The resulting traces contain long, problem-specific '<think>' passages even though no such reasoning was ever supervised, and the sparse reward scheme is what keeps training stable: dense similarity-based rewards collapse responses into repeated click hacking. The same recipe applied to Llama-3.1-8B adds 23.84 points over its base model.

Load-bearing premise

The load-bearing premise is that BrowserGym's automatically generated cheat() actions are correct, complete, and representative enough that exact matching to them is a valid reward for good web navigation; if those heuristics encode ServiceNow-specific or otherwise biased behavior, the learned reasoning inherits it.

Editorial extensions

If this is right

  • If the WorkArena numbers transfer, a 14B open-weight model can carry knowledge-work automation that today is often delegated to gpt-4o-class APIs, with cost and privacy advantages.
  • Explicit reasoning annotations are unnecessary: a format reward plus an exact-action reward is enough for the model to emit task-relevant '<think>' steps, removing the main data bottleneck of reasoning-distillation approaches.
  • Sparse discrete rewards are the safer design choice for interactive RL; dense similarity rewards caused reward hacking and response collapse in the ablation, so future web-agent RL should avoid them.
  • The single-step formulation means training does not need a learned long-horizon reward model or an outcome-supervised reward model, simplifying deployment when direct environmental feedback is expensive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reward's reliance on exact matching to one oracle policy limits what can be learned; any valid action the heuristic never produces is unreachable by the reward, so the method's ceiling is partly the oracle's coverage.
  • Editorial inference: the gains are demonstrated inside the WorkArena/ServiceNow environment, so the strongest generalization test is applying the same recipe to a different enterprise UI with an independently generated oracle; the paper itself notes its focus is workplace navigation.
  • Editorial inference: because dense rewards collapse into repeated clicks, this work suggests that coarse, environment-verifiable rewards are preferable in interactive settings; a natural extension is to replace action exact-match with backend task validation to test whether exact matching or task outcome is the better training target.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes WorkForceAgent-R1, a rule-based R1-style reinforcement learning framework for training LLM web agents, applied to the WorkArena benchmark. The method uses an SFT warm-up followed by GRPO, with a composite reward that scores format adherence, exact matching of the predicted action to a ground-truth action, and a penalty for extra tokens. The authors report that WorkForceAgent-R1 (3B/7B/14B) outperforms SFT baselines by 10.26-16.59 percentage points on WorkArena and that the 14B variant reaches an overall success rate of 46.79%, above the gpt-4o reference of 42.65%. They further claim that the method induces reasoning implicitly, without explicit reasoning annotations or extensive expert demonstrations.

Significance. If the claims are supported, the work is a useful contribution: it provides an open-source recipe for post-training open-weights web agents that improves over SFT and reaches proprietary-level performance on a standard benchmark. The paper includes ablations (GRPO vs PPO, reward granularity, warm-up effect), applies to multiple backbone sizes and families, and links to code. The central idea of using a rule-based reward for single-step action selection is interesting and potentially transferable. However, the empirical evidence is currently limited by the absence of error bars, the lack of comparisons with existing RL web-agent baselines, and the reliance on an oracle-action-matching reward, so the strength of the claims is not yet fully established.

major comments (5)
  1. [§4.3.3, §4.1, Abstract] The success reward component R_s is an exact match to BrowserGym's heuristic cheat()-generated action (type and parameters). The paper calls this 'an effective surrogate for real-world verification,' but provides no evidence that exact matching to this single oracle action sequence correlates with eventual task success. Because the SFT warm-up uses the same oracle labels, both training stages entrench the heuristic's biases; any valid alternative action that differs in element id or ordering receives zero reward. The authors should either reward with the task-level validation functions that Appendix A states are available, or empirically validate the surrogate by showing that oracle-action matching predicts task success and that the learned policy's actions agree with alternative valid actions.
  2. [Abstract, §4.1] The claim of training 'without ... extensive expert demonstrations' is contradicted by the paper's own Appendix A, which states that the Playwright oracle scripts 'supply ground-truth solutions for training and benchmarking purposes.' These are expert demonstrations, albeit generated by heuristics. The wording should be corrected or qualified; otherwise the headline contribution is misrepresented.
  3. [§5.1, Table 1] The main results are reported from a single run with no error bars, multiple seeds, or statistical tests. Given the rollout temperature of 0.6 and small per-task sample sizes (e.g., 4 tasks in Dashboards, 25.17% on Filter for the 14B model), the reported 10.26-16.59 percentage point improvements over SFT may be sensitive to run-to-run variance. At least three independent seeds with mean and standard deviation (or a significance test) should be reported for the central comparisons.
  4. [§5.1, Related Work] The paper discusses WebRL and OpenWebVoyager as prior RL web-agent methods but does not compare against them or report their published WorkArena numbers. Without these baselines, the claim that WorkForceAgent-R1 achieves superior performance among OSS RL web agents is not established. Please add these baselines to Table 1 or explain why a direct comparison is infeasible.
  5. [§4.3.1, §5.3] The origin of the reasoning tokens in the SFT warm-up is unspecified. Section 4.1 describes only cheat()-generated action trajectories, yet Section 4.3.1 says the SFT loss is computed on 'ground-truth action with reasoning steps,' and Figure 5's caption refers to 'o3-mini-annotated trajectories from BrowserGym.' If an external LLM generated the <think> tokens used in the warm-up, the claim of learning 'without explicit annotations' is misleading. The data-generation protocol for the reasoning steps must be clarified, including whether the warm-up and RL stages use the same underlying trajectories.
minor comments (5)
  1. [Figure 1] The caption text 'Reasons1-2 Sentences (< 100 Tokens)Problem Analysis(~100 Tokens)ObsAnalysis(>100 Tokens)Action Analysis(>100 Tokens)' is missing spaces between phrases; please fix the formatting.
  2. [Table 1] The header 'Base to Large Size Open-Source (OSS) LLMs' appears to contain a typo; 'Base' is likely intended as 'Base to Large' or simply 'Open-Source LLMs'.
  3. [§4.3.3] The penalty reward R_p is defined as '-0.9 if tokens appears after <action>,' but the surrounding text says the penalty applies to tokens after the termination tag </action>. Please correct the formula to refer to </action>.
  4. [Figure 5] The text says SFT-L models use deepseek-ai/DeepSeek-R1-Distill-Llama-70B to inject long-chain reasoning, but the figure caption only says 'SFT-L denotes long reasoning models.' Please clarify how the distillation was applied to the 3B/7B/14B backbones and whether the SFT baselines in Table 1 also use the same long-reasoning data.
  5. [§5.2] The statement that the 14B variant 'surpasses the proprietary state-of-the-art model GPT-4o by 4.99%' is based on a single evaluation without variance; please rephrase as 'outperforms the gpt-4o reference in this evaluation' to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL reward is a training surrogate, while the reported results are independent task-level evaluations on WorkArena.

full rationale

WorkForceAgent-R1 is an empirical RL pipeline, not a derivation whose conclusion is equivalent to its inputs. The success reward R_s in Section 4.3.3 uses exact matching to BrowserGym's ground-truth actions, but this is a training-time surrogate. The paper's headline results in Table 1 are success rates computed by WorkArena's own validation functions, which check actual task outcomes (e.g., backend database queries for forms, list ordering, catalog order contents), not by exact matching to the oracle trajectory. The validation accuracy in Figure 4(c) is an exact-match diagnostic, but it is not presented as the main benchmark result. The claim that RL improves over SFT baselines is an empirical comparison with a shared environment, not a statement forced by definition. The use of BrowserGym oracle trajectories for both SFT warm-up and RL training is a data-generation choice that may limit generalization claims, but it does not make the reported task-level success rates true by construction. No load-bearing self-citation chain or imported uniqueness theorem is used to justify the method. Consequently, no circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on oracle action supervision and a per-step exact-match reward; no new entities are introduced. The free parameters are hand-chosen reward constants and standard training hyperparameters.

free parameters (6)
  • Format reward Rf = 0.1
    Hand-chosen constant in Section 4.3.3; the only incentive for output format adherence.
  • Success reward Rs = 1.0 (full match), 0.1 (action type only)
    Hand-chosen in Section 4.3.3; makes RL an imitation objective against oracle actions.
  • Penalty reward Rp = -0.9
    Hand-chosen in Section 4.3.3; discourages tokens after </action>.
  • KL regularization coefficient beta = 1e-3
    Fixed in Section 5.1; controls deviation from reference policy.
  • SFT warm-up sample size = 1,000
    Section 5.1; behavior-cloning subset before RL; chosen by authors.
  • Rollout temperature = 0.6
    Section 5.1; sampling temperature during GRPO group generation.
assumptions (5)
  • domain assumption BrowserGym cheat() trajectories are correct, representative ground-truth actions for WorkArena tasks
    Used as SFT labels and as the exact-match target in Rs (Sections 4.1, 4.3.3). The oracle is heuristic and not validated against human performance.
  • domain assumption Exact matching to a single oracle action is a valid surrogate for task-level success feedback
    The RL reward is per-step action match, while the POMDP formalization defines R as binary task completion (Sections 3 and 4.3.3). This proxy is never validated.
  • ad hoc to paper Single-step decomposition of multi-step trajectories preserves enough credit-assignment signal for web agents
    Section 4.1 reformulates trajectories into independent single-step decisions; the paper argues multi-step planning is infeasible due to dynamic HTML but provides no comparison against multi-step RL.
  • standard math GRPO and KL regularization behave as in prior work
    Section 4.3.2 adopts GRPO and assumes its convergence and stability properties without re-derivation.
  • domain assumption WorkArena benchmark success rate is a meaningful proxy for workplace web-navigation performance
    All evaluation is on WorkArena (Section 5.1); no out-of-benchmark generalization test is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning." pith.science (2026). https://pith.science/paper/BRBLUY7I

@misc{pith2026250522942,
  author       = {Pith},
  title        = {Pith review of: WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BRBLUY7I}},
  note         = {Machine review of arXiv:2505.22942}
}
read the original abstract

Large language models (LLMs)-empowered web agents enables automating complex, real-time web navigation tasks in enterprise environments. However, existing web agents relying on supervised fine-tuning (SFT) often struggle with generalization and robustness due to insufficient reasoning capabilities when handling the inherently dynamic nature of web interactions. In this study, we introduce WorkForceAgent-R1, an LLM-based web agent trained using a rule-based R1-style reinforcement learning framework designed explicitly to enhance single-step reasoning and planning for business-oriented web navigation tasks. We employ a structured reward function that evaluates both adherence to output formats and correctness of actions, enabling WorkForceAgent-R1 to implicitly learn robust intermediate reasoning without explicit annotations or extensive expert demonstrations. Extensive experiments on the WorkArena benchmark demonstrate that WorkForceAgent-R1 substantially outperforms SFT baselines by 10.26-16.59%, achieving competitive performance relative to proprietary LLM-based agents (gpt-4o) in workplace-oriented web navigation tasks.

Figures

Figures reproduced from arXiv: 2505.22942 by the authors.

Figure 1
Figure 1. Example of reasoning capabilities that support [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of WorkForceAgent-R1. task in WorkArena is defined by a configuration file specifying target states, relevant webpage el￾ements, and expected outcomes. To ensure data integrity and prevent information leakage, we al￾locate separate training and testing data configura￾tions. Specifically, we reserve 10 distinct configu￾ration files per task, totaling 330 training configura￾tions. We further mitigate potentia… view at source ↗
Figure 3
Figure 3. Example of a thinking template with an agent [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Learning curves across training steps. formance via RL: WorkForceAgent-R1 consis￾tently outperforms existing OSS models by an av￾erage margin of 29.13% when employing similar model sizes. Remarkably, the 14B-parameter vari￾ant of WorkForceAgent-R1 surpasses the propri￾…
Figure 5
Figure 5. Figure 5: Performance comparison among different training strategies. “-L” denotes long reasoning models. 35 36 37 38 39 40 Success Rate (%) 39.56 36.08 GRPO PPO (a) Performance 0 25 50 75 100 125 150 Training Steps 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Reward PPO GRPO (b) Average Reward …
Figure 6
Figure 6. Figure 6: Comparison between PPO and GRPO with Qwen2.5-7B-Instruct as backbone LLM. Qwen2.5-Instruct models, while SFT models use the o3-mini-annotated trajectories from BrowserGym. For SFT-L models, we employ the deepseek-ai/DeepSeek-R1-Distill-Llama-70B to inject long-chain re…
Figure 7
Figure 7. Figure 7: Comparison between warmup model and in￾struct model as the initial checkpoint for RL training. 0 50 100 150 200 250 Training Steps 0.5 0.0 0.5 1.0 Reward Fully Dense Piecewise Dense Sparse (a) Average Reward 0 50 100 150 200 250 Training Steps 0 200 400 600 800 1000 Re…
Figure 8
Figure 8. Figure 8: Ablation on reward granularity, comparing our [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Case study of successful steps 1-3 of WorkForceAgent-R1 solving a task of List-Sorting. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Case study of successful steps 4-6 of WorkForceAgent-R1 solving a task of List-Sorting. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Case study of successful steps 0-3 of WorkForceAgent-R1 solving a task of Service Catalog. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, and 1 others. 2024. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704

    arXiv 2024

  4. [4]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    arXiv 2024

  5. [5]

    De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, L \'e o Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, and 1 others. 2024. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467

    arXiv 2024

  6. [6]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091--28114

    work page 2023

  7. [7]

    Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. 2024. https://openreview.net/forum?id=BRfqYrikdo Workarena: How capable are web agents at solving common knowledge work tasks? In Forty-first International Conference on Machine Learning

    work page 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 53 references
  1. [9]

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2024. Multimodal web navigation with instruction-finetuned foundation models. The Twelfth International Conference on Learning Representations

    2024

  2. [10]

    Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, and Yu Su. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.436 Middleware for LLM s: Tools are instrumental for language agents in complex environments . In Proceedings of the 2024 C...

    2024 doi

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  4. [12]

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856

    2023 arXiv

  5. [13]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024 a . Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  6. [14]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. 2024 b . Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. arXiv preprint arXiv:2410.19609

    2024 arXiv

  7. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [16]

    Lawrence Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. 2024. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks. arXiv preprint arXiv:2410.19100

    2024 arXiv

  9. [17]

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and 1 others. 2024. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Disco...

    2024

  10. [18]

    Thibault Le Sellier De Chezelles, Maxime Gasse, and 1 others. 2024. The browsergym ecosystem for web agent research. arXiv e-prints, pages arXiv--2412

    2024

  11. [19]

    Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2024. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703

    2024 arXiv

  12. [20]

    Changhao Li, Yuchen Zhuang, Rushi Qiang, Haotian Sun, Hanjun Dai, Chao Zhang, and Bo Dai. 2024. Matryoshka: Learning to drive black-box llms with llms. arXiv preprint arXiv:2410.20749

    2024

  13. [21]

    Yusheng Liao, Shuyang Jiang, Yanfeng Wang, and Yu Wang. 2024. Reflectool: Towards reflection-aware tool-augmented clinical agents. arXiv preprint arXiv:2410.17657

    2024 arXiv

  14. [22]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

    2023 arXiv

  15. [23]

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172

    2023 arXiv

  16. [24]

    Peixian Ma, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo. 2025. Sql-r1: Training natural language to sql reasoning model by reinforcement learning. arXiv preprint arXiv:2504.08600

    2025

  17. [25]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    2025 arXiv

  18. [26]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  19. [27]

    OpenAI. 2025 a . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api . OpenAI Blog

    2025

  20. [28]

    OpenAI. 2025 b . https://openai.com/index/o3-o4-mini-system-card/ Openai o3 and o4-mini system card . OpenAI Blog

    2025

  21. [29]

    Jie Ouyang, Ruiran Yan, Yucong Luo, Mingyue Cheng, Qi Liu, Zirui Liu, Shuo Yu, and Daoyu Wang. 2025. https://github.com/0russwest0/Agent-R1 Training powerful llm agents with end-to-end reinforcement learning

    2025

  22. [30]

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. 2024 a . Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474

    2024 arXiv

  23. [31]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and 1 others. 2024 b . Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373

    2024 arXiv

  24. [32]

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, and 1 others. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. The Thirteenth International Conference on Lear...

    2025

  25. [33]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958

    2025 arXiv

  26. [34]

    Qwen. 2025. https://qwenlm.github.io/blog/qwen3/ Qwen3: Think deeper, act faster . Qwen Blog

    2025

  27. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  28. [36]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256

    2024 arXiv

  29. [37]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  30. [38]

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998--3009

    2023

  31. [39]

    Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. Adaplanner: Adaptive planning from feedback with language models. Advances in neural information processing systems, 36:58202--58245

    2023

  32. [40]

    Haotian Sun, Yuchen Zhuang, Wei Wei, Chao Zhang, and Bo Dai. 2024. https://openreview.net/forum?id=jdRIaUu3xY Bbox-adapter: Lightweight adapting for black-box large language models . In ICML

    2024

  33. [41]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning

    2024

  34. [42]

    Zihan Wang, Kangrui Wang, Qineng Wang, and 1 others. 2025. https://arxiv.org/abs/2504.20073 Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning . Preprint, arXiv:2504.20073

    2025 arXiv

  35. [43]

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, and 1 others. 2024. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161

    2024 arXiv

  36. [44]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  37. [45]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744--20757

    2022

  38. [46]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  39. [47]

    Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, and Tianmin Shu. 2025. Realwebassist: A benchmark for long-horizon web assistance with real-world users. arXiv preprint arXiv:2504.10445

    2025

  40. [48]

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. Assistantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.15711

    2024 arXiv

  41. [49]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614

    2024 arXiv

  42. [50]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

  43. [51]

    Rossi, Somdeb Sarkhel, and Chao Zhang

    Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A. Rossi, Somdeb Sarkhel, and Chao Zhang. 2024 a . https://openreview.net/forum?id=B6pQxqUcT8 Toolchain*: Efficient action space navigation in large language models with a* search . In The Twelfth Internat...

    2024

  44. [52]

    Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. 2024 b . https://openreview.net/forum?id=CKgNgKmHYp HYDRA : Model factorization framework for black-box LLM personalization . In The Thirty-eighth Annual Conference on Neural Information Proce...

    2024

  45. [53]

    Yuchen Zhuang, Jingfeng Yang, Haoming Jiang, and 1 others. 2025. Hephaestus: Improving fundamental agent capabilities of large language models through continual pre-training. arXiv preprint arXiv:2502.06589

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.