Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SEAgent enables a computer-use agent to master unfamiliar desktop software entirely from its own trial-and-error experience—no human-labeled data—lifting task success from 11.3% to 34.5% on five applications.

desk verdict Real contribution in the judge and curriculum, but the OSWorld gain may be partly in-distribution training; needs a task-exclusion statement and variance before I'd trust the 34.5%. read the letter →

arxiv 2508.04700 v2 pith:YEZ6FCCB submitted 2025-08-06 cs.AI cs.CLcs.CVcs.LGcs.MAcs.MM

Zeyi Sun , Ziyu Liu , Yuhang Zang , Yuhang Cao , Xiaoyi Dong , Tong Wu , Dahua Lin , Jiaqi Wang This is my paper · ORCID
classification cs.AIcs.CLcs.CVcs.LGcs.MAcs.MM
keywords computeruseagentsself-evolvingreinforcementlearningfromexperienceworldstatemodelcurriculumgenerationGRPOGUIOSWorld
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEAgent sets out to establish that a computer-use agent—a vision-language model that operates software by reading screenshots and sending clicks and keystrokes—can become good at applications it has never encountered, using no human-annotated training data. The agent explores each application on its own, while a World State Model judge watches the full screenshot history and marks every action as a step toward completion or as a mistake, and a curriculum generator holding an evolving 'software guidebook' issues tasks of rising difficulty. The policy is updated by reinforcing the correct steps with GRPO and step-level distance rewards while suppressing the mistaken ones with an adversarial-imitation term. On five desktop applications from OSWorld, the loop takes the open-source UI-TARS agent from 11.3% to 34.5% success, and a generalist distilled from per-software specialists beats the ensemble of those specialists. If the paper is right, adapting an agent to new software no longer requires expensive human demonstrations.

What carries the argument

The load-bearing component is the World State Model: a judge fine-tuned from Qwen2.5-VL-7B that consumes the agent's entire trajectory of screenshots and actions and returns a per-step verdict—contributed to success (aT) or error/redundancy (aF). Co-training it to caption GUI changes between consecutive screenshots measurably improves judgment precision. These step-level labels are the pipeline's reward signal: positive actions feed GRPO with verifiable, action-type-specific distance rewards (normalized L1 for clicks, intersection-over-union for drags/selects, character BLEU for typed text); failure actions are penalized by an adversarial-imitation negative-KL loss that pushes the policy awa

What would settle it

An independent human audit: sample trajectories from each of the five applications, have human annotators label each step as correct, redundant, or failing, and measure the World State Model's per-step agreement. The claim is settled if per-application agreement tracks the reported gains—if the judge is markedly less accurate on, say, Writer or VLC, the headline 11.3%-to-34.5% improvement is likely a reward-mislabeling artifact. A complementary check: rerun the three-phase loop using only rule-based environment verifiers as reward (no judge labels) and compare how much of the 23.2-point gain s

Watch

Extended reading notes

Core claim

SEAgent claims a computer-use agent can master previously unseen desktop software through a closed loop of exploration, judgment, and policy update. A World State Model judge, fine-tuned from Qwen2.5-VL-7B, watches the full screenshot history and labels every action as correct or failing; a Curriculum Generator backed by a growing software guidebook issues progressively harder tasks; correct actions are reinforced with GRPO and action-specific distance rewards, while failures are suppressed by an adversarial-imitation term. On five OSWorld applications, the pipeline lifts the UI-TARS agent from 11.3% to 32.2% as separate specialists. A specialist-to-generalist strategy—distilling 3.5K succes

Load-bearing premise

The load-bearing premise is that the World State Model's per-step labels stay accurate in the five target applications even though the judge was trained on only 860 GPT-4o-annotated trajectories from a single application (Chrome) plus 1,000 change-description pairs; if its 'correct' and 'failure' labels are systematically wrong in VSCode, GIMP, Impress, VLC, or Writer, the 23.2-point success gain would reflect label bias, not acquired skill.

Editorial extensions

If this is right

  • If SEAgent is right, adapting a computer-use agent to novel software no longer requires human demonstrations: the agent bootstraps its own curriculum, judges its own actions, and improves from roughly one in nine to roughly one in three tasks solved on five professional applications (11.3% to 34.5%).
  • Step-level rewards derived from judging the whole trajectory outperform the sparse, critic-estimated rewards used by prior RL methods (DigiRL, WebRL) on the same model and environments (32.2% vs 21.8%).
  • Each design choice contributes: in the VSCode ablation, the World State Model judge beats a base-model judge, GRPO-based reinforcement beats plain behavior cloning, and adding adversarial imitation on failure steps further raises the success rate to 37.7%.
  • Specialization before generalization wins: a generalist produced by distilling successful trajectories from five specialists and then re-running the learning loop (34.5%) outperforms both the specialist ensemble (32.2%) and direct multi-software training (30.6%).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's inference: the authors leave per-application judge accuracy unmeasured; if a human audit of the World State Model's labels shows accuracy varying across the five apps, the specialist-to-generalist ranking could shift once label noise is corrected, making judge auditing the highest-value follow-up experiment.
  • Editor's inference: the specialist-to-generalist recipe is a general training pattern—train cheap per-domain experts, distill their successful trajectories, then refine on all domains. Any environment with a bootstrappable automatic judge, such as browser automation, game levels, or robot simulators, is a candidate for the same loop.
  • Editor's inference: the authors note their tasks require fewer than 20 expert steps; extending to hour-long, multi-application workflows is untested. A plausible stress test is whether judge accuracy degrades on much longer trajectories, which would motivate hierarchical or environment-grounded judging.
  • Editor's inference: nothing in the paper sets a horizon on how many curriculum phases remain beneficial, so the same loop could plausibly be re-applied to the generalist repeatedly, with each cycle generating harder tasks and further gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents SEAgent, a self-evolving computer-use-agent framework that starts from a pretrained LVLM (UI-TARS-7B-DPO) and, without human labels, trains on tasks automatically generated for unfamiliar software. Three components are introduced: a World State Model (WSM) that provides step-level success/failure labels and GUI change captions; a Curriculum Generator with a software guidebook memory that proposes progressively harder tasks; and an experiential RL update combining GRPO on labeled successful actions with adversarial imitation on labeled failure actions. A specialist-to-generalist strategy first trains per-application specialists and then distills their successful trajectories into one generalist before further RL. The main empirical claim is on five OSWorld applications (VSCode, GIMP, Impress, VLC, Writer): success rate improves from 11.3% for UI-TARS to 34.5% for SEAgent with specialist-to-generalist training, surpassing both specialist RL (32.2%) and generalist RL (30.6%).

Significance. If the headline result holds, the paper makes a useful step toward reducing human annotation cost in GUI agents: the WSM is independently benchmarked on AgentRewardBench and OSWorld trajectories, ablations isolate the contributions of WSM rewards, GRPO, and adversarial imitation (Table 3), and the ScienceBoard experiments (Appendix D) provide a sanity check on a distinct, harder target domain. The strongest and most valuable claim is that a curriculum generated from agent experience plus step-level reward labels can yield substantial gains on applications where the base model was weak. However, the central 11.3%→34.5% comparison currently lacks variance reporting, and the possibility of task overlap between the self-generated training curriculum and the OSWorld evaluation set is not addressed. These issues prevent the paper from supporting its strongest claim at the confidence level the authors assert.

major comments (3)
  1. [Sec. 4.2; Algorithm 1, lines 24-25; Fig. 4] The central claim is vulnerable to benchmark contamination. Training and evaluation are conducted on the same five OSWorld applications, and the paper never states that OSWorld evaluation task IDs were excluded from the Curriculum Generator. Algorithm 1's Task Update step (lines 24-25) feeds all generated tasks into the next training phase, and Sec. 4.2 shows that these generated task instructions (e.g., 'Open the video on desktop', 'Increase volume', 'Save current video to 1.mp4') are semantically close to OSWorld-style tasks. The specialist-to-generalist model is then SFT'd on successful trajectories from these same generated tasks. If any generated task coincides with or closely resembles an OSWorld evaluation task, the reported 23.2-point gain partly reflects training on test material rather than general self-evolution. The authors must provide an explicit exclusion statement, a task
  2. [Appendix A.2; Table 1; Sec. 4.2 ablation] The reward signal that drives GRPO and adversarial imitation comes from a WSM trained on only 860 GPT-4o-annotated Chrome trajectories and 1,000 change-description pairs. Table 1 reports aggregate WSM precision/NPV on OSWorld Prof/Office, but not per-application accuracy for the five target apps (VSCode, GIMP, Impress, VLC, Writer). Since the WSM's aT/aF labels are the only learning signal, systematically wrong labels in any of these apps would make the reported gains reflect label bias rather than skill acquisition. The paper's own limitation statement notes that WSM is a learned judge rather than an environment oracle. The authors should report per-app judge precision (or a validated held-out subset) and analyze how label error correlates with the observed policy improvements.
  3. [Table 2 and Sec. 4.2] The paper states 'All results are averaged over three runs' but reports no variance, confidence intervals, or significance tests. For the headline claim that SEAgent yields a significant improvement from 11.3% to 34.5%, and for the claim that specialist-to-generalist (34.5%) outperforms specialist RL (32.2%) and generalist RL (30.6%), the reader cannot assess whether these differences exceed run-to-run noise. Please report standard deviations or error bars and, where feasible, a significance test on the main comparisons.
minor comments (6)
  1. [Table 1] Header contains a typo: 'AgentReardBench' should be 'AgentRewardBench'.
  2. [Fig. 4 caption] 'Curriculum' is misspelled as 'Curriclum' in the caption, and 'VScode' is used inconsistently with 'VSCode' elsewhere.
  3. [Fig. 3 and Appendix A] The model is called both 'World State Model' and 'GUI-Judge' (Fig. 3 legend, Appendix A). Please use one name throughout, or explicitly state they are the same model.
  4. [Sec. 3.2 / Algorithm 1] The paper uses 'RFT' (reinforcement fine-tuning) in several places but the actual update is GRPO with a KL penalty and adversarial-imitation auxiliary loss. Consider aligning terminology so the reader is not confused about the algorithm being run.
  5. [Appendix C.2] The citation to ScienceBoard appears as '[? ]' in the text and should be filled in (it is reference [57] in the bibliography).
  6. [General] The GitHub URL is provided, but the paper does not state whether code, generated task lists, or model weights will be released. Adding an explicit data/code availability statement would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline improvement is measured by the independent OSWorld ground-truth protocol, while the fitted World State Model is used only as a training-time reward signal.

full rationale

The claimed derivation chain is empirical rather than definitional. The final success rates in Table 2 are computed with OSWorld's own rule-based evaluation, which is external to the World State Model; WSM's aT/aF labels are training signals only (Sec. 3.2, Eqs. 2-5), and no reported OSWorld success value is produced by WSM. WSM itself is validated against the external AgentRewardBench benchmark and the out-of-domain Prof/Office OSWorld columns (Table 1), and Appendix D shows the same pipeline improves UI-TARS-1.5 on ScienceBoard, an independent benchmark, which breaks any self-referential loop. The Limitations section concedes that the system is 'bounded by GUI-Judge to provide reliable reward signal instead of real signal from environment,' but that is a reward-accuracy robustness limitation, not circularity. The skeptic's concern that curriculum tasks may overlap OSWorld evaluation tasks (Sec. 4.2/Fig. 4) is a data-contamination/leakage risk, not a reduction of the result to its inputs: even if overlap existed, the evaluation score would be inflated or confounded, but the derivation chain would still terminate in an external ground-truth measurement rather than in the fitted model. The self-citations present ([13], [33], [58], [59], [69]) are background references and are not load-bearing for the main result. I therefore find no step where a prediction is definitionally identical to a fitted input or a cited prior result.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

All free parameters are optimization choices selected on VSCode, so the headline OSWorld number is partly tuned to that app; the judge-model assumptions (GPT-4o teacher, Chrome-to-app transfer, OSWorld oracle) are the true hidden load. The result is an empirical performance improvement, not a derived law, so no equation-level circularity is present, but the chain of assumptions is long.

free parameters (5)
  • gamma (adversarial imitation weight) = 0.2
    Sup. F Table 7: VSCode success ranges from 23.1% at gamma=0.8 to 37.7% at 0.2; chosen by ablation.
  • generated tasks per phase = 100
    Sup. E Table 6: VSCode SR plateaus at 100 tasks.
  • change descriptions per task set = 50 to 100
    Sup. E Table 6: 100 best (37.68), 200 hurts (34.78).
  • number of training phases = 3
    Sec 4.2 and Fig. 4: authors stop at three phases because gains saturate.
  • GRPO group size G, batch size, learning rate = G=8, batch=16, LR=2e-5
    Sec 4.2: fixed across all runs; not independently searched.
assumptions (5)
  • domain assumption OSWorld rule-based evaluator is a valid ground truth for success/failure
    Used to filter WSM training judgments and to compute Table 2; if OSWorld judgments are noisy, both training labels and evaluation inherit error.
  • domain assumption GPT-4o judgments, filtered by OSWorld ground truth, are a valid teacher for WSM
    Sup. A.2: WSM is fine-tuned on 0.86K GPT-4o evaluations; no human audit is reported.
  • domain assumption WSM step-level aT/aF labels transfer from Chrome to the five evaluated apps
    The step labels that drive GRPO and adversarial imitation in all five apps come from a model trained only on Chrome; Table 1 is evidence, not guarantee.
  • domain assumption Curriculum Generator tasks are executable from the initial state and ordered by difficulty
    Curriculum Generator's tasks are not verified for executability before being assigned; the paper assumes ill-posed tasks are rare.
  • standard math GRPO and DPO-style loss formulas hold as in prior work
    Eqs. 3 to 5 are taken from DeepSeekMath/DPO; no new guarantees are given.
invented entities (2)
  • World State Model independent evidence
    purpose: Step-level trajectory judge and state-change captioner that provides RL rewards
    Benchmarked on AgentRewardBench and OSWorld judge precision (Table 1), so it has an external handle.
  • Software guidebook memory
    purpose: Accumulating memory of software functions used by the Curriculum Generator to write harder tasks
    Internal to the pipeline; no falsifiable prediction outside the system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience." pith.science (2026). https://pith.science/paper/YEZ6FCCB

@misc{pith2026250804700,
  author       = {Pith},
  title        = {Pith review of: SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEZ6FCCB}},
  note         = {Machine review of arXiv:2508.04700}
}
read the original abstract

Repurposing large vision-language models (LVLMs) as computer use agents (CUAs) has led to substantial breakthroughs, primarily driven by human-labeled data. However, these models often struggle with novel and specialized software, particularly in scenarios lacking human annotations. To address this challenge, we propose SEAgent, an agentic self-evolving framework enabling CUAs to autonomously evolve through interactions with unfamiliar software. Specifically, SEAgent empowers computer-use agents to autonomously master novel software environments via experiential learning, where agents explore new software, learn through iterative trial-and-error, and progressively tackle auto-generated tasks organized from simple to complex. To achieve this goal, we design a World State Model for step-wise trajectory assessment, along with a Curriculum Generator that generates increasingly diverse and challenging tasks. The agent's policy is updated through experiential learning, comprised of adversarial imitation of failure actions and Group Relative Policy Optimization (GRPO) on successful ones. Furthermore, we introduce a specialist-to-generalist training strategy that integrates individual experiential insights from specialist agents, facilitating the development of a stronger generalist CUA capable of continuous autonomous evolution. This unified agent ultimately achieves performance surpassing ensembles of individual specialist agents on their specialized software. We validate the effectiveness of SEAgent across five novel software environments within OS-World. Our approach achieves a significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a competitive open-source CUA, i.e., UI-TARS.

Figures

Figures reproduced from arXiv: 2508.04700 by the authors.

Figure 1
Figure 1. SEAgent enables computer use agents self-evolving in novel environments by au￾tonomously exploring and learning from their own experiences without human intervention. The specialist-to-generalist training strategy further enhances the development of a strong generalist agent. from demonstrations [34, 78, 18, 51, 75] or video tutorials in the wild [70]. However, new software continuously emerges and existing software… view at source ↗
Figure 2
Figure 2. SEAgent autonomous exploration and experiential learning pipeline. Guided by tasks generated by the Curriculum Generator, the Actor Model is updated according to step-level rewards from the World State Model through verifiable reward functions tailored for different action types. (3) Curriculum Generator Mtask: This component utilizes a powerful Large Language Model (LLM) to automatically generate novel exploration … view at source ↗
Figure 3
Figure 3. The Average Precision on AgentRe￾wardBench [35], where GUI-Judge exhibits an improvement in AP as the number of input middle states increases, showing a similar trend to that of the closed sourced GPT-4o [23] when compared with its base model. We evaluate World State Model and our full-process screenshot-conditioned strategy on AgentRewardBench [35], as well as on agent trajectories from OSWorld [68]. As shown in Ta… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Self-evolved task instructions and success rate (SR) curves across different software. Tasks are progressively upgraded by the Curriculum Generator without human intervention, based on the evolving capabilities of the Actor Model at different training phases. Models Be…
Figure 5
Figure 5. Figure 5: SEAgent autonomous exploration pipeline. The agent (policy model) and World State Model iteratively generate new task and perform RL to become a specialist in novel software. Experimental Setup We adapted the official code and prompts from these prior works from web en…
Figure 6
Figure 6. Figure 6: Prompt Template of GUI-Judge for web agent trajectories evaluations with history screenshots as input, its difference with default prompt of AgentRewardBench [35] is highlighted in bold. K Broader Impacts Potential positive societal impacts: SEAgent introduces a self-e…
Figure 7
Figure 7. Figure 7: Prompt Template of GUI-Judge for OSWorld [68] trajectories, which prompts judge model to provide step level reward signal. L SEAgent Self-Evolution Algorithm Algorithm 1 presents the core self-evolution training loop of SEAgent in a specialized software environment. Th…
Figure 8
Figure 8. Figure 8: Prompt Template for task buffer update, which generates new tasks in a curriculum manner and update software documents. The new tasks are used for actor to perform next phase of RL. (4) Task Update. The Curriculum Generator leverages feedback signals (J ) and GUI state…
Figure 9
Figure 9. Figure 9: Automatically generated usage manual during self exploration on VScode [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Automatically generated usage manual during self exploration on GIMP [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Automatically generated usage manual during self exploration on LibreOffice_Impress. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Automatically generated usage manual during self exploration on LibreOffice_Writer. Algorithm 1 SEAgent Specialized Self-Evolution Training Loop 1: Input: Initial policy π0, World State Model Mstate, Curriculum Generator Mtask, Initial GUI state S0 2: 1. Task Initiali…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

    cs.AI 2026-07 conditional novelty 6.5 of 10

    A compact multi-agent judge with a shared 9B backbone matches or beats rule-based reward signals in online RL for computer-use agents, per the authors' held-out success-rate measurements.

  2. CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.

  3. Learning What to Remember and What to Internalize in LLM Self-Evolution via Adaptive Memory-Parameter Coordination

    cs.AI 2026-08 conditional novelty 5.0 of 10

    COVE coordinates memory-based and parameter-based LLM self-evolution by routing tasks, scheduling training on plateau or cold-start signals, and keeping volatile API knowledge out of model weights via an anti-recitati...

  4. HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

    cs.HC 2026-06 unverdicted novelty 5.0 of 10

    HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.

  5. Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward

    cs.MA 2026-02 unverdicted novelty 4.0 of 10

    The paper surveys agent skills for LLMs across architecture, acquisition, deployment, and security, proposing a four-tier Skill Trust and Lifecycle Governance Framework to address vulnerabilities in community skills.

Reference graph

Works this paper leans on

97 extracted references · 23 canonical work pages · cited by 5 Pith papers

  1. [1]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    arXiv 2024

  2. [2]

    Agent s2: A compositional generalist-specialist framework for computer use agents

    Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents. arXiv preprint arXiv:2504.00906, 2025

    arXiv 2025

  3. [3]

    Claude computer use

    Anthropic. Claude computer use. 2024

    2024

  4. [4]

    Claude 3.7 sonnet system card

    Anthropic. Claude 3.7 sonnet system card. https://assets.anthropic.com/m/ 785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf , 2025

    2025

  5. [5]

    Claude’s extended thinking

    Anthropic. Claude’s extended thinking. 2025

    2025

  6. [6]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems , 37:12461–12495, 2024

    2024

  7. [7]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  8. [8]

    Grounding large language models in interactive environments with online reinforcement learning

    Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning , pages 3676–3713. PMLR, 2023

    2023

Show all 97 references
  1. [9]

    Guicourse: From general vision language models to versatile gui agents

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317, 2024

    2024 arXiv

  2. [10]

    Bail: Best- action imitation learning for batch deep reinforcement learning.Advances in Neural Information Processing Systems, 33:18353–18363, 2020

    Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning.Advances in Neural Information Processing Systems, 33:18353–18363, 2020

    2020

  3. [11]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  4. [12]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  5. [13]

    Mm-ifengine: Towards multimodal instruction following

    Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Mm-ifengine: Towards multimodal instruction following. arXiv preprint arXiv:2504.07957, 2025

    2025 arXiv

  6. [14]

    Gemini 2.5 Pro Preview (03-25)

    Google DeepMind. Gemini 2.5 Pro Preview (03-25). https://deepmind.google/ technologies/gemini, 2025

    2025

  7. [15]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  8. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 11

    2025 arXiv

  10. [18]

    A real-world webagent with planning, long context understanding, and program synthesis

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023

    2023 arXiv

  11. [19]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  12. [20]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14281–1...

    2024

  13. [21]

    Cogagent: A visual language model for GUI agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for GUI agents. CoRR, abs/2312.08914, 2023

    2023 arXiv

  14. [22]

    Os agents: A survey on mllm-based agents for general computing devices use, 2024

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use, 2024

    2024

  15. [23]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [24]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161...

    2024

  17. [25]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  18. [26]

    Autowebglm: A large language model- based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model- based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and D...

    2024

  19. [27]

    Screenspot-pro: Gui grounding for professional high-resolution computer use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  20. [28]

    On the effects of data scale on computer control agents

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv e-prints, pages arXiv–2406, 2024

    2024

  21. [29]

    Showui: One vision-language-action model for gui visual agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent. arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  22. [30]

    Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36:34892–34916, 2023

    2023

  23. [31]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    2024 arXiv

  24. [32]

    Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960, 2023. 12

    2023 arXiv

  25. [33]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  26. [34]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  27. [35]

    Agen- trewardbench: Evaluating automatic evaluations of web agent trajectories

    Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Ale- jandra Zambrano, Karolina Sta´nczak, Peter Shaw, Christopher J Pal, and Siva Reddy. Agen- trewardbench: Evaluating automatic evaluations of web agent trajectories. arXiv preprint arXiv:2504.0...

    2025

  28. [36]

    Laser: Llm agent with state-space exploration for web navigation

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172, 2023

    2023 arXiv

  29. [37]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  30. [38]

    Bagel: Bootstrapping agents by guiding exploration with language, 2024

    Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Bootstrapping agents by guiding exploration with language, 2024

    2024

  31. [39]

    Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsuper- vised learning of browser agents through environment interaction in the wild, 2025

    2025

  32. [40]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  33. [41]

    Gui agents: A survey

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. arXiv preprint arXiv:2412.13501, 2024

    2024

  34. [42]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023

    2023 arXiv

  35. [43]

    Operator

    OpenAI. Operator. 2025

    2025

  36. [44]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  37. [45]

    Au- tonomous evaluation and refinement of digital agents

    Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024

    2024 arXiv

  38. [46]

    Agent q: Advanced reasoning and learning for autonomous ai agents

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024

    2024 arXiv

  39. [47]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024

    2024 arXiv

  40. [48]

    UI-TARS: pioneering automated GUI interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, C...

    2025 arXiv

  41. [49]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728–53741, 2023. 13

    2023

  42. [50]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  43. [51]

    An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    2023

  44. [52]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  45. [53]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  46. [54]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  47. [55]

    Welcome to the era of experience

    David Silver and Richard S Sutton. Welcome to the era of experience. Preprint of a chapter to appear in Designing an Intelligence, edited by George Konidaris, MIT Press (forthcoming) , 2025

    2025

  48. [56]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  49. [57]

    Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505.19897, 2025

    2025 arXiv

  50. [58]

    X-prompt: Towards universal in-context image generation in auto-regressive vision language foundation models, 2024

    Zeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. X-prompt: Towards universal in-context image generation in auto-regressive vision language foundation models, 2024

    2024

  51. [59]

    Bootstrap3d: Improving 3d content creation with synthetic data

    Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Bootstrap3d: Improving 3d content creation with synthetic data. arXiv e-prints, pages arXiv–2406, 2024

    2024

  52. [60]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  53. [61]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  54. [62]

    Omniparser: A unified framework for text spotting key information extraction and table recognition

    Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: A unified framework for text spotting key information extraction and table recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

    2024

  55. [63]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  56. [64]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 14

    2024 arXiv

  57. [65]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  58. [66]

    Os-copilot: Towards generalist computer agents with self-improvement

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024

    2024 arXiv

  59. [67]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    2024 arXiv

  60. [68]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  61. [69]

    Scalecap: Inference-time scalable image captioning via dual-modality debiasing

    Long Xing, Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jinsong Li, Shuangrui Ding, Weiming Zhang, Nenghai Yu, et al. Scalecap: Inference-time scalable image captioning via dual-modality debiasing. arXiv preprint arXiv:2506.19848, 2025

    2025 arXiv

  62. [70]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09605, 2024

    2024 arXiv

  63. [71]

    Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562, 2023

    2023 arXiv

  64. [72]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  65. [73]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning

    Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems , 37:110935– ...

    2024

  66. [74]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , pages 1–20, 2025

    2025

  67. [75]

    Agentohana: Design unified data and training pipeline for effective agent learning

    Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Ming Zhu, Juntao Tan, Thai Hoang, Zuxin Liu, Liangwei Yang, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXiv:2402.15506, 2024

    2024 arXiv

  68. [76]

    Android in the zoo: Chain-of-action-thought for gui agents

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024

    2024 arXiv

  69. [77]

    Towards building specialized generalist ai with system 1 and system 2 fusion, 2024

    Kaiyan Zhang, Biqing Qi, and Bowen Zhou. Towards building specialized generalist ai with system 1 and system 2 fusion, 2024

    2024

  70. [78]

    You only look at screens: Multimodal chain-of-action agents

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436, 2023

    2023 arXiv

  71. [79]

    Gpt-4v (ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  72. [80]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 15

    2023 arXiv

  73. [81]

    Archer: Training language model agents via hierarchical multi-turn rl

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  74. [82]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 16 A World State Model The World State Model (WSM) is a centr...

    1909 arXiv

  75. [83]

    Trajectory Judgment: Input: A sequence of screenshot images captured during an episode. Output: Short captions for each screenshot, the reasoning process for the judgment, and a structured judgment dictionary (containing fields such as Correctness, Redundant, and First Error S...

  76. [84]

    Output: A detailed description of the visual differences between the two images

    State Change Description: Input: Two screenshot images, one from before and one after a single action was executed. Output: A detailed description of the visual differences between the two images. A.2 Fine-Tuning Dataset and Process To equip the WSM with these capabilities, a ...

  77. [85]

    These trajectories were saved as screenshot sequences

    Trajectory Sampling: A Computer Using Agent (CUA), powered by UI-TARS and Gemini- 2.5-Pro, was used to sample trajectories from 43 feasible tasks in Google Chrome within the OSWorld benchmark. These trajectories were saved as screenshot sequences

  78. [86]

    Only samples where the judgment matched the ground truth from OSWorld evaluation protocols were retained, resulting in 860 high-quality annotated trajectories

    GPT-4o Annotation: Using the prompts detailed in Figures 6 and 7 of the supplementary material, GPT-4o was employed to annotate the sampled trajectories, generating judgments and screenshot captions. Only samples where the judgment matched the ground truth from OSWorld evaluat...

  79. [87]

    add a rectangle

    Change Description Data: An additional 1,000 pairs of (before action, after action) screenshots were sampled. GPT-4o was used to generate detailed descriptions of the differences, creating a 1,000-sample Change Description (CD) dataset. Fine-Tuning Process The fine-tuning was ...

  80. [88]

    add a rectangle

    Analysis and Feedback: The agent successfully completes an initial task, "add a rectangle." The WSM analyzes the execution trajectory and extracts two key pieces of information: a task evaluation (Exam) and a list of observed state changes (CD_list). CD_list: {"add a rectangle...

  81. [89]

    properties of a rectangle,

    Knowledge Integration and Task Generation: The CD_list and Exam are fed into the Curriculum Generator. It distills new knowledge, such as "properties of a rectangle," and integrates it into its internal Software guidebook . Based on this new knowledge, it generates more challe...

  82. [90]

    Add a Rectangle,

    Iterative Learning: In the next RL phase, the agent samples from this updated, more challenging task buffer. The continuously enriched Software guidebook acts as the system’s long-term memory, driving the Curriculum Generator to propose increasingly sophisticated and unexplore...

  83. [91]

    software guidebook

    benchmark—a suite of scientific applications that are truly novel to UI-TARS-1.5—our pipeline delivers significant and substantial improvements. This strongly validates our core claim: SEAgent is most impactful when performing self-evolution learning on truly OOD software. We ...

  84. [92]

    Correctness — Did the agent successfully complete the task as instructed?

  85. [93]

    Redundant Steps — Identify any unnecessary or repeated actions that do not contribute to the goal

  86. [94]

    Optimization — Did the agent follow an efficient plan with a minimal number of steps?

  87. [95]

    keyframe where a mistake occurred

    First Error Step — If the execution is incorrect or sub-optimal, determine the index of the first 5. keyframe where a mistake occurred

  88. [96]

    Error Analysis — Provide a brief explanation of the mistake at that step

  89. [97]

    Correctness

    Correct Action Suggestion — Explain what the agent should have done instead at the point of error. Important Instructions: The agent may have made progress toward the goal, but unless the task is fully and correctly completed, you must set 'Correctness' to False. Be cautious i...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.