Pith. sign in

REVIEW 4 major objections 6 minor 27 references

A single human screen recording lets a GUI agent automate tasks that stumped the best open-source agents, reaching 60% success on a 20-task OSWorld set.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 22:50 UTC pith:WS7DCTGE

load-bearing objection A training-free GUI agent that replays a single human demonstration with verifier/backtracker could be a practical step forward, but the 60% headline needs artifacts and demo-validation before I'd trust it. the 4 major comments →

arxiv 2509.07098 v1 pith:WS7DCTGE submitted 2025-09-08 cs.AI cs.CL

Instruction Agent: Enhancing Agent with Expert Demonstration

Yinheng Li , Hailey Hultquist , Justin Wagle , Kazuhito Koishida This is my paper
classification cs.AI cs.CL
keywords GUI agentexpert demonstrationtest-time inferencetraining-freeOSWorldUI groundingverificationbacktracking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Instruction Agent claims that a single recorded human demonstration is enough to let a GUI agent reliably complete tasks that top open-source agents fail. Instead of training on large trajectory datasets, it converts the recording into a step-by-step instruction list, then executes each step with a grounding model, a screen-diff verifier, and a backtracker. On 20 OSWorld tasks that three leading open-source agents all failed, the agent reached 60% success, compared with 72.36% for humans on the full benchmark. Removing the backtracker drops success to 45%, and removing both verifier and backtracker to 40%, so the verification loop carries the result. If the claim holds, end users can automate their own bespoke, long-horizon workflows with a short recording and no model training.

Core claim

The paper's claim is that a single expert demonstration can replace both the planning and much of the grounding burden of a GUI agent. The Instructor turns (screenshot, action) pairs from a recorded trajectory into natural-language step descriptions that include the expected outcome of each action; the Actor then executes those steps with a grounding model, verifies after each action by comparing before/after screenshots, and backtracks to a saved observation when verification fails. On 20 OSWorld tasks that the top three open-source agents all failed, this pipeline reaches 60% success (12 of 20), versus 72.36% for humans on the full OSWorld benchmark; ablations show the verifier and backtra

What carries the argument

The central mechanism is the Instructor–Actor split with a verification loop. The Instructor converts recorded pre-action screenshots and input events into step-by-step natural-language actions annotated with their intended effect; the Actor iterates through them using UI-Tars to ground coordinates, GPT-4o to generate PyAutoGUI code, a Verifier that compares the screenshots before and after each action to decide success, and a Backtracker that restores the prior observation and retries on failure. The expected-outcome phrasing in the instructions is what lets the verifier distinguish a successful step from a blind one.

Load-bearing premise

The whole pipeline assumes the recorded human demonstration is a successful trajectory with each screenshot exactly paired with the action that follows it, so a mistaken action or misaligned screenshot would bake an error into every subsequent step.

What would settle it

Run the same 20 OSWorld tasks with demonstrations whose screenshots are deliberately misaligned by one action (each step paired with the screenshot from the following step) while keeping the instruction text identical. If success stays near 60%, precise state-action alignment is not load-bearing; if it collapses toward the 40% no-verifier baseline, the recording assumption is the load-bearing premise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Tasks that all three selected top open-source agents failed become automatable at 60% with one demonstration, so a human can offload long, idiosyncratic workflows instead of scripting them.
  • The verifier and backtracker are load-bearing: removing the backtracker drops success to 45%, and removing both to 40%, so reliable automation depends on checking each action and recovering from divergence.
  • Because the system is training-free and runs on existing LLM APIs, an end user can create a demonstration without machine-learning expertise or compute.
  • Strictly following the demonstrated trajectory counters the exponential compounding of per-step failures, since each step's success probability is raised by the high-quality plan and grounding hints.
  • Demonstrations can be packaged as reusable APIs or tools and lightly edited for similar tasks, amortizing the one-time recording cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the paper's framing suggests the recorded screenshots carry grounding hints beyond the text, but it never isolates how much of the 60% comes from re-using the demonstration's visual context at test time; running the actor on instruction text alone would separate those contributions.
  • Testable extension: a library of reusable demonstration steps could be composed for variant tasks, since the paper only notes that demos 'can often be adapted to similar tasks with minor edits' and does not test composition.
  • Editorial: because grounding errors are reported as a top failure mode and grounding models are improving, the same 20-task set is a moving target — success should rise without any method change, which also makes the exact 60% figure time-sensitive.
  • Testable extension: deliberately recording a demonstration that takes an inefficient path (extra scrolls, redundant clicks) would test whether strict trajectory-following hurts on tasks where the human's route is suboptimal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Instruction Agent, a training-free, test-time-only GUI agent. From a single human demonstration (screenshots plus mouse/keyboard events), an Instructor module generates step-by-step natural-language instructions. At execution time, an Actor module grounds each instruction with UI-Tars 1.5, generates PyAutoGUI code via GPT-4o, verifies each step's effect from before/after screenshots, and invokes a backtracker on failure. The authors evaluate on 20 OSWorld tasks that three top open-source agents (ranks 3, 4, 6) all failed, reporting 60% success (12/20), with ablations showing 45% without the backtracker and 40% without both verifier and backtracker. One full instruction list and one successful 21-step trace are provided in the appendix.

Significance. If the 60% result is reproducible, it is significant: it suggests that a single expert demonstration, converted offline into instructions, can let a modest test-time pipeline solve long-horizon OSWorld tasks that strong agents fail, with no training or fine-tuning. The idea is simple and the appendix trace shows a coherent execution. The modular design (instruction generation, grounding, verification, backtracking) is sensible, and the choice to evaluate on tasks failed by all three baselines is an appropriate way to demonstrate added value. However, the empirical support is currently too thin: the demonstrations are not validated, the sample is small, and no artifacts are provided. The significance is therefore conditional on the missing evidence.

major comments (4)
  1. [§3.2.1, §4.1.2] The entire pipeline consumes state-action pairs from a human recording. The paper states in §3.2.1 that 'for each user action during the recording phase, we capture the screenshot immediately preceding that action,' but it never validates that (a) each of the 20 recorded demonstrations actually completed its OSWorld task under the benchmark's evaluator, or (b) the screenshots are correctly time-aligned with the recorded actions. Because the Instructor (Section 3.2.1) generates each instruction from these pairs, a failed or misaligned demonstration embeds the error into every subsequent step; the verifier/backtracker can only catch a subset. The single worked trace in Appendix A.2 does not establish the quality of the other 19 demonstrations. Without this validation the 60% headline is not interpretable. Please report per-demonstration success validation (e.g., OSWorld evaluator on the re
  2. [§4.1.1, §4.1.4, Table 1] The headline result is 12/20 = 60%. With n=20 the exact binomial 95% CI is roughly [36%, 81%], so the comparison to the baseline agents' 0% (also on these 20 tasks) is suggestive but not precise. The paper does not report the sampled task IDs, the random seed, or per-task outcomes. Please provide the task list and per-task success/failure, plus a confidence interval or a larger evaluation. This is load-bearing because the abstract's '60% success rate' is the paper's central quantitative claim.
  3. [§4.1.2, §A.1] The evaluation is not reproducible from the manuscript: no code, Docker configuration, prompts for the Instructor/Verifier/Backtracker/Executor, grounding-model version beyond 'UI-Tars 1.5 7B', or GPT-4o model snapshot are provided. The full instruction list in Appendix A.1 is for one task only. Since the method is training-free and test-time-only, the exact prompts and recording script are essential to reproduce the 60% number. Please release these artifacts or describe them in sufficient detail.
  4. [§4.2, Table 2] The ablation claims that the verifier and backtracker are 'critical' (Section 3.2.2). The numbers 60%, 45%, 40% on n=20 correspond to 12, 9, and 8 successes; removing both modules changes the count by only 4 tasks. With no confidence intervals or paired analysis, the differences are not statistically distinguishable. Please report per-task ablation outcomes and an appropriate uncertainty estimate.
minor comments (6)
  1. [Introduction] Typo: 'a substantial performance gap substantial gap remains'.
  2. [§4.1.1] Duplicate phrase: 'We randomly sampled 20 of these 130 tasks 20 of these 130 tasks'.
  3. [§4.2] Duplicate word: 'an incorrect action pushed the environment into a different state which which required'.
  4. [Table 1] Human 72.36% is OSWorld's overall human success rate, not the human rate on the 20 selected tasks; the comparison would be fairer if the authors also measured human performance on these 20 tasks. Also, the 0% baseline rows are by construction (selection criterion), not fresh measurements; the table should state this explicitly.
  5. [§2] The claim that this is 'the first to use test-time inference based solely on expert demonstrations' is too strong given Ruoss et al. (LMACT) and Jang et al., which also study direct use of demonstrations at test time; please qualify the novelty.
  6. [§A.1] The instruction list uses JSON-like braces but is not valid JSON; this is cosmetic but should be cleaned up.

Circularity Check

0 steps flagged

No significant circularity: the method is a replay-based pipeline evaluated on an external benchmark, with no fitted parameters or self-citation chain carrying the central claim.

full rationale

The paper's derivation chain is: a single human demonstration is recorded, an LLM converts each recorded state-action pair into a natural-language instruction, and an actor executes those instructions with a verifier and backtracker. No step in this chain is defined in terms of the evaluation outcome, and no parameter is fitted to the benchmark. The 60% success rate on OSWorld tasks that three top agents failed is an empirical result conditional on the provided demonstrations; the demonstrations are external inputs, not derived from the success signal. The instruction list is indeed generated from the same trajectory that defines the target task, but the paper explicitly frames this as intended replay behavior ('Given a single demonstration, the agent extracts step-by-step instructions and executes them by strictly following the trajectory intended by the user'), not as generalization to unseen tasks. This is a design property, not a circular reduction. Self-citations (e.g., Jang et al. 2025, Bonatti et al. 2024, WinClick) appear only in related-work positioning and are not load-bearing for the central claim. The skeptical concern about unvalidated demonstration quality (alignment and task completion) is a correctness or evaluation-quality risk, not a circularity risk. Overall, the paper is self-contained against an external benchmark and does not reduce its claims to its inputs by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central result rests on unverified success and time-alignment of human demonstrations, on LLM instruction generation, and on verifier reliability. No mathematical constants or physical entities are introduced; the named modules are software components built from existing LLM and grounding APIs.

free parameters (1)
  • Backtracker recovery attempt limit
    Section 3.2.2 limits recovery attempts but gives no value; the success rate likely depends on this hand-chosen threshold.
axioms (4)
  • domain assumption Human demonstrations are successful and each recorded action is exactly time-aligned with its preceding screenshot.
    Stated in Section 3.2.1 but no validation is reported; the entire instruction set is derived from these state-action pairs.
  • domain assumption Screenshots plus keyboard and mouse event logs are a sufficient observation for instruction generation; accessibility trees and HTML are not needed.
    Section 3.2.1 explicitly chooses not to use accessibility trees or HTML, assuming visual information and event logs are enough.
  • domain assumption GPT-4o can generate accurate, location-aware step descriptions from annotated screenshots and action logs.
    The Instructor module's quality is asserted, not measured independently of end-to-end task success.
  • domain assumption The verifier LLM can judge action success by comparing before and after screenshots, including for novel UI elements.
    Section 3.2.2 makes this the gate for advancing or backtracking; the paper reports verification errors still occur in Section 4.3.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction Agent: Enhancing Agent with Expert Demonstration." pith.science (2026). https://pith.science/paper/WS7DCTGE

@misc{pith2026250907098,
  author       = {Pith},
  title        = {Pith review of: Instruction Agent: Enhancing Agent with Expert Demonstration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WS7DCTGE}},
  note         = {Machine review of arXiv:2509.07098}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Graphical user interface (GUI) agents have advanced rapidly but still struggle with complex tasks involving novel UI elements, long-horizon actions, and personalized trajectories. In this work, we introduce Instruction Agent, a GUI agent that leverages expert demonstrations to solve such tasks, enabling completion of otherwise difficult workflows. Given a single demonstration, the agent extracts step-by-step instructions and executes them by strictly following the trajectory intended by the user, which avoids making mistakes during execution. The agent leverages the verifier and backtracker modules further to improve robustness. Both modules are critical to understand the current outcome from each action and handle unexpected interruptions(such as pop-up windows) during execution. Our experiments show that Instruction Agent achieves a 60% success rate on a set of tasks in OSWorld that all top-ranked agents failed to complete. The Instruction Agent offers a practical and extensible framework, bridging the gap between current GUI agents and reliable real-world GUI task automation.

Figures

Figures reproduced from arXiv: 2509.07098 by Hailey Hultquist, Justin Wagle, Kazuhito Koishida, Yinheng Li.

Figure 1
Figure 1. Figure 1: Instructor–Actor Agent reliability of each action execution, we added verification and backtracking modules. Specifically, our framework comprises the following components ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example of a non-intuitive UI AxureBoutique [2023]. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An overview of Instruction Agent Although screen recording videos and tutorials are widely available online, most cannot be directly leveraged for instruction generation, as they typically do not capture the user’s input events. Even when user inputs are visible in some recordings (such as those demo videos provided by Xie et al. [2024]), these inputs are usually not precisely aligned with the correspondin… view at source ↗
Figure 4
Figure 4. Figure 4: Instruction Generation 5 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages

  1. [1]

    Agent s2: A compositional generalist-specialist framework for computer use agents, 2025

    Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents, 2025. URL https://arxiv.org/abs/2504.00906

    Pith/arXiv arXiv 2025

  2. [2]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku

    Anthropic PBC . Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https://www.anthropic.com/news/3-5-models-and-computer-use, October 2024. Accessed 2 July 2025

    work page 2024

  3. [3]

    Navigating the maze: Examples of bad navigation in ui/ux

    AxureBoutique . Navigating the maze: Examples of bad navigation in ui/ux. https://www.youtube.com/watch?v=D-RMsyZrt38, September 2023. YouTube video, accessed 2 July 2025

    work page 2023

  4. [4]

    Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL https://arxiv.org/abs/2409.08264

    Pith/arXiv arXiv 2024

  5. [5]

    Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024. URL https://arxiv.org/abs/2401.10935

    Pith/arXiv arXiv 2024

  6. [6]

    Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025. URL https://arxiv.org/abs/2410.05243

    Pith/arXiv arXiv 2025

  7. [7]

    Efficient agent training for computer use, 2025

    Yanheng He, Jiahe Jin, and Pengfei Liu. Efficient agent training for computer use, 2025. URL https://arxiv.org/abs/2505.13909

    arXiv 2025

  8. [8]

    Winclick: Gui grounding with multimodal large language models, 2025

    Zheng Hui, Yinheng Li, Dan zhao, Tianyi Chen, Colby Banbury, and Kazuhito Koishida. Winclick: Gui grounding with multimodal large language models, 2025. URL https://arxiv.org/abs/2503.04730

    Pith/arXiv arXiv 2025

  9. [9]

    Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P

    Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, and Graham Neubig. Cowpilot: A framework for autonomous and human-agent collaborative web navigation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demons...

    doi:10.18653/v1/2025.naacl-demo.17 2025

  10. [10]

    Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2025

    Lawrence Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2025. URL https://arxiv.org/abs/2410.19100

    Pith/arXiv arXiv 2025

  11. [11]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL https://arxiv.org/abs/2401.13649

    Pith/arXiv arXiv 2024

  12. [12]

    On the effects of data scale on ui control agents, 2024

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents, 2024. URL https://arxiv.org/abs/2406.03679

    Pith/arXiv arXiv 2024

  13. [13]

    Magentic-ui: Towards human-in-the-loop agentic systems, 2025

    Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, Eric Zhu, Griffin Bassman, Jacob Alber, Peter Chang, Ricky Loynd, Friederike Niedtner, Ece Kamar, Maya Murad, Rafah Hosn, and Saleema Amershi. Magentic-ui: Towards human-in-the-loop agentic s...

    Pith/arXiv arXiv 2025

  14. [14]

    Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild, 2025. URL https://arxiv.org/abs/2410.02907

    Pith/arXiv arXiv 2025

  15. [15]

    Test automation for windows gui application

    Juho-Jaakko Oksanen. Test automation for windows gui application. Bachelor's thesis, Oulu University of Applied Sciences, Oulu, Finland, 2023. URL https://www.theseus.fi/handle/10024/801926

    work page 2023

  16. [16]

    Introducing operator

    OpenAI . Introducing operator. https://openai.com/index/introducing-operator/, January 2025. Accessed 2 July 2025

    work page 2025

  17. [17]

    Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou

    Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale, 2024. URL https://arxiv.org/abs/2409.15637

    Pith/arXiv arXiv 2024

  18. [18]

    Clueweb22: 10 billion web documents with visual and semantic information, 2022

    Arnold Overwijk, Chenyan Xiong, Xiao Liu, Cameron VandenBerg, and Jamie Callan. Clueweb22: 10 billion web documents with visual and semantic information, 2022. URL https://arxiv.org/abs/2211.15848

    Pith/arXiv arXiv 2022

  19. [19]

    Ui-tars: Pioneering automated gui interaction with native agents, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Ya...

    Pith/arXiv arXiv 2025

  20. [20]

    Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations, 2025

    Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations, 2025. URL https://arxiv.org/abs/2412.01441

    Pith/arXiv arXiv 2025

  21. [21]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025. URL https://arxiv.org/abs/2412.19723

    Pith/arXiv arXiv 2025

  22. [22]

    Python for automation and scripting: Streamlining operations and increasing efficiency

    Preeti Tupsakhare. Python for automation and scripting: Streamlining operations and increasing efficiency. The Journal of Scientific and Engineering Research, pages 222--227, 09 2019. doi:10.5281/zenodo.13918609

    work page doi:10.5281/zenodo.13918609 2019

  23. [23]

    Gui agents with foundation models: A comprehensive survey, 2025

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. Gui agents with foundation models: A comprehensive survey, 2025. URL https://arxiv.org/abs/2411.04890

    Pith/arXiv arXiv 2025

  24. [24]

    Os-atlas: A foundation action model for generalist gui agents, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv.org/abs/2410.23218

    Pith/arXiv arXiv 2024

  25. [25]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972

    Pith/arXiv arXiv 2024

  26. [26]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025. URL https://arxiv.org/abs/2412.09605

    Pith/arXiv arXiv 2025

  27. [27]

    Ufo: A ui-focused agent for windows os interaction

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Qingwei Lin Kang, Yu and, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Ufo: A ui-focused agent for windows os interaction. In NAACL'25, pages 597--622, April 2025. URL https://www.microsoft.com/en-us/research/publication/ufo-a-ui-focused-agent-for-windows-os-interaction/

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.