Pith. sign in

REVIEW 2 major objections 7 minor 5 cited by

ScaleTrack: Scaling and back-tracking Automated GUI Agents

T0 review · 2 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that a GUI agent trained to predict both the previous and the next action from the current screen becomes a better planner, and that merging grounding data from complementary sources improves visual grounding.

desk verdict The grounding-data scaling is a solid, useful contribution; the back-tracking objective is a neat idea but the paper's headline planning gains are within sampling noise and need stronger evidence. read the letter →

arxiv 2505.00416 v1 pith:DIJBDTCQ submitted 2025-05-01 cs.AI

Jing Huang , Zhixiong Zeng , Wenkang Han , Yufeng Zhong , Liming Zheng , Shuai Fu , Jingyuan Chen , Lin Ma This is my paper · ORCID
classification cs.AI
keywords GUIagentsvisualgroundingback-trackingactionplanningdatascalingmultimodallargelanguagemodelsAndroidControl
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ScaleTrack is a training recipe for visual GUI agents that does two things: it merges many existing grounding datasets built under different annotation schemes into one template, and it adds a back-tracking objective that asks the model to predict the action immediately before the current screenshot while it predicts the next action. The paper's central claim is that back-tracking makes agents better planners, not just better grounders, and that scaling grounding data with complementary synthesis criteria improves generalization. On the grounding benchmark it reports 86.8 percent average, above prior open-data models; on offline planning benchmarks it reports step-success improvements of 1.8 points on AndroidControl-High and 0.7 points on GUI Odyssey from back-tracking alone. If true, the contribution is simple: a data-only way to improve GUI agents without extra human annotation or online interaction.

What carries the argument

The load-bearing device is Equation (2), the joint forward/backward prediction objective $a_{n-1}, a_n = M_\theta(\text{task}, (o_1,a_1),\dots,(o_{n-1},a_{n-1}), o_n)$, which turns every trajectory step into two training targets from the same state: the next action and the action that produced the current state. The other machinery is the unified grounding template: relative point coordinates mapped to the 0-to-1 range and a fixed instruction format that lets grounding samples from different sources be merged into multi-turn conversations.

What would settle it

Construct or sample trajectories in which the same screenshot is reachable by two different previous actions under the same instruction, train on back-tracking labels, and test whether the step-success gain disappears or reverses relative to no-backtracking; if the gain survives only on unambiguous states, then ambiguity is the limiting factor. A simpler check: measure the fraction of states in the training data whose predecessor action is not uniquely determined by the state and compare it with the size of the ablation gain.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a GUI agent trained to reconstruct the previous action from the current screenshot learns a better correspondence between screens and actions, and that this transfers to improved step-success on held-out tasks. At each step the model receives the current screenshot, task instruction, and previous actions, and is asked to output both the next action and the previous n(1) action. Training on this joint objective improved step success on AndroidControl-High from 76.1 to 77.9 and on GUI Odyssey from 64.6 to 65.3, while leaving AndroidControl-Low unchanged; it also improved action-type accuracy on all three offline sets. The grounding-stage result is the complementary claim: when grounding data created under different synthesis criteria are merged and unified into point coordinates, the resulting model reaches 86.8 average on ScreenSpot, higher than any model trained only on open-source data.

Load-bearing premise

Given only the current screenshot and the task instruction, the single previous action that led to this screen is assumed to be uniquely recoverable; in real trajectories many different histories can end at the same screen, so some back-tracking labels are arbitrary or noisy.

Editorial extensions

If this is right

  • Back-tracking requires no new data collection: any existing trajectory dataset can be transformed by pairing each state with its predecessor's action, so the recipe is directly reusable.
  • Agents trained with back-tracking are stronger at predicting action types even when step success does not move, suggesting the signal sharpens action understanding.
  • Merging complementary grounding synthesis criteria beats each isolated criterion, especially on icons and widgets, so pooling open data is a cheap way to improve grounding.
  • Consistency between the form used to describe historical actions in training and at test time matters; mismatches cost several points in step success.
  • The grounding scaling curve is still rising at the end of training, implying larger merged datasets would likely keep helping.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ablation gains are small and concentrated in high-level tasks, back-tracking may act mainly as a regularizer that discourages the model from ignoring state history; an explicit test would be whether the gain grows with trajectory length or task difficulty.
  • The paper leaves the ambiguity of the previous-action label unexamined: multiple histories can reach the same screen, so the target in Equation (2) is not always uniquely defined. One testable extension is to weight back-tracking loss by how often the predecessor action appears in the data, or to sample among compatible histories.
  • Back-tracking could be extended to predicting several previous actions (n(k)) or to contrastive learning between compatible and incompatible histories; the current design uses only one step, which may under-use the signal.
  • If the back-tracking objective becomes standard, evaluation should control for the expression form of historical actions, since the paper shows train/test mismatch alone costs accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The manuscript describes ScaleTrack, a two-stage training recipe for vision-based GUI agents built on Qwen2-VL-7B. In the grounding stage, the authors merge several public GUI grounding datasets (Uground, OS-Atlas, Aria-UI, Aguvis) into a unified relative-coordinate format and train a grounding model. In the planning stage, they augment Aguvis trajectory data with a "back-tracking" objective that asks the model to predict the immediately previous action as well as the next action from the current screenshot, task instruction, and prior-action context. Evaluation covers grounding on ScreenSpot, offline planning on AndroidControl and GUI Odyssey, and online task success on AndroidWorld and MobileMiniWob. The main claimed contributions are improved grounding from data scaling and improved planning from back-tracking.

Significance. If the reported gains hold, the contribution is a modest but useful data-centric recipe: it demonstrates that combining existing open grounding corpora improves ScreenSpot accuracy (86.8 vs. 84.4 for Aguvis-7B), and it proposes a novel planning objective. The grounding-stage result is reasonably supported by the comparison tables, and the use of public datasets with a planned release is a strength. However, the headline back-tracking claim is not yet established: as written, the back-tracking target may be readable directly from the input context, the ablation differences are within sampling noise, and grounding accuracy drops in the ablation. These issues are fixable within the manuscript's scope, but they need to be addressed before the central contribution can be accepted.

major comments (2)
  1. [Section 3.3, Eq. (2)] The back-tracking objective may be testing token copying rather than historical inference. Equation (2) conditions the model on (o1,a1),...,(o_{n-1},a_{n-1}) and then asks it to predict a_{n-1} together with a_n; the Figure 2 prompt likewise lists "Previous actions" in the input. Under this formulation the model can read the target a_{n-1} directly from its context, so the reported gains do not demonstrate that predicting an unobserved history helps planning. Please mask the target action from the input (or remove the previous-action field during back-tracking training) and re-run the ablation; if the gains disappear, the headline claim is an artifact of the input format rather than evidence for back-tracking.
  2. [Section 4.5, Table 7] The central ablation is underpowered as reported. The gains are +1.8 points on AndroidControl-High and +0.7 points on GUI Odyssey from single runs, with no confidence intervals, bootstrap estimates, or multiple seeds. For a binomial rate of 76-78% on n=800, the standard error is roughly 1.5 points, so the +1.8 difference is about 1.2 SE; for n=500, +0.7 is under 0.5 SE. Grounding accuracy simultaneously drops on all three datasets (84.9 vs. 85.6, 72.8 vs. 73.7, 69.3 vs. 69.4), which is not discussed. The comparison is also confounded by output format: the back-tracking model emits extra tokens that the "w/o back-tracking" model does not. Please report variances, run multiple seeds, and separate the format effect from the back-tracking signal.
minor comments (7)
  1. [Throughout] There are several typos and grammatical errors, including "Futhermore" in Section 1, "forward-planningg" in Section 1, "containts" in Section 4.4, and "CaleTrack-7B" in Section 4.4; the abstract's opening sentence should be "Automated GUI agents aim to...".
  2. [Table 1] The 'Ours' row is ambiguous: "Ours Ours * 7.5M * 8.2" does not clearly indicate which columns refer to grounding elements, screenshots, or multistep traces; clarify the meaning of the asterisks and the column alignment.
  3. [Figure 2 and Section 3.3.2] The notation "previous n(1) action" is not defined; state explicitly whether n is fixed at 1 throughout and how the objective would generalize to longer historical horizons.
  4. [Section 4.3, Table 5] The four Train-Test settings (Instruction-Action, Action-Action, Action-Instruction, Instruction-Instruction) are not described in the text; explain what "Instruction" and "Action" mean as descriptions of previous actions and how they map to the training data and the AndroidControl evaluation format.
  5. [Section 4.4] The online setup uses GPT-4o as the planner and ScaleTrack-7B only for grounding, so the sentence "data-scaling and back-tracking help the model handle diverse element descriptions" is not directly supported; the comparison isolates the grounding model, not the back-tracking objective.
  6. [Data Availability] The abstract and conclusion state "Data and code will be available at url," but no URL is provided; please include the repository link or state the intended publication venue for the artifacts.
  7. [Tables 2 and 4] In Table 2 the phrase "clearly surpasses... baselines" is too strong for single-run comparisons without variance estimates; in Table 4, Aguvis-7B is listed with '-' on GUI Odyssey, so the reader cannot see why this public-data baseline is absent.

Circularity Check

1 steps flagged · score 6.0 of 10

Back-tracking objective as written is self-definitional: the target previous action a_{n-1} is already in the model's input history, so the claimed 'prediction of past actions' reduces to copying context; the external benchmark claims remain independent.

  1. self definitional [Section 3.3.1, Eq. (2); Figure 2 prompt template]
    "ScaleTrack not only predicts the next action under the current overall goal but also predicts the historical actions that led to the current state. This can be formulated as follows: an−1,an =M θ(task,(o 1,a 1),...,(o n−1,an−1),on)(2) ... Please generate the next move, next action and the previous n(1) action according to the previousUI screenshot <image> , current UI screenshot <image> , instruction and previous actions."

    By Eq. (2), the model conditions on (o_{n-1}, a_{n-1}) in the input history and is asked to output a_{n-1} as the back-tracking target. The 'previous action' to be predicted is therefore already supplied as an input token; the objective is a copying/recall task rather than inference of the history from the current GUI image. The paper's motivating claim that back-tracking 'explains the correspondence between GUI images and actions' is satisfied by construction from the prompt's 'previous actions', not by recovering a_{n-1} from the screen. This makes the distinctive back-tracking training signal an input/output tautology: the reported planning gains cannot be attributed to historical inference as opposed to multi-task or formatting effects.

full rationale

The paper's two empirical claims are otherwise self-contained against external benchmarks: grounding results on ScreenSpot (Table 2) are compared with public-data baselines, and the planning ablations (Table 7) are evaluated on AndroidControl and GUI Odyssey; no fitted parameter is renamed as a prediction, and there is no load-bearing self-citation chain. However, Eq. (2) defines the back-tracking target a_{n-1} as part of the same conditioning sequence (o_{n-1}, a_{n-1}) used in Eq. (1), so the formal formulation reduces the back-tracking prediction to re-emitting a token already in context. If the actual data template withholds that token, then Eq. (2) is internally inconsistent with Figure 2; either way, the written derivation is definitionally circular on the paper's headline contribution. Additional concerns about single-run ablations and missing error bars are correctness/statistical-support issues, not circularity, and are therefore not counted as circular steps.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ledger is light: the paper depends on public datasets and benchmarks, with one hand-chosen design parameter (n_previous_actions) and a set of domain assumptions about the validity of the training signal.

free parameters (1)
  • n_previous_actions = 1
    The back-tracking template in Figure 2 predicts the previous n(1) action, i.e., only the immediately preceding action. This hand-chosen value determines the amount of back-tracking supervision; the paper does not ablate n.
assumptions (4)
  • domain assumption The GUI interaction is a partially observable Markov decision process as in Eq. 1.
    The formulation an = M_theta(task, (o1,a1),...,(on-1,an-1), on) assumes the environment is Markovian conditional on history, which is standard but unproven for arbitrary GUI apps.
  • ad hoc to paper Back-tracking labels are well-posed: the previous action is uniquely determined by the current state and task.
    Eq. 2 and Figure 2 train the model to output a single previous action; the paper never discusses that multiple histories can lead to the same screenshot.
  • domain assumption Merging grounding datasets from different synthesis criteria into a unified point-coordinate format preserves supervision value.
    Section 3.2 converts bounding boxes to points and merges multiple QA pairs per screenshot; this assumes no harmful format mismatch between Uground, Aria-UI, Aguvis, OS-Atlas.
  • domain assumption The benchmarks (ScreenSpot, AndroidControl, GUI Odyssey, AndroidWorld, MobileMiniWob) are valid proxies for GUI agent performance.
    All conclusions are drawn from these benchmarks; the paper does not validate on real human tasks or other metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScaleTrack: Scaling and back-tracking Automated GUI Agents." pith.science (2026). https://pith.science/paper/DIJBDTCQ

@misc{pith2026250500416,
  author       = {Pith},
  title        = {Pith review of: ScaleTrack: Scaling and back-tracking Automated GUI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIJBDTCQ}},
  note         = {Machine review of arXiv:2505.00416}
}
read the original abstract

Automated GUI agents aims to facilitate user interaction by automatically performing complex tasks in digital environments, such as web, mobile, desktop devices. It receives textual task instruction and GUI description to generate executable actions (\emph{e.g.}, click) and operation boxes step by step. Training a GUI agent mainly involves grounding and planning stages, in which the GUI grounding focuses on finding the execution coordinates according to the task, while the planning stage aims to predict the next action based on historical actions. However, previous work suffers from the limitations of insufficient training data for GUI grounding, as well as the ignorance of backtracking historical behaviors for GUI planning. To handle the above challenges, we propose ScaleTrack, a training framework by scaling grounding and backtracking planning for automated GUI agents. We carefully collected GUI samples of different synthesis criterions from a wide range of sources, and unified them into the same template for training GUI grounding models. Moreover, we design a novel training strategy that predicts the next action from the current GUI image, while also backtracking the historical actions that led to the GUI image. In this way, ScaleTrack explains the correspondence between GUI images and actions, which effectively describes the evolution rules of the GUI environment. Extensive experimental results demonstrate the effectiveness of ScaleTrack. Data and code will be available at url.

Figures

Figures reproduced from arXiv: 2505.00416 by the authors.

Figure 1
Figure 1. Difference of forward-planning and back-tracking. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overall description of our proposed ScaleTrack in processing task instruction and generating [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Scaling curve of ScaleTrack-7B on ScreenSpot. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  2. Software Engineering for and with GUI Agent

    cs.SE 2026-08 conditional novelty 5.0 of 10

    A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.

  3. OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A generalist agent with shared shallow layers and task-separated deep experts outperforms single-domain GUI and embodied agents on AndroidControl, GUI-Odyssey, and LIBERO benchmarks.

  4. FADE: Adversarial Concept Erasure in Flow Models

    cs.CV 2025-07 reject novelty 4.0 of 10

    FADE combines adversarial training with trajectory preservation to erase concepts from diffusion models, reporting state-of-the-art erasure on Stable Diffusion benchmarks, but the evidence is incomplete and the theore...

  5. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

Reference graph

Works this paper leans on

21 extracted references · 21 linked inside Pith · cited by 5 Pith papers

  1. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609,

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

    arXiv

  2. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    arXiv

  3. [5]

    Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243,

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243,

    arXiv

  4. [6]

    Is your llm secretly a world model of the internet? model-based planning for web agents.arXiv preprint arXiv:2411.06559,

    Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your llm secretly a world model of the internet? model-based planning for web agents.arXiv preprint arXiv:2411.06559,

    arXiv

  5. [7]

    The dawn of gui agent: A preliminary case study with claude 3.5 computer use.arXiv preprint arXiv:2411.10323,

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use.arXiv preprint arXiv:2411.10323,

    arXiv

  6. [8]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    arXiv

  7. [9]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a. Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ...

    arXiv

  8. [10]

    Laser: Llm agent with state-space exploration for web navigation.arXiv preprint arXiv:2309.08172,

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Llm agent with state-space exploration for web navigation.arXiv preprint arXiv:2309.08172,

    arXiv

Show all 21 references
  1. [11]

    Agent q: Advanced reasoning and learning for autonomous ai agents.arXiv preprint arXiv:2408.07199,

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents.arXiv preprint arXiv:2408.07199,

    arXiv

  2. [12]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326,

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326,

    arXiv

  3. [13]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573,

    11 Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573,

    arXiv

  4. [14]

    Beyond browsing: Api-based web agents.arXiv preprint arXiv:2410.16464,

    Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents.arXiv preprint arXiv:2410.16464,

    arXiv

  5. [15]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    arXiv

  6. [16]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a. Weihan Wang, Qingsong Lv,...

    arXiv

  7. [17]

    Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454,

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454,

    arXiv

  8. [18]

    Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256,

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256,

    arXiv

  9. [19]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704,

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704,

    arXiv

  10. [21]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control.arXiv preprint arXiv:2306.07863,

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control.arXiv preprint arXiv:2306.07863,

    arXiv

  11. [2023]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  12. [2024]

    Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935,

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935,

    arXiv

  13. [2025]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614,

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.