Pith. sign in

REVIEW 4 major objections 4 minor 106 references

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By training reflection at every stage—verifying actions, reversing mistakes, and reattempting—an 8B multimodal GUI agent learns to catch its own errors, raising level-2 task success from 14.58% to 34.72% in the authors' environment and…

desk verdict A well-engineered three-stage reflection pipeline with credible internal ablations, but the external validation is a single modified-protocol AndroidWorld number without error bars. read the letter →

arxiv 2506.08012 v1 pith:A5U26OZ2 submitted 2025-06-09 cs.AI cs.CV

Penghao Wu , Shengnan Ma , Bo Wang , Jiaheng Yu , Lewei Lu , Ziwei Liu This is my paper
classification cs.AIcs.CV
keywords GUIautomationself-reflectionerrorcorrectionmultimodallargelanguagemodelsmobileagentsiterativeonlinetuningactionverificationbehaviorcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI-Reflection claims that mistake-handling can be trained into an end-to-end GUI agent rather than left to prompt-engineered scaffolds or scale. The paper shows that an 8B multimodal model can learn to recognize its own wrong actions, undo them, and retry, by making reflection an explicit objective at every training stage: a pre-training task suite for verification, reversal, and reattempt skills; offline supervised fine-tuning on automatically generated error scenarios; and iterative online tuning with pre-error and post-error correction labels. On the authors' 215-task Android emulator, adding reflection data to offline SFT raises level-2 success from 14.58% to 23.61%, and adding online reflection tuning raises it further to 34.72%. On the external AndroidWorld benchmark the final model reaches 34.5% success, comparable to the best end-to-end baselines at this scale. The paper's core suggestion is that reflection is a learnable behavior for GUI agents, and that standard training on error-free trajectories actively suppresses it.

What carries the argument

The load-bearing mechanism is the three-step reflection loop operationalized as training targets: (1) verify the previous action by inspecting the outcome screenshot, (2) undo the incorrect action (often Press Back) to restore the pre-mistake state, and (3) summarize the mistake and issue a corrected action from the restored state. The paper encodes this loop as concrete supervision: the GUI-Reflection Task Suite (Action Verification, Action Reversal, Mistake-informed Reattempt) for pre-training, automatically constructed reflection samples for offline SFT, and the iterative online reflection-tuning algorithm whose 'pre-error correction' and 'post-error reflection' annotations replicate the same loop during online rollouts. The model is an InternVL2.5-8B backbone that outputs an action thought, a natural-language action description, and a grounded atomic action, with a history of up to four screenshots, a memory bank, and an action history as context.

What would settle it

Train the full pipeline but evaluate on an independent held-out set of Android tasks drawn from a different distribution (for example, the AndroidControl test split or a third-party benchmark not used in training), and compare against a filtered-behavior-cloning model of equal size; if the success-rate gap collapses outside the in-house environment, the reflection skills are environment-specific rather than general. Additionally, an independent human audit of a random sample of rollouts judged successful by the programmatic and MLLM verifiers would test whether the reward signal itself is trustworthy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that self-reflection and error recovery can be installed into an end-to-end multimodal GUI model through three coordinated training phases. The authors decompose reflection into three atomic abilities—Action Verification (did my last action achieve its purpose?), Action Reversal (what action undoes the mistake?), and Mistake-informed Reattempt (given the known error, what is a better next attempt?)—and train each during GUI pre-training. They then generate offline SFT data containing deliberately injected mistakes: tasks whose goal is perturbed so a recorded action becomes wrong, and ineffective no-op actions inserted into otherwise successful trajectories, with an MLLM writing the post-error reflection. Finally, an iterative online reflection-tuning loop rolls the model out in a 215-template Android environment, keeps the correct prefixes of successful trajectories, and annotates the first error step of failed trajectories with a pre-error correction and a post-error reflection; the model is fine-tuned on this mixture, with sampling reweighted toward harder tasks each iteration. The measured result is that each stage contributes: reflection data in offline SFT adds roughly nine points on level-2 tasks, and the online loop adds roughly eleven more, for a final level-2 success rate of 34.72%.

Load-bearing premise

The load-bearing premise is that the automatically generated error annotations and the 215-task in-house emulator are correct and representative enough that reflection skills learned from them transfer to tasks outside that environment, such as AndroidWorld; if the emulator tasks are too easy, too repetitive, or the MLLM annotations too noisy, the measured gains may not generalize.

Editorial extensions

If this is right

  • A GUI agent trained this way can recognize a mis-click from the resulting screen and press back before continuing, so a single mistake no longer derails a long task.
  • The pipeline runs without human annotation: MLLMs generate the error scenarios, the undo actions, and the reflection thoughts, so the approach scales to new apps by adding task templates to the environment.
  • Standard GUI pre-training and offline SFT on error-free trajectories degrade the base model's verification ability, so reflection must be explicitly trained for rather than assumed to survive fine-tuning.
  • At 34.5% on AndroidWorld, an 8B reflection-trained model lands in the same range as much larger end-to-end baselines and narrows the gap to agent-based systems that use accessibility trees.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer assumption holds, the same three-stage recipe should extend to desktop and web interfaces by swapping in domain-specific screenshots and action spaces; the paper notes this adaptation is not yet demonstrated.
  • The GUI-Reflection Task Suite could serve as a standalone diagnostic before deployment: an agent's verification and reversal scores in pre-training predict whether it will recover from mistakes in online use.
  • A clean ablation would separate the two online annotations: keeping only post-error reflections while dropping pre-error corrections would show whether the gain comes from recognizing mistakes or from learning to backtrack.
  • Because the online loop reweights sampling toward unsolved tasks, adding a policy-gradient objective on top of the programmatic verifier rewards might push success rates further without requiring new data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces GUI-Reflection, a training framework that adds self-reflection and error recovery to end-to-end multimodal GUI agents built on InternVL2.5-8B. The framework has three components: a GUI-Reflection Task Suite consisting of Action Verification, Action Reversal, and Mistake-informed Reattempt pre-training tasks; an automatic pipeline that converts existing successful GUI trajectories into offline SFT data with injected mistakes and reflective corrections; and an online Android emulator environment with 215 task templates together with an iterative online reflection tuning algorithm. The central empirical claims are that standard GUI pre-training and offline SFT suppress reflection abilities, that the proposed task suite restores them (Table 1), and that adding reflection data in SFT plus online reflection tuning raises level-2 task success from 14.58% to 23.61% to 34.72% in the authors' environment (Table 2), with a 34.5% success rate on AndroidWorld (Table 3) that is competitive with UI-TARS-7B.

Significance. If the results hold, the paper makes a useful contribution: it decomposes GUI reflection into concrete, trainable capabilities, shows that standard GUI pre-training can degrade these abilities, and provides a fully automatic pipeline (no human annotation) spanning pre-training, SFT, and online tuning. The task suite itself is a valuable diagnostic instrument, and the promise to release data, models, environment, and tools is a strong reproducibility asset. The internal ablations in Table 2 support the claim that reflection-specific data improves performance on the authors' own environment. The main weakness is external validity: the only external benchmark result, AndroidWorld in Table 3, is obtained under a modified evaluation protocol with no rerun of baselines under that protocol, and no error bars or multiple seeds are reported. The paper is honest about its scope (Appendix B notes it focuses on visual/action-grounded errors and on mobile environments), which tempers but does not eliminate the transfer concern.

major comments (4)
  1. [§E.2, Table 3] The AndroidWorld evaluation is not a controlled comparison. The protocol changes described in §E.2 (scroll trajectory follows the model's predicted start/end points, type no longer clicks the target element, and maximum steps increased by 5 for all test cases) are applied to the authors' model, but the published baselines (UI-TARS-7B at 33.0, Aguvis-7B at 26.1, etc.) are not re-run under the same modified protocol. Since the reported margin over UI-TARS-7B is only 1.5 points, the 34.5% success rate cannot be attributed to reflection tuning as opposed to the protocol changes. The authors should either re-run baselines under the modified protocol or report results under both the original and modified protocols.
  2. [§3.3, Table 2, Fig. 5] No error bars, confidence intervals, or information about the number of evaluation runs/seeds are reported for any success-rate numbers, including the central Table 2 progression (14.58%, 23.61%, 34.72%) and the Fig. 5 iterative curves. In a dynamic emulator environment, single-run estimates can easily vary by several points, so the reported differences, and especially the 1.5-point AndroidWorld margin, need variance estimates or multiple seeds to be statistically meaningful.
  3. [§2.5 and §3.3] The main ablation is measured in the same 11-app environment used for online training, and the level-2 evaluation tasks share apps and UI layouts with the level-1 training tasks. Because the final model is additionally fine-tuned on data collected in this environment, the observed gains could reflect environment-specific overfitting rather than a general reflection capability. The paper should include a held-out set of apps or tasks, or evaluate on an external benchmark under an unchanged protocol, to support the transfer claim.
  4. [§2.4 and §2.5.2] The reflection and correction training labels are generated automatically by MLLMs (Gemini-2.5-Pro, GPT-4o) and by the model's own grounding outputs, with no human verification or inter-annotator agreement reported for the generated training data. Since the central claim is that these data teach genuine mistake recognition and recovery, the paper should report a human evaluation of a random sample of generated reflection data, including the rate of incorrect, ambiguous, or unhelpful annotations. This is particularly important because the same MLLMs are also used to verify trajectory step correctness in the online loop.
minor comments (4)
  1. [Appendix F.3] The sentence 'we evaluate directly on ScreenSpot [14] and ScreenSpot [43]' should read 'ScreenSpot [14] and ScreenSpotv2 [43]' to match the table headings in Table 1b.
  2. [§3.3] The baseline name 'Filtered BC' is used in Table 2 without a definition in the main text; it should be explicitly defined (e.g., behavior cloning on step-wise filtered successful trajectories) at first use.
  3. [Fig. 5] The axes of Fig. 5 are not labeled; add an x-axis label (iteration) and y-axis label (success rate) and clarify whether these curves are from the same runs reported in Table 2 or from separate evaluation runs.
  4. [Table 1b] The reattempt results (2nd and 3rd attempts) do not specify the decoding temperature, while pass@3 is reported at temperature=1.0; please state the temperature used for the reattempt evaluations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the reflection-training claims are tested on held-out level-2 tasks, the external ScreenSpot benchmark, and the independent AndroidWorld benchmark, with no self-citation chain; the main unresolved issues are external-validity and protocol matters, not by-construction circularity.

full rationale

The paper's derivation chain — (i) reflection-oriented pre-training tasks, (ii) reflection-augmented offline SFT, (iii) iterative online reflection tuning — is not circular in the by-construction sense: no metric or equation used as evidence is defined in terms of the quantity it is claimed to establish. The pre-training claim (Table 1) is anchored on external benchmarks: Mistake-informed Reattempt is evaluated on the public ScreenSpot and ScreenSpotv2 grounding benchmarks, where the reflection-trained model (GUI-Pretrain-Ref) exceeds pass@3 on 3rd-attempt accuracy, an external checkpoint that cannot be produced by the authors' own evaluation data. The offline-SFT and online-tuning claims (Table 2) are measured on level-2 tasks held out from the level-1 training used in all three rows, so the 14.58% to 23.61% to 34.72% progression is a behavioral comparison rather than a metric forced by the training labels; the MLLM-annotated corrections are inputs, while the level-2 success rate is an independent measurement. The final claim (Table 3) uses the independent AndroidWorld benchmark. No load-bearing self-citation exists: the reference list contains no works by the present authors, and the key baselines (UI-TARS, OS-ATLAS, SeeClick, Aguvis) are external. The genuine weaknesses are external-validity and protocol issues, which belong under correctness risk rather than circularity: the single AndroidWorld run uses the modified protocol of Appendix E.2 (scroll follows predicted endpoints, type no longer clicks the target, +5 max steps) with baselines not re-run under that protocol; Fig. 5 adds level-2 tasks to the training pool after iteration 3, so its later level-2 curve partly reflects training-set accuracy; Table 2's 34.72% is hard to reconcile with Fig. 5's 29.36% at the same nominal point; and for tasks judged by the MLLM-based verifier, the evaluation reward and the annotation signal come from the same model family (Gemini-2.0-Flash), a verifier-uniformity concern rather than an equivalence of metric to label. The paper's own limitation (Appendix B) concedes that only visual/action-grounded errors are covered, which narrows the claim's scope but does not make it circular.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The framework's central claims rest on several hand-chosen design decisions (history length, resolution, thresholds, training schedules) and on MLLM-generated labels being accurate enough to serve as training signal. The environment is a new artifact, but it is externally checked through AndroidWorld. The number of hidden choices is moderate, and many are unablated.

free parameters (5)
  • screenshot history length n=4 = 4
    Chosen by hand in E.1 without ablation, affects how much context the model sees and could influence reflection behavior.
  • past screenshots downsampled to 448x448 = 448x448
    Chosen in E.1 without ablation; resolution affects grounding and verification accuracy.
  • max steps increased by 5 in AndroidWorld = +5 steps
    E.2 reports this modification to AndroidWorld evaluation for all test cases, justified as needed for agents without UI element info; changes the evaluation protocol relative to published AndroidWorld numbers.
  • curriculum and sampling thresholds = 80% success threshold; level-1 first k iterations
    Chosen by hand in 2.5.2 without sensitivity analysis; could affect the online tuning curve.
  • per-component training epochs and learning rates = pre-train 4e-5, SFT 3e-5, online 1e-5, final 2e-5
    Training hyperparameters fixed without ablations; reasonable defaults but their interaction with the claimed gains is not explored.
assumptions (5)
  • domain assumption Base MLLM InternVL2.5-8B has usable GUI grounding and instruction-following abilities.
    Section 2.1.2 chooses InternVL2.5-8B as the base without comparison to other base models; the central results could depend on this choice.
  • domain assumption Gemini-2.5-Pro and Gemini-2.0-Flash annotations (action purpose, undo action, negative purpose, error analysis, reflection thoughts) are accurate enough to serve as ground truth.
    The entire data pipeline in Sections 2.2, 2.3, 2.4, and Appendix F relies on MLLM-generated labels; only the Action Verification and Action Reversal evaluation sets are human-filtered.
  • domain assumption The 215 task templates across 11 apps constitute a diverse enough online training environment.
    Section 2.5.1 asserts diversity but provides no comparison or evidence of coverage of real-world app distributions.
  • domain assumption Step-wise correctness of successful trajectories judged by Gemini-2.0-Flash is a reliable dense reward.
    Section 2.5.2 and Appendix H use the MLLM verifier to filter training data; a wrong judge would systematically inject incorrect labels.
  • domain assumption Reflection behavior trained on mobile screenshots transfers to the AndroidWorld benchmark tasks.
    The AndroidWorld evaluation in Table 3 is the main external validation; the paper does not ablate the transfer or measure distribution shift.
invented entities (3)
  • GUI-Reflection Task Suite (Action Verification, Action Reversal, Mistake-informed Reattempt) independent evidence
    purpose: Training and evaluation tasks for reflection-oriented capabilities in GUI pre-training.
    The tasks are evaluated against Gemini-2.5-Pro, GPT-4o, and other baselines, providing external comparison, and the underlying capabilities are grounded in ScreenSpot/AndroidControl/GUI-Odyssey benchmarks.
  • Online Android environment with 215 task templates independent evidence
    purpose: Environment for iterative online reflection tuning and data collection.
    Will be released publicly and is used to train a model evaluated on an independent benchmark (AndroidWorld), giving it a falsifiable handle.
  • Action Verification, Action Reversal, Mistake-informed Reattempt as named capabilities
    purpose: Decomposition of reflection into trainable atomic skills.
    The decomposition is a design choice of this paper; there is no external evidence that this specific split is necessary or optimal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior." pith.science (2026). https://pith.science/paper/A5U26OZ2

@misc{pith2026250608012,
  author       = {Pith},
  title        = {Pith review of: GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A5U26OZ2}},
  note         = {Machine review of arXiv:2506.08012}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.

Figures

Figures reproduced from arXiv: 2506.08012 by the authors.

Figure 1
Figure 1. Illustrative comparison of typical GUI models versus our proposed GUI model with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The GUI-Reflection framework includes (1) Learning basic reflection-oriented skills from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of Action Verification (left), Action Reversal (middle), and Mistake-informed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The Iterative Online Reflection Tuning algorithm. It features weighted task sampling, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Success Rate (%) on Level-1 and Level-2 tasks across iterative reflection tun￾ing iterations. Our iterative reflection tun￾ing with curriculum learning strategy pro￾gressively improves model performance. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Illustrative examples of our GUI model’s self-reflection and correction behaviours. The [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Example of the reflection data generated with the first approach in the offline SFT stage. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Example of the reflection data generated with the second approach in the offline SFT stage. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

106 extracted references · 50 canonical work pages

  1. [1]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    arXiv 2024

  2. [2]

    Wave-UI Dataset

    AgentSea Team. Wave-UI Dataset. https://huggingface.co/datasets/agentsea/ wave-ui, 2023

    2023

  3. [3]

    Wave-UI-25K Dataset

    AgentSea Team. Wave-UI-25K Dataset. https://huggingface.co/datasets/agentsea/ wave-ui-25k, 2024

    2024

  4. [4]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet/

  5. [5]

    Screenai: A vision-language model for ui and infographics understanding

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor C˘arbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. InIJCAI, 2024

    2024

  6. [6]

    Uibert: Learning generic multimodal representations for ui understanding

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. InIJCAI, 2021

    2021

  7. [7]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896, 2024

    arXiv 2024

  8. [8]

    Digi-q: Learning q-value functions for training device-control agents.arXiv preprint arXiv:2502.15760, 2025

    Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training device-control agents.arXiv preprint arXiv:2502.15760, 2025

    arXiv 2025

Show all 106 references
  1. [9]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  2. [10]

    Andrea Burns, Kate Saenko, and Bryan A. Plummer. Tell me what’s next: Textual foresight for generic ui representations. InACL Findings, 2024

    2024

  3. [11]

    Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    2024 arXiv

  4. [12]

    Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024

    2024 arXiv

  5. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  6. [14]

    SeeClick: Harnessing GUI grounding for advanced visual GUI agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. InACL, 2024

    2024

  7. [15]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  8. [16]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. InICLR, 2025

    2025

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InCVPR, 2024

    2024

  11. [19]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025. 10

    2025 arXiv

  12. [20]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  13. [21]

    On the effects of data scale on ui control agents

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents. InNeurIPS, 2024

    2024

  14. [22]

    Appagent v2: Advanced agent for flexible mobile interactions.arXiv preprint arXiv:2408.11824, 2024

    Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions.arXiv preprint arXiv:2408.11824, 2024

    2024

  15. [23]

    Widget captioning: Generating natural language description for mobile user interface elements

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. InEMNLP, 2020

    2020

  16. [24]

    Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection.arXiv preprint arXiv:2501.04575, 2025

    Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection.arXiv preprint arXiv:2501.04575, 2025

    2025 arXiv

  17. [25]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  18. [26]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019

    2019

  19. [27]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  20. [28]

    Omniparser for pure vision based gui agent, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024

    2024

  21. [29]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  22. [30]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  23. [31]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023

    2023

  24. [32]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  25. [33]

    An- droidinthewild: A large-scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. InNeurIPS, 2023

    2023

  26. [34]

    Rethinking reflection in pre-training.arXiv preprint arXiv:2504.04022, 2025

    Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training.arXiv preprint arXiv:2504.04022, 2025

    2025 arXiv

  27. [35]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 2020

    2020

  28. [36]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.arXiv preprint arXiv:2412.19723, 2024

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  29. [37]

    Towards better semantic understanding of mobile interfaces

    Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James Stout. Towards better semantic understanding of mobile interfaces. arXiv preprint arXiv:2210.02663, 2022

    2022 arXiv

  30. [38]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 11

    2023 arXiv

  31. [39]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. InNeurIPS, 2024

    2024

  32. [40]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024

    2024 arXiv

  33. [41]

    Distrl: An asynchronous distributed reinforcement learning framework for on-device control agent

    Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye HAO, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agent. In ICLR, 2025

    2025

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022

    2022

  35. [43]

    OS-ATLAS: Foundation action model for generalist GUI agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: Foundation action model for generalist GUI agents. InICLR, 2025

    2025

  36. [44]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  37. [45]

    Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  38. [46]

    Tree of thoughts: Deliberate problem solving with large language models.NeurIPS, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.NeurIPS, 2023

    2023

  39. [47]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  40. [48]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  41. [49]

    Appagent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771, 2023

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771, 2023

    2023 arXiv

  42. [50]

    Android in the zoo: Chain-of-action-thought for GUI agents

    Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for GUI agents. InEMNLP Findings, 2024

    2024

  43. [51]

    Breaking the data barrier – building gui agents through task generalization.arXiv preprint arXiv:2504.10127, 2025

    Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data barrier – building gui agents through task generalization.arXiv preprint arXiv:2504.10127, 2025

    2025 arXiv

  44. [52]

    Gpt-4v(ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. InICML, 2024

    2024

  45. [53]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 12 Acknowledgments ...

    2025 arXiv

  46. [54]

    To obtain the failed attempts in training data, we first train a GUI-Pretrain model with the three target datasets for this task excluded

    datasets and directly evaluate on the grounding benchmark ScreenSpot [ 14] and ScreenSpot [43]. To obtain the failed attempts in training data, we first train a GUI-Pretrain model with the three target datasets for this task excluded. Then we conduct inference on these three d...

  47. [55]

    Two screenshots: the first screenshot corresponds to the current step, and the outcome screenshot corresponds to the next step

  48. [56]

    Current Action: the action executed at the current step

  49. [57]

    new purpose

    Current Action Purpose: The purpose of executing the current action The current action successfully achieved the current purpose (can be seen from the outcome of this action, which is the outcome screenshot). Your task is to come up with a new purpose, such that it is not sati...

  50. [58]

    Task Instruction: The original task instruction for the GUI agent

  51. [62]

    The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screenshot

    Screenshots: The screenshot at the current step and the screenshot at the next step after the current action is executed. The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screensho...

  52. [63]

    Ignore the following if not

    Based on the information, judge the feasibility of having a reasonable instruction meeting the mentioned requirements. Ignore the following if not

  53. [64]

    Provide your modified instruction

  54. [65]

    Explain why the modified instruction is reasonable and realistic, and does not change the correctness of the past actions

  55. [66]

    Task Feasibility

    Explain why the **current action** becomes incorrect with your modified action, and explain why it is an easy or natural mistake a user might make. Note that the red dot and red arrows are just for visualizing the actions; do not mention them in your response. Note that the fu...

  56. [68]

    Empty if it is the first step

    Past Actions: A list of past actions prior to the previous incorrect step. Empty if it is the first step

  57. [69]

    Previous Incorrect Action: The incorrect action taken at the previous step

  58. [71]

    The click point is highlighted using a red dot in the first screenshot

    Screenshots: The first screenshot corresponds to the step before the incorrect action is taken while the second screenshot corresponds to the current step after executing the incorrect action. The click point is highlighted using a red dot in the first screenshot. Your job is ...

  59. [72]

    Analyze the provided information

  60. [73]

    In the action thought, use first-person description, that is, using ’I’ instead of ’the user’ or ’the agent’

    Provide your action thought. In the action thought, use first-person description, that is, using ’I’ instead of ’the user’ or ’the agent’

  61. [74]

    Action Thought

    Provide your action type which must be consistent with your action thought. The correct action type should be one of the following: [Click, Long Press, Scroll, Type, Press Home, Press Back, Press Enter, Wait, Task Complete, Task Impossible] ## Important Notes: Note that the re...

  62. [75]

    Empty if it is the first step

    Past Actions: A list of past actions prior to this step. Empty if it is the first step

  63. [76]

    Only the current screenshot is provided if it is the first step

    Screenshots: The first screenshot corresponds to the previous step while the second screenshot corresponds to the current step. Only the current screenshot is provided if it is the first step. The click point is highlighted using a red dot in the first screenshot. Your job is ...

  64. [77]

    In the action thought, use first-person description, that is, using ’I’ instead of ’the user’ or ’the agent’

    Based on the provided information, provide the correct action thought. In the action thought, use first-person description, that is, using ’I’ instead of ’the user’ or ’the agent’

  65. [78]

    Action Thought

    Provide the correct action type which must be consistent with your action thought. The correct action type should be one of the following: [Click, Long Press, Open App, Scroll, Type, Press Home, Press Back, Press Enter, Wait, Task Complete, Task Impossible] ## Important Notes ...

  66. [80]

    Incorrect Action: The previous incorrect action

  67. [81]

    Error Action Analysis: An analysis from an expert explaining why the previous action is incorrect

  68. [83]

    Updated Action Thought

    Screenshot: The screenshot after performing the press back action to go back to the normal status Your job is to analyze the provided information and modify the correct action thought by adding some reflection and lessons learned about the incorrect action narrated in the firs...

  69. [84]

    Empty if it is the first step

    Past Actions: The past actions taken by the agent to complete this task before this step. Empty if it is the first step

  70. [85]

    Empty if it is the last step

    Future Actions: The future actions taken by the agent to complete this task after this step. Empty if it is the last step

  71. [87]

    The click point is highlighted using a red dot in the current screenshot

    Screenshot: The screenshot at the current step. The click point is highlighted using a red dot in the current screenshot. Your job is to analyze the provided information and decide whether it is possible to come up with an incorrect ineffective action. The requirement of the i...

  72. [88]

    The action is incorrect; it is clearly wrong or unnecessary for completing the task or deviates from the correct way

  73. [89]

    The user will probably realize the mistake when seeing the execution of this action

    It is an easy or natural mistake a user who is unfamiliar with the App, button functions, or certain operations might make. The user will probably realize the mistake when seeing the execution of this action. 27

  74. [90]

    This action has no effect on the current screen, which means the current screen will remain exactly the same after executing the incorrect action. Some examples: scroll down while it is already at the bottom, click the entry name instead of the actual text field for entering i...

  75. [91]

    Ignore the following if not

    Based on the information, judge the feasibility of having an incorrect ineffective action meeting the mentioned requirements. Ignore the following if not

  76. [92]

    Provide your incorrect ineffective action if possible

  77. [93]

    Task Feasibility

    Explain why the incorrect ineffective action is incorrect, and explain why it is an easy or natural mistake a user might make. Note that you have to strictly follow the complete format for your action. ## Input: Task Instruction: {task_instruction} Past Actions: {past_actions}...

  78. [95]

    Error Action Analysis: An analysis from an expert explaining why the previous action is incorrect and why it might happen

    Incorrect Action: The previous incorrect action 3. Error Action Analysis: An analysis from an expert explaining why the previous action is incorrect and why it might happen

  79. [96]

    Correct Action Thought: The actual correct action thought leading to the correct action

  80. [97]

    Updated Action Thought

    Screenshot: The screenshot of the current step after executing the incorrect action Your job is to analyze the provided information and modify the correct action thought by adding some observation and reflection content realizing and acknowledging the incorrect action narrated...

  81. [98]

    2: Task Guidance: An overall description about how to **correctly** complete this task as a reference

    Task Instruction: The original task instruction for the GUI agent. 2: Task Guidance: An overall description about how to **correctly** complete this task as a reference. 3: Correct Answer: The ground truth answer for this task. Empty if the answer is not available or the task ...

  82. [99]

    Operation History: A list of actions (in the form of images and corresponding actions) taken by the agent to execute the Task Instruction. 29

  83. [100]

    Empty if there is no answer action

    Agent Answer: The answer provided by the agent for the task. Empty if there is no answer action

  84. [101]

    The status is either Task Complete or Task Impossible

    Final Status: The final task status is indicated by the agent. The status is either Task Complete or Task Impossible. Your job is to analyze the provided information to determine whether the agent successfully completed the task, based on the alignment between the Task Instruc...

  85. [102]

    Analyze the Task Instruction: You should first analyze what critical milestones have to be completed and what necessary outcomes are expected for this task

  86. [103]

    Analyze the agent’s operations, provide a definitive verdict on whether the task has been successfully completed together your reasoning process

  87. [104]

    Task Analysis

    Provide your final answer: ’SUCCESS’ or ’NOT SUCCESS’ ## Important Notes When the task instruction asks a certain question or seeks certain information, the agent has to provide the **correct** answer before completing the task, otherwise, the task should be NOT SUCCESS. When ...

  88. [106]

    3: GT Answer: The ground truth answer for this task

    Task Guidance: An overall description of how to correctly complete this task as a reference. 3: GT Answer: The ground truth answer for this task. Empty if the answer is not available or the task does not require an answer. 4: Stored Memory: Information the agent has stored in ...

  89. [109]

    The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screenshot

    Screenshots: The screenshot at the current step and the screenshot at the next step after the current action is executed. The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screensho...

  90. [112]

    Reasoning

    Based on the current progress and the screenshot for the next step, update the task progress summarizing the overall progress after taking this current action. One or two sentences. Note that some sub-tasks or expected outcomes might need multiple actions (e.g. typing somethin...

  91. [113]

    Task Instruction: The overall task instruction for the GUI agent

  92. [114]

    32 3: GT Answer: The ground truth answer for this task

    Task Guidance: An overall description of how to correctly complete this task as a reference. 32 3: GT Answer: The ground truth answer for this task. Empty if the answer is not available or the task does not require an answer. 4: Stored Memory: Information the agent has stored ...

  93. [115]

    The progress is empty if it is the first step

    Task Progress: A summary describing the current progress of the overall task before taking the action at the current step. The progress is empty if it is the first step

  94. [116]

    Current Action: The action taken by the agent at the current step

  95. [117]

    The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screenshot

    Screenshots: The screenshot at the current step and the screenshot at the next step after the current action is executed. The click point is highlighted using a red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screensho...

  96. [118]

    Based on the provided information, analyze the agent’s action, provide a definitive verdict on whether the current action is correct, together with your reasoning process

  97. [119]

    Provide your conclusion: ’CORRECT’ or ’INCORRECT’

  98. [120]

    Reasoning

    Based on the current progress and the screenshot for the next step, update the task progress summarizing the overall progress after taking this current action. One or two sentences. Note that some sub-tasks or expected outcomes might need multiple actions (e.g. typing somethin...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.