Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a fully offline post-training recipe — action-aware SFT plus KL-regularized, success-adaptive RL — makes small open native GUI agents match closed-source systems on long-horizon web and mobile tasks without online int

desk verdict Solid, internally consistent post-training recipe for small GUI agents; the headline gains look real but unvalidated model-generated boxes and missing error bars keep me from signing off on the exact magnitudes. read the letter →

arxiv 2602.22190 v2 pith:2T2BJZRP submitted 2026-02-25 cs.LG cs.AIcs.CL

Rui Yang , Qianhui Wu , Zhaoyang Wang , Hanyang Chen , Ke Yang , Hao Cheng , Huaxiu Yao , Baolin Peng
show 3 more authors
Huan Zhang Jianfeng Gao Tong Zhang
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords GUIagentsvision-languagemodelsreinforcementlearningfromverifiablerewardspartialverifiabilityvisualgroundingaction-awaresupervisedfine-tuningKLregularizationdatacuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that open-source GUI agents lag behind closed systems not because they lack data scale or environment interaction, but because generic post-training pipelines mishandle two GUI-specific problems: long chain-of-thought reasoning degrades visual grounding, and step-wise reinforcement learning is only partially verifiable (multiple actions can be correct, but offline data credits only one). To fix this, it proposes a three-part recipe: a curated 81K reasoning dataset with filtering; action-aware SFT that mixes reasoning and direct-action data and upweights action/grounding tokens; and conservative RL that keeps a KL trust region and downweights ambiguous negative gradients. The payoff: on AndroidWorld, WebArena-Lite-v2, and Online-Mind2Web, 4B–8B models consistently beat their base models and match or exceed much larger native and modular systems — all trained offline. A sympathetic reader would care because it suggests careful data curation and training-design choices can substitute for costly online rollouts and proprietary data.

What carries the argument

Two mechanisms carry the argument. (1) Action-aware SFT (ASFT): a weighted cross-entropy objective over reasoning, action, and grounding tokens (default αa=2, αg=4) trained on a mixture of reasoning-then-action and direct-action samples, so long CoT traces no longer dilute the action/grounding signal. (2) The partially-verifiable reward analysis: Theorem 5.1 bounds online success J(π) ≥ 1 − H·C(π)·(1 − Moff(π) − η̄π), where C(π) is the occupancy mismatch between the policy's state distribution and the offline data distribution, and η̄π is the unmeasured mass the policy places on valid-but-undemonstrated actions. The theorem identifies these two quantities as the reason offline step-wise metr

What would settle it

Take a random sample of GUI-Libra-81K and the RL subset, and for each step compare the Qwen3-VL-32B bounding box to a human label or accessibility-tree element boundary. If a substantial fraction of original point_2d coordinates fall inside the model box but outside the true element — or vice versa — the reward and filter are biased. A second decisive check: re-train GUI-Libra without the box-based filter/reward, using the original coordinates only; if performance does not drop, the claimed grounding supervision is not the source of the gains.

Watch

Extended reading notes

Core claim

GUI-Libra claims that reasoning and grounding need not conflict in GUI agents if training is action-aware: standard SFT on long CoT traces degrades grounding, so it mixes reasoning-then-action and direct-action data and reweights tokens so action/grounding dominate. Its second claim is that step-wise RL is partially verifiable — multiple actions can be correct but only the demonstrated one is credited — and that this ambiguity, together with distribution shift, breaks offline-to-online predictability. A theorem (Thm 5.1) ties online success to occupancy mismatch and off-demo validity mass; KL regularization controls both. With moderate KL (β=0.001–0.005) and success-adaptive negative-gradien

Load-bearing premise

The training signal rests on bounding boxes predicted by Qwen3-VL-32B-Instruct being treated as ground truth for both reward and data filtering; if those boxes are loose or systematically misaligned, the RL reward credits off-target coordinates and the filtered data becomes biased.

Editorial extensions

If this is right

  • If the recipe is right, small open models (3B–8B) can substitute for much larger closed systems on long-horizon GUI tasks, lowering the barrier for building competent computer-use agents.
  • Data curation and filtering, not just raw scale, drive post-training gains: a filtered 81K-step dataset outperforms larger noisier corpora on both offline and online benchmarks.
  • KL regularization, which RLVR pipelines often discard, is essential for partially verifiable multi-step tasks; removing it causes policy collapse and weakens offline-to-online correlation (Pearson 0.63 vs 0.89 with KL).
  • Explicit reasoning traces become more valuable after RL: ablations show removing CoT at training time hurts online generalization far more than on offline metrics.
  • Offline step-wise metrics can be reliable predictors of online success, if the training process respects a KL trust region — useful for safely validating agents before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to validate the Qwen3-VL-32B bounding boxes against human or accessibility-tree labels; if they are loose, the same recipe with human-verified boxes would likely improve further and reduce filter bias.
  • The partial verifiability bound is domain-agnostic: it suggests any offline-trained interactive policy (web navigation, desktop control) should use conservative KL-regularized updates and success-adaptive negative scaling to keep offline metrics meaningful.
  • The paper's finding that adding grounding data to RL improves grounding but degrades navigation hints at a competition between objectives; an extension would weight the two losses dynamically or add a navigation-specific reward to the grounding mix.
  • The two refined benchmarks (AndroidControl-v2, MM-Mind2Web-v2) are author-constructed; an independent human study of label quality would establish whether the reported gains reflect real improvement or partially the revised evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GUI-Libra, a fully offline post-training recipe for native GUI agents. It constructs a curated 81K GUI reasoning dataset from public trajectory corpora by augmenting steps with GPT-4.1-generated chain-of-thought and filtering with Qwen3-VL-based agreement and bounding-box checks. Training consists of action-aware SFT (mixed reasoning-then-action and direct-action data, with token reweighting αa=2, αg=4) followed by KL-regularized GRPO with success-adaptive negative gradient scaling (SNGS). The authors give a formal analysis of partial verifiability (Theorem 5.1, Lemmas E.1–E.2) and report step-wise accuracy on AndroidControl-v2 and MM-Mind2Web-v2 plus task success on AndroidWorld, WebArena-Lite-v2, and Online-Mind2Web. The headline claims are consistent gains over base models and parity with much larger or closed systems, e.g., GUI-Libra-4B/8B reaching 42.6 on AndroidWorld, matching GPT-4o+UGround.

Significance. If the results hold, this is a significant and useful contribution: it demonstrates that careful offline data curation and conservative RL can make small open-weight native GUI agents competitive with much larger or proprietary systems, and it provides open dataset/code/model releases. The paper also gives a clear theoretical framing for why step-wise offline matching can fail to predict online success under partial verifiability, and why KL regularization should help. The proofs of Theorem 5.1 and Lemmas E.1–E.2 are correct under their stated assumptions. The empirical work is extensive: ablations of ASFT, KL coefficient, SNGS, data filtering, and grounding mixing (Tables 9–11, Figs. 9–11), plus decomposed grounding/action metrics in Appendix D.4, strengthen the paper beyond a single benchmark comparison.

major comments (4)
  1. [Secs. 4.1.4, 5.5 (Eq. 12), 6.3] Both the SFT coordinate filter and the RL grounding reward treat bounding boxes generated by Qwen3-VL-32B-Instruct as ground truth: a step is retained only if the original point_2d lies inside the predicted box, and r_g = 1{u ∈ b*}. These boxes are never validated against accessibility-tree labels, human annotations, or any independent source, despite the paper itself using accessibility trees for AndroidControl evaluation (Appendix C.2). If b* is loose or systematically shifted, the RL reward credits off-target clicks and the SFT filter preferentially retains steps that the Qwen3 family can describe. This is load-bearing because Section 6.3's claim that RL 'fully mitigates grounding degradation' rests on this reward geometry. Please report agreement statistics (e.g., precision/IoU against accessibility-tree bounds) or a sensitivity analysis with an alternative box source.
  2. [Secs. 6.1, C.2] The primary offline benchmarks are author-constructed: AndroidControl-v2 is filtered from 500 to 398 samples using Qwen3-VL-32B-Instruct, and MM-Mind2Web-v2 is produced by rewriting symbolic action histories with the same model. No external validation, inter-annotator agreement, or comparison against the original labels is provided. Because the filtering/rewriting model is from the same family as the Qwen3 backbones used for GUI-Libra-4B/8B, this can bias the offline comparison in favor of the proposed models. The headline offline gains (+9.5 to +20.9 Pass@1) are measured on these v2 benchmarks, so the construction needs independent validation or, failing that, results on the original unfiltered benchmarks.
  3. [Tables 5 and 7] AndroidWorld (115 tasks) and Online-Mind2Web (300 tasks) are reported as single runs without standard errors, confidence intervals, or repeated evaluations. The abstract's quantitative claims (+15.6%, +12.2%, +4.0%, +8.7%) are point estimates. Given the high variance typical of live and interactive benchmarks—WebArena-Lite-v2 is averaged over four runs in Table 6—the paper should provide variance information or multiple seeds for at least the main online comparisons before claiming 'consistently improves.'
  4. [Secs. 5.5, 6.3] Section 6.3 states that 'our RL stage does not use direct grounding supervision' and attributes the closing of the reasoning/grounding gap to high-level step data. However, Eq. (12) includes r_g = 1{u ∈ b*}, which is explicit spatial supervision on every click action. This is not a separate grounding-only dataset, but it is still direct grounding reward. The claim as written is inaccurate and should be revised; the paper should also discuss how much of the Table 8 improvement is attributable to this reward component versus the KL/SNGS mechanisms.
minor comments (6)
  1. [Sec. 6.1 vs Appendix B] The main text says αa=2 and αg=4 by default, but Appendix B reports αa=αg=1 for GUI-Libra-4B. Please clarify which setting was used for each table entry.
  2. [Table 11] The GUI-Libra-4B/8B numbers in Table 11 do not match the corresponding rows in Table 3 (e.g., AC-v2 High: 59.8 vs 62.3 for 4B). If these rows use the no-SNGS variant, the caption should state this explicitly.
  3. [Eq. (10)] The SNGS scaling factor has no lower bound in the displayed formula. If λ0 + κ p̂_g(s) becomes negative, negative advantages would be flipped in sign. Please state the clipping/implementation range (e.g., max(·, 0)).
  4. [Sec. 5.3.2, Lemmas E.1–E.2] The theoretical lemmas assume a per-state KL constraint, whereas GRPO applies an expected KL penalty. The paper already labels the bounds informal, but the main text should more clearly state that the formal connection to GRPO is heuristic rather than a direct implication.
  5. [Figs. 9 and 10] The correlation analysis in Fig. 10 is computed over checkpoints from runs with different KL coefficients and model sizes. Please state the number of checkpoints and whether the correlations are pooled across models; this affects interpretation of the p-values.
  6. [Sec. 1 / Abstract] Minor typos: 'RL VR-tyle' in the abstract and 'even GPT5' in Section 6.2.1 should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation chain is self-contained; the unvalidated Qwen3-VL grounding boxes are a data-fidelity caveat, not a by-construction reduction.

full rationale

The paper's claimed derivation chain is not circular. ASFT (Eq. 3) is a reweighted token-level objective whose coefficients αa and αg are engineering constants selected via validation, not quantities derived from the benchmarks being predicted. The theoretical results (Theorem 5.1, Lemmas E.1/E.2) are proved from explicit assumptions (Assumption 5.1, support condition, per-state KL constraint) and do not assume their conclusions; they provide a substantive lower bound relating J, M_off, C, and η̄ that holds independent of the training recipe. The RL reward (Eq. 12) and SNGS (Eqs. 10-11) are training objectives, not evaluation metrics, and the headline claims are assessed on external benchmarks (AndroidWorld, WebArena-Lite-v2, Online-Mind2Web, ScreenSpot-v2/Pro) with ground truth not generated by the training loop. Self-citations to prior work by the same group (e.g., GUI-Actor, ERA) appear only in related-work context and are not load-bearing; no uniqueness theorem is imported from the authors to force the method choice. The main caveat—Qwen3-VL-32B-generated bounding boxes are used both to filter SFT data (Sec. 4.1.4) and as the grounding reward target b* (Sec. 5.5), without validation against human or accessibility-tree labels—is a data-fidelity and possible-distillation concern, not a circular reduction: the policy is not correct-by-construction, and the key empirical claims are verified against independent ground truth. Similarly, the author-refined offline benchmarks AndroidControl-v2 and MM-Mind2Web-v2 are benchmark-construction choices, but the recipe's hyperparameters were not fitted to those refined labels, and the online results do not depend on them. Therefore, no load-bearing step in the derivation reduces to its own input.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The headline empirical claims are not fitted to the headline benchmarks (those are held-out external sets), so the paper's main load is in engineering parameters (ASFT weights, KL β, SNGS settings, filter thresholds) and in unvalidated model-generated annotations used as reward targets. The theory is self-contained but rests on standard distribution-shift assumptions whose satisfaction in the experiments is not directly verified.

free parameters (6)
  • ASFT action/grounding weights αa, αg = αa=2, αg=4 default; αa=αg=1 for 4B
    Token weights in Eq 3 tuned to validation/online performance; different values for different backbones (Appendix B).
  • KL coefficient β = 0.001 (0.005 for 7B)
    Chosen per model to stabilize RL; Table 9 shows sensitivity (0.0/0.001/0.01/0.05 give AndroidWorld 21.7/25.2/21.7/20.0).
  • SNGS offset λ0 and slope κ = (0.9,0.5) 3B; (1.4,−0.5) 7B; (0.5,1.5) 4B; (0.5,2.0) 8B
    Four different settings for four models; paper concedes SNGS 'is sensitive to hyperparameters' (Sec 6.1, Appendix B).
  • Format-reward weight w_fmt = 0.1
    Weight between format and accuracy verifiers in Eq 12, chosen by hand.
  • SFT filter threshold = re-prediction accuracy ≥ 0.3
    Steps below 0.3 agreement across 10 Qwen3-VL-8B stochastic runs are discarded (Sec 4.1.4).
  • RL data downsampling = 40K steps after removing early/mobile-heavy steps
    Downsampling ratios chosen by hand to balance step-index and domain distributions (Sec 4.1.5).
assumptions (6)
  • domain assumption Assumption 5.1: failure of an episode implies at least one step action was invalid (a_t ∉ A*(s_t))
    Needed for the union-bound step in Theorem 5.1; excludes failures caused by goal ambiguity or observation noise.
  • domain assumption Support containment Supp(d_{π,t}) ⊆ Supp(d_μ) with bounded density ratio C(π)
    Defines the occupancy-mismatch term; strong in practice and acknowledged in Sec 5.3 as a condition for the bound.
  • domain assumption Offline distribution has positive lower bound ρ = inf_{s:d_μ(s)>0} d_μ(s) > 0
    Required for Lemma E.1's occupancy bound; offline datasets generally have thin tails.
  • domain assumption Reference (SFT) policy is demo-concentrated: π_ref(ã(s)|s) ≥ 1−δ(s)
    Needed for Lemma E.2 to bound off-demo validity mass; plausible for an SFT init but not verified for the trained policies.
  • ad hoc to paper Qwen3-VL-32B-generated bounding boxes are accurate enough to serve as RL reward targets and data-filter criteria
    r_g = 1{u ∈ b*} (Sec 5.5) and the Sec 4.1.4 filter both rely on these boxes; no validation vs human/accessibility-tree labels is provided.
  • standard math The GUI POMDP is analyzed as an MDP over history/belief states
    Sec 5.3 setup: 'treating the agent's history (or belief state) as the effective state' — a standard but simplifying reduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL." pith.science (2026). https://pith.science/paper/2T2BJZRP

@misc{pith2026260222190,
  author       = {Pith},
  title        = {Pith review of: GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2T2BJZRP}},
  note         = {Machine review of arXiv:2602.22190}
}
read the original abstract

Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks. This gap stems from two limitations: a shortage of high-quality, action-aligned reasoning data, and the direct adoption of generic post-training pipelines that overlook the unique challenges of GUI agents. We identify two fundamental issues in these pipelines: (i) standard SFT with CoT reasoning often hurts grounding, and (ii) step-wise RLVR-tyle training faces partial verifiability, where multiple actions can be correct but only a single demonstrated action is used for verification. This makes offline step-wise metrics weak predictors of online task success. In this work, we present GUI-Libra, a tailored training recipe that addresses these challenges. First, to mitigate the scarcity of action-aligned reasoning data, we introduce a data construction and filtering pipeline and release a curated 81K GUI reasoning dataset. Second, to reconcile reasoning with grounding, we propose action-aware SFT that mixes reasoning-then-action and direct-action data and reweights tokens to emphasize action and grounding. Third, to stabilize RL under partial verifiability, we identify the overlooked importance of KL regularization in RLVR and show that a KL trust region is critical for improving offline-to-online predictability; we further introduce success-adaptive scaling to downweight unreliable negative gradients. Across diverse web and mobile benchmarks, GUI-Libra consistently improves both step-wise accuracy and end-to-end task completion. Our results suggest that carefully designed post-training and data curation can unlock significantly stronger task-solving capabilities without costly online data collection. We release our dataset, code, and models to facilitate further research on data-efficient post-training for reasoning-capable GUI agents.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A proposal-aided interleaved grounding framework with action-aware supervision reports SOTA accuracy on dense sports video QA, evaluated on the authors' self-curated test sets.

  2. AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    AsyncWebRL reports up to 2.9x training speedup and new SOTA on WebGym OOD split via async overlap plus constant normalizer in GRPO, with largest gains on harder tasks.

  3. Software Engineering for and with GUI Agent

    cs.SE 2026-08 conditional novelty 5.0 of 10

    A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.

Reference graph

Works this paper leans on

87 extracted references · 37 linked inside Pith · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Fara-7b: An efficient agentic model for computer use

    Ahmed Awadallah, Yash Lara, Raghav Magazine, Hussein Mozannar, Akshay Nambi, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Vibhav Vineet, Spencer Whitehead, and Andrew Zhao. Fara-7b: An efficient agentic model for computer use. arXiv:2511.19663, 2025

    arXiv 2025

  3. [3]

    Uibert: Learning generic multimodal representations for ui understanding, 2021

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. Uibert: Learning generic multimodal representations for ui understanding, 2021

    2021

  4. [4]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, 2024 a

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, 2024 a . URL https://arxiv.org/abs/2406.11896

    arXiv 2024

  5. [5]

    Digi RL : Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digi RL : Training in-the-wild device-control agents with autonomous reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview.net/forum?id=4XTvXMSZPO

    2024

  6. [6]

    Webgym: Scaling training environments for visual web agents with realistic tasks

    Hao Bai, Alexey Taymanov, Tong Zhang, Aviral Kumar, and Spencer Whitehead. Webgym: Scaling training environments for visual web agents with realistic tasks. arXiv preprint arXiv:2601.02439, 2026

    arXiv 2026

  7. [7]

    Qwen3-vl technical report, 2025 a

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  8. [8]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025 b

    arXiv 2025

Show all 87 references
  1. [9]

    AMEX : Android multi-annotation expo dataset for mobile GUI agents

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Guozhi Wang, Dingyu Zhang, Shuai Ren, and Hongsheng Li. AMEX : Android multi-annotation expo dataset for mobile GUI agents. In Findings of the Association for Computational Linguistics: ACL 2025, 2025. URL https://acl...

    2025

  2. [10]

    Era: Transforming vlms into embodied agents via embodied prior learning and online reinforcement learning, 2025 a

    Hanyang Chen, Mark Zhao, Rui Yang, Qinwei Ma, Ke Yang, Jiarui Yao, Kangrui Wang, Hao Bai, Zhenhailong Wang, Rui Pan, Mengchao Zhang, Jose Barreiros, Aykut Onol, ChengXiang Zhai, Heng Ji, Manling Li, Huan Zhang, and Tong Zhang. Era: Transforming vlms into embodied agents via em...

    2025

  3. [11]

    Ui-ins: Enhancing gui grounding with multi-perspective instruction-as-reasoning, 2025 b

    Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Quyu Kong, Xu Zhang, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, Qin Jin, and Steven Hoi. Ui-ins: Enhancing gui grounding with multi-perspective instruction-as-reasoning, 2025 b . URL https://arxiv.org/abs/...

    2025

  4. [12]

    GUIC ourse: From general vision language model to versatile GUI agent

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. GUIC ourse: From general vision language model to versatile GUI agent. In Proceedings of the 63rd Annual Me...

    2025

  5. [13]

    Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024 a

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024 a . URL https://arxiv.org/abs/2401.10935

    2024 arXiv

  6. [14]

    S ee C lick: Harnessing GUI grounding for advanced visual GUI agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. S ee C lick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2024

  7. [15]

    Rico: A mobile app dataset for building data-driven design applications

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Tech...

    2017

  8. [16]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=kiYqbO3wqw

    2023

  9. [17]

    Openvlthinker: Complex vision-language reasoning via iterative sft-rl cycles, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: Complex vision-language reasoning via iterative sft-rl cycles, 2025. URL https://arxiv.org/abs/2503.17352

    2025 arXiv

  10. [18]

    Group-in-group policy optimization for llm agent training, 2025

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training, 2025. URL https://arxiv.org/abs/2505.10978

    2025 arXiv

  11. [19]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  12. [20]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrev...

    2025

  13. [21]

    Ui-venus technical report: Building high-performance ui agents with rft, 2025

    Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Lian...

    2025 arXiv

  14. [22]

    Scalable data synthesis for computer use agents with step-level filtering

    Yifei He, Pranit Chawla, Yaser Souri, Subhojit Som, and Xia Song. Scalable data synthesis for computer use agents with step-level filtering. arXiv preprint arXiv:2512.10962, 2025

    2025

  15. [23]

    Cogagent: A visual language model for gui agents, 2023

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023

    2023

  16. [24]

    Cogagent: A visual language model for gui agents, 2024

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2024. URL https://arxiv.org/abs/2312.08914

    2024 arXiv

  17. [25]

    Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025

    2025 arXiv

  18. [26]

    Screenspot-pro: Gui grounding for professional high-resolution computer use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pp.\ 8778--8786, 2025

    2025

  19. [27]

    On the effects of data scale on UI control agents

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on UI control agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. ...

    2024

  20. [28]

    Mapping natural language instructions to mobile UI action sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020 a . URL https://aclanthology.org/2020.acl-main.729/

    2020

  21. [29]

    Widget captioning: Generating natural language description for mobile user interface elements

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020 b . URL ...

    2020

  22. [30]

    Showui: One vision-language-action model for gui visual agent, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent, 2024. URL https://arxiv.org/abs/2411.17465

    2024 arXiv

  23. [31]

    What can rl bring to vla generalization? an empirical study, 2026

    Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang. What can rl bring to vla generalization? an empirical study, 2026. URL https://arxiv.org/abs/2505.19789

    2026

  24. [32]

    Autoglm: Autonomous foundation agents for guis

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024

    2024 arXiv

  25. [33]

    Visualagentbench: Towards large multimodal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhe...

    2025

  26. [34]

    Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization

    Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, et al. Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization. arXiv preprint arXiv:2508.05731, 2025 b

    2025

  27. [35]

    Scalecua: Scaling open-source computer use agents with cross-platform data

    Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, Shenglong Ye, Qingyun Li, Xuan Dong, Yue Yu, Chenyu Lu, YunXiang Mo, Yao Yan, Zeyue Tian, Xiao Zhang, Yuan Huang, Yiqian Liu, Weijie Su, Gen Luo, Xiangyu Yu...

    2025

  28. [36]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025 d

    2025 arXiv

  29. [37]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  30. [38]

    Ui-r1: Enhancing action prediction of gui agents by reinforcement learning

    Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025

    2025 arXiv

  31. [39]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025

    2025 arXiv

  32. [40]

    Gui-360 ^ : A comprehensive dataset and benchmark for computer-using agents, 2025

    Jian Mu, Chaoyun Zhang, Chiming Ni, Lu Wang, Bo Qiao, Kartik Mathur, Qianhui Wu, Yuhang Xie, Xiaojun Ma, Mengyu Zhou, Si Qin, Liqun Li, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Gui-360 ^ : A comprehensive dataset and benchmark for computer-using a...

    2025

  33. [41]

    Rodriguez, Montek Kalsi, Nicolas Chapados, M

    Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Nicolas Chapados, M. Tamer \"O zsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. UI -vision: A desktop-centric GUI benchmark for visual p...

    2025

  34. [42]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  35. [43]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  36. [44]

    Agent q: Advanced reasoning and learning for autonomous ai agents, 2024

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024. URL https://arxiv.org/abs/2408.07199

    2024 arXiv

  37. [45]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025. URL https:/...

    2025 arXiv

  38. [47]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025 b

    2025 arXiv

  39. [48]

    Android in the wild: a large-scale dataset for android device control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: a large-scale dataset for android device control. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, 2023

    2023

  40. [49]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P Lillicrap, and Oriana Riva. Androidworld: A d...

    2025

  41. [50]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  42. [51]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  43. [52]

    Gui-g ^2 : Gaussian reward modeling for gui grounding, 2025

    Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang. Gui-g ^2 : Gaussian reward modeling for gui grounding, 2025. URL https://arxiv.org/abs/2507.15846

    2025 arXiv

  44. [53]

    Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning, 2025 a

    Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbi...

    2025 arXiv

  45. [54]

    Think or not? selective reasoning via reinforcement learning for vision-language models

    Jiaqi Wang, Kevin Qinghong Lin, James Cheng, and Mike Zheng Shou. Think or not? selective reasoning via reinforcement learning for vision-language models. arXiv preprint arXiv:2505.16854, 2025 b

    2025

  46. [55]

    Vagen: Reinforcing world model reasoning for multi-turn vlm agents, 2025 c

    Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Yejin Choi, and Manling Li. Vagen: Reinforcing world model reasoning for multi-turn vlm agents, 20...

    2025

  47. [56]

    Open CUA : Open foundations for computer-use agents

    Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Zheng Boyuan, LI PEIHANG, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, ...

    2025

  48. [57]

    Aligning large language models with human: A survey

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023

    2023 arXiv

  49. [58]

    Webui: A dataset for enhancing visual ui understanding with web semantics

    Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey Bigham. Webui: A dataset for enhancing visual ui understanding with web semantics. ACM Conference on Human Factors in Computing Systems (CHI), 2023

    2023

  50. [59]

    Gui-actor: Coordinate-free visual grounding for gui agents

    Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143, 2025 a

    2025 arXiv

  51. [60]

    Os-atlas: A foundation action model for generalist gui agents, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv.org/abs/2410.23218

    2024 arXiv

  52. [61]

    OS - ATLAS : Foundation action model for generalist GUI agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS - ATLAS : Foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representations, 2...

    2025

  53. [62]

    Scaling computer-use grounding via user interface decomposition and synthesis, 2025

    Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. URL ...

    2025

  54. [63]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openrevi...

    2025

  55. [64]

    Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025 b

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025 b . URL https://arxiv.org/abs/2412.09605

    2025 arXiv

  56. [65]

    Aguvis: Unified pure vision agents for autonomous GUI interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUI interaction. In Forty-second International Conference on Machine Learning, 2025 c . URL https://openreview.net/...

    2025

  57. [66]

    An illusion of progress? assessing the current state of web agents

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=6jZi4HSs6o

    2025

  58. [67]

    Pixelweb: The first web gui dataset with pixel-wise labels, 2025 a

    Qi Yang, Weichen Bi, Haiyang Shen, Yaoqi Guo, and Yun Ma. Pixelweb: The first web gui dataset with pixel-wise labels, 2025 a . URL https://arxiv.org/abs/2504.16419

    2025 arXiv

  59. [68]

    Regularizing hidden states enables learning generalizable reward model for llms

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. Advances in Neural Information Processing Systems, 37: 0 62279--62309, 2024 a

    2024

  60. [69]

    Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment

    Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207, 2024 b

    2024 arXiv

  61. [70]

    Gta1: Gui test-time scaling agent, 2025 b

    Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Silvio Savarese, Caiming Xiong, and Junnan Li. Gta1: Gui test-time scaling agent, 2025 b . URL https://arxiv.org/abs/2507.05791

    2025 arXiv

  62. [71]

    Aria-ui: Visual grounding for gui instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024 c

    2024 arXiv

  63. [72]

    Ferret-ui lite: Lessons from building small on-device gui agents, 2025 c

    Zhen Yang, Zi-Yi Dou, Di Feng, Forrest Huang, Anh Nguyen, Keen You, Omar Attia, Yuhao Yang, Michael Feng, Haotian Zhang, Ram Ramrakhya, Chao Jia, Jeffrey Nichols, Alexander Toshev, Yinfei Yang, and Zhe Gan. Ferret-ui lite: Lessons from building small on-device gui agents, 2025...

    2025

  64. [73]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  65. [74]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning, 2024

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning, 2024. URL https://arxiv.org/abs/2405.10292

    2024 arXiv

  66. [75]

    Visual backdoor attacks on mllm embodied decision making via contrastive trigger learning

    Qiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen, Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, and Daniel Kang. Visual backdoor attacks on mllm embodied decision making via contrastive trigger learning. arXiv preprint arXiv:2510.27623, 2025

    2025

  67. [76]

    Tongui: Internet-scale trajectories from multimodal web tutorials for generalized gui agents

    Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. Tongui: Internet-scale trajectories from multimodal web tutorials for generalized gui agents. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025 a

    2025

  68. [77]

    Android in the zoo: Chain-of-action-thought for GUI agents

    Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for GUI agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. URL https://aclanthology.org/2024.finding...

    2024

  69. [78]

    Agent learning via early experience

    Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558, 2025 b

    2025 arXiv

  70. [79]

    Phi-ground tech report: Advancing perception in gui grounding

    Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, et al. Phi-ground tech report: Advancing perception in gui grounding. arXiv preprint arXiv:2507.23779, 2025 c

    2025 arXiv

  71. [80]

    Gpt-4v(ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. 2024. URL https://openreview.net/forum?id=piecKJ2DlB

    2024

  72. [81]

    Group sequence policy optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025 a

    2025 arXiv

  73. [82]

    Agentstudio: A toolkit for building general virtual agents

    Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng YAN. Agentstudio: A toolkit for building general virtual agents. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=axUf8BOjnH

    2025

  74. [83]

    Mai-ui technical report: Real-world centric foundation gui agents, 2025 a

    Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, and Steven Hoi. Mai-ui technical report: Real-world centric foundation gui agents, 2025 a . URL https://arxiv.org/abs/2512.22047

    2025

  75. [84]

    GUI -g1: Understanding r1-zero-like training for visual grounding in GUI agents

    Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. GUI -g1: Understanding r1-zero-like training for visual grounding in GUI agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 b . URL https://openreview.net/forum?i...

    2025

  76. [85]

    The surprising effectiveness of negative reinforcement in llm reasoning

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347, 2025

    2025

  77. [86]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  78. [87]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  79. [88]

    point_2d

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.