Pith. sign in

REVIEW 3 major objections 4 minor 42 references

A new pretraining stage on screen transitions gives GUI agents a consistent win over trajectory-only fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:00 UTC pith:PBZ6R74J

load-bearing objection A well-controlled empirical paper on a cheap pretraining signal for GUI agents; the desktop gains look real, but the mobile gains are too small and statistically unanchored to support the 'consistently across all settings' claim. the 3 major comments →

arxiv 2607.24112 v1 pith:PBZ6R74J submitted 2026-07-27 cs.AI

Scaling GUI Agents with Visual State Transitions

Xiangyan Liu , Kaixin Li , Haonan Wang , Biao Wu , Meng Fang , Longxu Dou , Chao Du , Michael Qizhe Shieh
show 1 more author
Tianyu Pang
This is my paper
classification cs.AI
keywords GUI agentsstate transition pretraininginverse dynamicsforward dynamicsworld modeltrajectory fine-tuningmultimodal pretrainingscaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that GUI agents can be scaled more cheaply by pretraining on step-level visual state transitions—(current screen, action, next screen)—before the usual trajectory fine-tuning. The proposed State Transition Pretraining (STP) jointly optimizes inverse dynamics (recover the action from two consecutive screenshots) and forward dynamics (predict the next screenshot from a screen and an action). Across desktop and mobile benchmarks, the same fine-tuning setup produces higher success rates when initialized with STP, with average gains of 2.3–6.2 percentage points on AgentNetBench and smaller but consistent gains on AndroidControl and GUIOdyssey. The paper also argues that the gains are not simply extra data exposure, and that downstream performance keeps rising as transition data grows. If correct, transition data becomes a cheaper, annotation-free scaling axis for GUI agents.

Core claim

STP claims to improve GUI agents' initialization by decomposing trajectories into atomic transition tuples and jointly optimizing two dynamics objectives in a unified multimodal model. Inverse dynamics forces the model to localize which interface element changed and infer the action that produced the change; forward dynamics forces it to internalize how GUIs respond to actions, forming an internal world model of GUI behavior. After this pretraining, trajectory fine-tuning outperforms fine-tuning alone on every tested setting. Notably, the improvement persists even when the transitions are extracted from the exact same trajectories later used for fine-tuning, and it grows with the volume of t

What carries the argument

The central object is the transition tuple (s_t, a_t, s_{t+1}), converted into two objectives within a single multimodal model: inverse dynamics (two screenshots to action text) and forward dynamics (screen plus action to next-screen image), optimized jointly before supervised trajectory fine-tuning. These objectives separate visual grounding from task planning and give the fine-tuning stage an action-grounded visual representation plus a lightweight internal model of GUI dynamics.

Load-bearing premise

The comparison that isolates STP assumes the fine-tune-only baseline has converged; the paper supports convergence with cross-entropy loss plateaus rather than success-rate curves, so if more fine-tuning would have closed the gap, part of the reported gain could be an extra-compute effect instead of the pretrained initialization.

What would settle it

Run the FT w/o STP baseline for the same total number of optimizer steps as the STP+FT pipeline, or until its success rate on the benchmark is flat rather than only its loss; if it reaches the same final success rate, the central claim fails. A sharper check: pretrain with the same transition data but shuffled or random action/screen targets, and see whether the downstream gain disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Starting trajectory fine-tuning from an STP checkpoint improves average success rate by 2.3–6.2 percentage points on AgentNetBench, 0.9 on AndroidControl, and 0.6 on GUIOdyssey under identical fine-tuning configurations.
  • The improvement is not simply exposure to unseen data: STP still helps when its transitions come from the same 18K trajectories later used for fine-tuning.
  • Downstream performance scales with transition pretraining volume even when fine-tuning data is fixed, so collecting more step-level transitions should keep helping.
  • Joint inverse and forward dynamics outperform either objective alone and beat reconstruction-only pretraining, indicating that action-aware next-state prediction is the informative signal.
  • Updating the full model matters more than improving the visual encoder alone, so the language component must be jointly optimized during STP.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because STP needs only step-level transitions, automated exploration of interactive environments could produce effectively unlimited pretraining data; the paper identifies this as future work, and its step-level sampling ablation suggests the approach would work.
  • The forward-dynamics head may have a practical side benefit not exercised here: at inference, a GUI agent could simulate the next screen before committing to an action, enabling lookahead or safety checks.
  • STP is largely orthogonal to the trajectory fine-tuning pipeline, so it could plausibly combine with reinforcement learning or other post-training methods, not just supervised fine-tuning.
  • If the paper's explanation for smaller mobile gains is right—backbone prior exposure and easier tasks—STP's value should grow as benchmarks become harder and less represented in base-model pretraining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces State Transition Pretraining (STP), a pretraining stage for GUI agents in which a unified multimodal model (BAGEL) is continually trained on step-level visual transitions (s_t, a_t, s_{t+1}) using two objectives: inverse dynamics (predict the action from consecutive screenshots) and forward dynamics (generate the next screenshot from the current state and action). The STP checkpoint is then used to initialize standard trajectory fine-tuning. The authors compare STP + fine-tuning against direct trajectory fine-tuning on AgentNetBench, AndroidControl, and GUIOdyssey, reporting consistent improvements in average or step success rate across all settings. They also present ablations on pretraining objectives, sampling granularity, model components updated, and transition-data scale, and release code.

Significance. If the reported effects are reliable, STP is a practical and annotation-light scaling axis for GUI agents: transition tuples can be extracted from existing trajectories or collected automatically, and the controlled comparison protocol (same backbone, same fine-tuning data, same training configuration) is a methodological strength. The paper also contains useful ablations—joint inverse+forward beats single objectives, forward dynamics beats reconstruction, full-model tuning is necessary, step-level and trajectory-level sampling are comparable, and scaling transition data helps. The Group 1c control, showing gains even when STP uses no data beyond the fine-tuning pool, is a good attempt to separate data exposure from learning-signal effects. However, the absence of any variance estimates or significance tests, together with the small magnitude of the mobile gains, means the universality claim in the abstract and Section 3.2 is not yet established. The desktop results are substantially more convincing than the mobile results.

major comments (3)
  1. [Section 3.2, Tables 1–2] The central claim that STP consistently improves performance in both desktop and mobile scenarios rests on single-run comparisons with no error bars, significance tests, or multiple seeds. The mobile gains are +0.9 pp SR on AndroidControl and +0.6 pp SR on GUIOdyssey (Table 2); at 7B scale, run-to-run variation on such benchmarks is typically on the order of 0.5–1.5 pp, so these differences are within plausible noise. Because the abstract and Section 3.2 use these numbers to claim improvement across all settings, the mobile claim is load-bearing and currently unsupported. Please report at least 3 seeds with mean±std or confidence intervals, or explicitly restrict the universality claim to the desktop settings where the gains are larger.
  2. [Section 3.3, 'Practical cost'; Section 1, Figure 1 and Figure 4 (left)] The paper states that all trajectory fine-tuning runs are trained to convergence, as reflected by the plateaued cross-entropy loss in Figure 4 (left). However, Figure 4 (left) compares datasets rather than fine-tuning with and without STP, and a CE plateau does not imply that success rate has converged. If the FT-only baseline would still improve with more compute, the STP gains could be partly an extra-compute effect. The convergence claim needs success-rate curves for both conditions over the full training horizon, and ideally compute-matched baselines (e.g., FT-only trained for the same total number of steps as STP+FT). Without this, the conclusion in Group 1c that gains come from structured learning signals rather than additional compute is not fully supported.
  3. [Section 3.1, 'Implementation details'; Figure 1 caption] Reporting 'the performance of the best-performing checkpoint on the corresponding benchmark' for a single run can amplify small differences: with a single seed, selecting the best epoch can inflate the apparent effect, and there is no way to assess the variance. The same issue affects the scaling curve in Figure 6 (right) and the ablation bars in Figure 5. Please provide full training curves, fixed-epoch results, or averaged statistics over seeds; at minimum, report variance estimates for the headline numbers.
minor comments (4)
  1. [Section 3.1, 'Experimental Setup'] The evaluation decodes with temperature 0.6 and averages four samples per instance, but no standard deviation across samples is reported. Reporting this spread would help calibrate the small differences in Table 2.
  2. [Section 2.3, Eq. (2)] The joint loss weight λ is set to 0.25 for AgentNet and GUIOdyssey and 0.5 for AndroidControl, but no sensitivity analysis is provided. Since λ is a free parameter, the authors should show that the main results are stable for nearby values or justify the choice.
  3. [Figure 1 and Figure 4 captions] Figure 1 (left) mixes two metrics (cross-entropy loss and average success rate) in a single plot; the axes and curve labels need to be explicit. Also, Figure 4 (left) is cited in the convergence argument but does not compare fine-tuning with and without STP, so the citation is misleading.
  4. [Section 3.2, 'Smaller gains in mobile scenarios'] The explanation that smaller mobile gains stem from prior data exposure and lower task complexity is plausible but untested. It would be more appropriate to present this as a hypothesis, or to support it with a benchmark less likely to be in the backbone's pretraining distribution.

Circularity Check

0 steps flagged

No significant circularity: the central result is an empirical pretraining comparison, not a derived prediction, and its benchmarks are outside the training objective.

full rationale

The paper's central claim is that STP (joint inverse and forward dynamics pretraining on (s_t, a_t, s_{t+1}) tuples) improves subsequent trajectory fine-tuning. This is established by controlled comparisons (FT w/ STP vs FT w/o STP) on external benchmarks (AgentNetBench, AndroidControl, GUIOdyssey) under identical fine-tuning data and configurations. No quantity claimed as a prediction is fitted from the downstream benchmark: the STP objective (Eq. 2) is defined over transitions, and downstream success rates are measured rather than derived from the pretraining loss. The per-dataset values of lambda (0.25/0.5) are pretraining hyperparameters, not fitted to the downstream metric, and both arms share the same checkpoint-selection protocol. The convergence argument (Section 3.3) is an empirical control claim, not a definitional equivalence. The clearest self-citation by the present authors is Wang et al. 2025b, cited in Section 4.2 only as an example of existing self-supervised forward-dynamics/world-model work; it is peripheral and not load-bearing. The paper's own ablations and external benchmark comparisons carry the argument. There is no uniqueness theorem, no fitted parameter renamed as a prediction, and no benchmark defined in terms of the method's own outputs. Therefore no circular step can be exhibited. Concerns about small mobile gains, missing error bars, or whether the FT-only arm had fully converged are statistical or evaluation-validity issues, not circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The central claims rest on three main premises: forward-generation supervision transfers to action understanding, CE-loss plateaus are a valid proxy for performance convergence, and the desktop benchmark does not overlap with the transition corpus. The only tuned free parameter is the inverse/forward weight lambda, which varies across datasets.

free parameters (1)
  • joint-pretraining loss weight lambda = 0.25 (AgentNet, GUIOdyssey), 0.5 (AndroidControl)
    Set separately per dataset in Section 3.1; no sensitivity analysis or selection procedure is reported, so downstream results may depend on this hand-chosen value.
axioms (3)
  • domain assumption The forward-dynamics generative objective provides useful action-grounded supervision to the shared representations.
    Central to the claim that joint inverse+forward beats single-objective; the paper cites prior results (Su et al. 2026; Wu et al. 2026b) but reports no image-generation quality or representation diagnostics for the forward objective.
  • ad hoc to paper CE-loss convergence of the FT-only baseline implies downstream success-rate convergence.
    Used in Section 3.3 to argue that gains cannot be recovered by more fine-tuning compute; only training-loss plateaus are shown, not downstream success-rate curves for the baseline.
  • domain assumption AgentNetBench is a clean holdout relative to the 18K Win&Mac trajectories used for STP extraction and fine-tuning.
    Footnote 1 calls AgentNetBench standalone, but overlap is not quantified; Group 1c intentionally reuses the same trajectories, so the separation of evaluation from training data matters for the desktop claims.

pith-pipeline@v1.3.0-alltime-deepseek · 18279 in / 13758 out tokens · 116095 ms · 2026-07-31T23:00:49.145374+00:00 · methodology

0 comments
read the original abstract

We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.

Figures

Figures reproduced from arXiv: 2607.24112 by Biao Wu, Chao Du, Haonan Wang, Kaixin Li, Longxu Dou, Meng Fang, Michael Qizhe Shieh, Tianyu Pang, Xiangyan Liu.

Figure 1
Figure 1. Figure 1: State Transition Pretraining (STP) improves both the training optimization and downstream [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Examples of visual state transition data from three GUI control sources: AgentNet, AndroidControl, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of inverse and forward dynamics in STP. Forward dynamics models the transition [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics under different STP conditions. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation on pretraining objectives. We compare inverse-only, forward-only, inverse+forward, and inverse+reconstruction pretraining across different fine-tuning settings. Results are reported as average success rate on AgentNetBench and step accuracy on AndroidControl. Each bar corresponds to a pretraining objective, and the red dashed line indicates the fine-tune-only baseline without state transition pret… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on visual understanding encoder tuning and pretraining scale. The left plot compares different pretraining strategies, including no STP, full-model STP, and STP with only the visual understanding encoder (ViT in BAGEL) being updated, with Cross-Entropy (CE) loss and average SR tracked over training steps. The right plot compares different pretraining data sizes, reporting CE loss after 3500 fine-t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 23 linked inside Pith

  1. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv pr...

    Pith/arXiv arXiv 2024

  2. [5]

    Internalizing world models via self-play finetuning for agentic rl.arXiv preprint arXiv:2510.15047, 2025a

    Shiqi Chen, Tongyao Zhu, Zian Wang, Jinghan Zhang, Kangrui Wang, Siyang Gao, Teng Xiao, Yee Whye Teh, Junxian He, and Manling Li. Internalizing world models via self-play finetuning for agentic rl.arXiv preprint arXiv:2510.15047, 2025a. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified ...

    arXiv

  3. [6]

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan

    arXiv:2401.10935. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Pith/arXiv arXiv

  4. [8]

    Grounding computer use agents on human demonstrations.arXiv preprint arXiv:2511.07332,

    13 Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A Rodriguez, Nicolas Chapados, et al. Grounding computer use agents on human demonstrations.arXiv preprint arXiv:2511.07332,

    Pith/arXiv arXiv

  5. [9]

    GUI-Shift: Enhancing VLM-based GUI agents through self-supervised reinforcement learning.arXiv preprint arXiv:2505.12493,

    Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, and Mengwei Xu. GUI-Shift: Enhancing VLM-based GUI agents through self-supervised reinforcement learning.arXiv preprint arXiv:2505.12493,

    arXiv

  6. [10]

    Generalization in online reinforcement learning for mobile agents.arXiv preprint arXiv:2603.07432,

    Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Glen Berseth, and Yang Wang. Generalization in online reinforcement learning for mobile agents.arXiv preprint arXiv:2603.07432,

    arXiv

  7. [11]

    Ui-venus technical report: Building high-performance ui agents with rft.arXiv preprint arXiv:2508.10833,

    Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, et al. Ui-venus technical report: Building high-performance ui agents with rft.arXiv preprint arXiv:2508.10833,

    Pith/arXiv arXiv

  8. [12]

    Computer-using world model.arXiv preprint arXiv:2602.17365,

    Yiming Guan, Rui Yu, John Zhang, Lu Wang, Chaoyun Zhang, Liqun Li, Bo Qiao, Si Qin, He Huang, Fangkai Yang, et al. Computer-using world model.arXiv preprint arXiv:2602.17365,

    arXiv

  9. [13]

    WebSTAR: Scalable data synthesis for computer use agents with step-level filtering.arXiv preprint arXiv:2512.10962,

    Yifei He, Pranit Chawla, Yaser Souri, Subhojit Som, and Xia Song. WebSTAR: Scalable data synthesis for computer use agents with step-level filtering.arXiv preprint arXiv:2512.10962,

    arXiv

  10. [14]

    Screenqa: Large-scale question-answer pairs over mobile app screenshots

    Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Cărbune, Jason Lin, Maria Wang, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2025

  11. [15]

    Weblica: Scalable and reproducible training environments for visual web agents.arXiv preprint arXiv:2605.06761,

    Oğuzhan Fatih Kar, Roman Bachmann, Yuting Gong, Anders Boesen Lindbo Larsen, and Afshin De- hghan. Weblica: Scalable and reproducible training environments for visual web agents.arXiv preprint arXiv:2605.06761,

    Pith/arXiv arXiv

  12. [16]

    Generative visual code mobile world models.arXiv preprint arXiv:2602.01576,

    Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, and Jamin Shin. Generative visual code mobile world models.arXiv preprint arXiv:2602.01576,

    Pith/arXiv arXiv

  13. [17]

    Computerrl: Scaling end-to-end online reinforcement learning for computer use agents

    Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer use agents. arXiv preprint arXiv:2508.14040,

    arXiv

  14. [18]

    Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, et al

    arXiv:2406.03679. Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, et al. Ui-voyager: A self-evolving gui agent learning via failed experience.arXiv preprint arXiv:2603.24533,

    Pith/arXiv arXiv

  15. [19]

    Mobileforge: Annotation-free adaptation for mobile gui agents with hierarchical feedback-guided policy optimization.arXiv preprint arXiv:2606.19930,

    Guangyi Liu, Pengxiang Zhao, Gao Wu, Yiwen Yin, Mading Li, et al. Mobileforge: Annotation-free adaptation for mobile gui agents with hierarchical feedback-guided policy optimization.arXiv preprint arXiv:2606.19930,

    Pith/arXiv arXiv

  16. [20]

    Visu- alwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955,

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visu- alwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955,

    Pith/arXiv arXiv

  17. [21]

    ScaleCUA: Scaling open-source computer use agents with cross-platform data.arXiv preprint arXiv:2509.15221,

    Zhaoyang Liu, JingJing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, et al. ScaleCUA: Scaling open-source computer use agents with cross-platform data.arXiv preprint arXiv:2509.15221,

    arXiv

  18. [22]

    Videoagenttrek: Computer use pretraining from unlabeled videos.arXiv preprint arXiv:2510.19488, 2025a

    14 Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, et al. Videoagenttrek: Computer use pretraining from unlabeled videos.arXiv preprint arXiv:2510.19488, 2025a. Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Arpo: End-to-end policy optimization for gui agents with e...

    arXiv

  19. [23]

    Brad A Myers

    URLhttps://arxiv.org/abs/2511.04307. Brad A Myers. A brief history of human-computer interaction technology.interactions, 5(2):44–54,

    arXiv

  20. [24]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  21. [25]

    Describes Codex desktop app updates including computer use, built-in browser, image generation, memory, and plugins

    Product announcement. Describes Codex desktop app updates including computer use, built-in browser, image generation, memory, and plugins. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. UI-TARS: Pioneering automated GUI interaction with native agents.arXiv preprint arXiv:...

    Pith/arXiv arXiv

  22. [26]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573,

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573,

    Pith/arXiv arXiv

  23. [27]

    NeuralOS: Towards simulating operating systems via neural generative models.arXiv preprint arXiv:2507.08800,

    Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, and Yuntian Deng. NeuralOS: Towards simulating operating systems via neural generative models.arXiv preprint arXiv:2507.08800,

    arXiv

  24. [28]

    Falcon-ui: Understanding gui before following user instructions.arXiv preprint arXiv:2412.09362,

    15 Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang Ji. Falcon-ui: Understanding gui before following user instructions.arXiv preprint arXiv:2412.09362,

    Pith/arXiv arXiv

  25. [29]

    World-model- augmented web agents with action correction.arXiv preprint arXiv:2602.15384,

    Zhouzhou Shen, Xueyu Hu, Xiyun Li, Tianqing Fang, Juncheng Li, and Shengyu Zhang. World-model- augmented web agents with action correction.arXiv preprint arXiv:2602.15384,

    arXiv

  26. [30]

    Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment.arXiv preprint arXiv:2507.05720,

    Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu. Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment.arXiv preprint arXiv:2507.05720,

    Pith/arXiv arXiv

  27. [31]

    Generation enhances understanding in unified multimodal models via multi-representation generation

    Zihan Su, Hongyang Wei, Kangrui Cen, Yong Wang, Guanhua Chen, Chun Yuan, and Xiangxiang Chu. Generation enhances understanding in unified multimodal models via multi-representation generation. arXiv preprint arXiv:2601.21406,

    Pith/arXiv arXiv

  28. [32]

    Clawgui: A unified framework for training, evaluating, and deploying gui agents, 2026a

    Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Clawgui: A unified framework for training, evaluating, and deploying gui agents, 2026a. URLhttps: //arxiv.org/abs/2604.11784. Zhengyang Tang, Yuxuan Liu, et al. Phoneworld: Scaling phone-use agent environments.arXiv preprint arXiv:2605.29486, 2026b. Kimi Team, T...

    Pith/arXiv arXiv

  29. [33]

    Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, and Hao Tian

    arXiv:2404.07972. Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, and Hao Tian. Video2gui: Synthesizing large-scale interaction trajectories for generalized gui agent pretraining.arXiv preprint arXiv:2605.14747,

    Pith/arXiv arXiv

  30. [34]

    Mobile-agent-v3.5: Multi-platform fundamental gui agents, 2026a

    Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, Zhiyuan Chen, Jitong Liao, Qi Zheng, Jiahui Zeng, Ze Xu, Shuai Bai, Junyang Lin, Jingren Zhou, and Ming Yan. Mobile-agent-v3.5: Multi-platform fundamental gui agents, 2026a. URL https://arxiv.org/abs/2602.16855. Weikai Xu, Kun Huang, Yu...

    arXiv

  31. [35]

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li

    URLhttps://arxiv.org/abs/2601.15876. Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-UI: Visual grounding for GUI instructions.arXiv preprint arXiv:2412.16256,

    arXiv

  32. [36]

    Zhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han, Hang Xu, Chun-Mei Feng, Jianwei Ma, and Wangmeng Zuo

    URLhttps://arxiv.org/abs/2510.17790. Zhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han, Hang Xu, Chun-Mei Feng, Jianwei Ma, and Wangmeng Zuo. Cgl: Advancing continual gui learning via reinforcement fine-tuning.arXiv preprint arXiv:2603.02951,

    Pith/arXiv arXiv

  33. [37]

    Mobile-agent-v3: Fundamental agents for gui automation.arXiv preprint arXiv:2508.15144,

    Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation.arXiv preprint arXiv:2508.15144,

    Pith/arXiv arXiv

  34. [38]

    Seerguard: A safety framework for mobile gui agents via world model prediction.arXiv preprint arXiv:2607.15550,

    Xue Yu, Bo Yuan, Pengjian Yang, Kailin Zhao, Hong Hu, and Junlan Feng. Seerguard: A safety framework for mobile gui agents via world model prediction.arXiv preprint arXiv:2607.15550,

    Pith/arXiv arXiv

  35. [39]

    Agent learning via early experience.arXiv preprint arXiv:2510.08558, 2025a

    Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience.arXiv preprint arXiv:2510.08558, 2025a. 17 Wenyao Zhang, Bozhou Zhang, Zekun Qi, Wenjun Zeng, Xin Jin, and Li Zhang. Disentangled robot learning via separate forward and inverse dynamics pretra...

    Pith/arXiv arXiv

  36. [40]

    Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, et al

    URLhttps://openreview.net/forum?id=DdrsHWobR1. Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, et al. Agentcpm-gui: Building mobile-use agents with reinforcement fine-tuning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 1...

    arXiv 2025

  37. [41]

    Code2world: A gui world model via renderable code generation.arXiv preprint arXiv:2602.09856,

    Yuhao Zheng, Li’an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, and Kevin Qinghong Lin. Code2world: A gui world model via renderable code generation.arXiv preprint arXiv:2602.09856,

    arXiv

  38. [42]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. InICLR, 2025a. Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jing...

    arXiv

  39. [2023]

    CWM: An open-weights llm for research on code generation with world models.arXiv preprint arXiv:2510.02387,

    FAIR CodeGen team, Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, et al. CWM: An open-weights llm for research on code generation with world models.arXiv preprint arXiv:2510.02387,

    arXiv

  40. [2024]

    Xiaomi-gui-0 technical report.arXiv preprint arXiv:2606.31410, 2026a

    Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, et al. Xiaomi-gui-0 technical report.arXiv preprint arXiv:2606.31410, 2026a. Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao, and Wan Guanglu. MobileDreamer: Generative sketch world model for GUI agent.arXiv preprint arXiv:2601.04035, 2026b. Yue Cao, Yingyao ...

    Pith/arXiv arXiv

  41. [2025]

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo

    URLhttps://arxiv.org/abs/2512.21302. Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation.arXiv preprint arXiv:2410.13232,

    arXiv

  42. [2026]

    Patchworld: Gradient-free optimization of executable world models

    Jiaxin Bai, Yue Guo, Yifei Dong, et al. Patchworld: Gradient-free optimization of executable world models. arXiv preprint arXiv:2605.30880,

    Pith/arXiv arXiv