Artificial Intelligence · arXiv:2601.04035 v2 · submitted 7 Jan 2026
MobileDreamer: Generative Sketch World Model for GUI Agent
The paper says
“Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23 · no language-model review
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.12 · human participants 0.05 · controlled comparison 0.31 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.80 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Mobile GUI agents have shown strong potential in real-world automation and practical applications. | background 1.00 | 0.06 | 0.02 | asserted 0.99 |
| However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. | gap 0.66 | 0.06 | 0.02 | asserted 0.99 |
| Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. | background 0.90 | 0.09 | 0.09 | asserted 0.87 |
| This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. | background 0.54 | 0.06 | 0.02 | asserted 0.99 |
| In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. | contribution 0.99 | 0.87 | 0.19 | asserted 0.99 |
| It consists of textual sketch world model and rollout imagination for GUI agent. | method 0.83 | 0.65 | 0.04 | asserted 1.00 |
| Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. | method 0.98 | 0.90 | 0.05 | asserted 1.00 |
| The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. | method 0.95 | 0.73 | 0.06 | asserted 1.00 |
| Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. | result 1.00 | 0.96 | 0.65 | asserted 1.00 |
| World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements. | result 0.99 | 0.95 | 0.12 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,406 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 6 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents cs.CV · Oct 2026
- GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents cs.AI · Oct 2026
- AutoGUIWorld: Image Generators as Visual World Models for GUI Agent cs.CV · Oct 2026
- BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents cs.AI · Sep 2026
- WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning cs.AI · Aug 2026
- Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use cs.RO · Oct 2026
How the literature uses it
11 arXiv papers cite this in 15 sentences
Papers citing it · 8 sentences without a confident label
Uses its method or tool 1 paper
-
When a visual state predictor is available (, MobileDreamer [ref] ), the agent generates $N$ candidate actions and VisCritic scores each based on the predicted post-action state $ŝ_t+1^(a)$, rendered as simplified layout images.
VisCritic: Visual State Comparison as Process Reward for GUI Agents · Best-of-N Selection (Pre-Execution, Optional). · 0.94
Cites it as support 1 paper
-
This abstraction discards critical GUI information, including fine-grained spatial layout and visual attributes (e.g., iconography, typography, and color) [ref] .
Generative Visual Code Mobile World Models · Introduction · 0.96
Cites it as background 5 papers
-
MobileWorld [ref] represents mobile interface transitions as structured natural language triplets, while MobileDreamer [ref] extends textual world models for multi-step rollout planning over candidate actions.
SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction · World Models · 0.83
-
Recently, a number of GUI world models have emerged that predict future UI states to score or search over candidate actions before execution [ref] , including MobileDreamer's recursive tree-of-prediction over candidate actions and states.
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees · Related work · 0.99
-
GUI approaches fall into three categories: text-based (language/DOM prediction [ref] ), pixel-based (diffusion-generated screenshots [ref] ), and code-based (renderable HTML [ref] ).
WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning · Related Work · 0.97
What it cites · 21 sentences without a confident label
Uses the method or tool of 2 papers
-
To convert raw screenshots into structured textual sketches, we annotate each screenshot using PaddleOCR-VL [ref] and represent the GUI state as a set of UI elements, each associated with its label, recognized text, and position (bounding box).
2510.14528 · Experiments on World Model · 1.00
-
We also use Qwen3-8B [ref] as the backbone of our textual sketch world model, and report results with and without the proposed order-invariant learning objective.
Qwen3 Technical Report · Experiments on World Model · 0.97
Compares with 4 papers
-
To isolate the benefit of MobileDreamer, we also evaluate several backbone models as direct GUI agents, including Qwen3-VL-Plus [ref] , GPT-4.1 [ref] , GPT-4o [ref] , Gemini-3-Flash [ref] , and Claude-Sonnet-4.5 [ref] .
GPT-4o System Card · Experiments on mobile GUI Agent · 0.97
-
We compare against previous methods that incorporate a world model for planning, including ViMo's image-based world model [ref] and MobileWorld's text-based world model [ref] .
ViMo: A Generative Visual GUI World Model for App Agents · Experiments on mobile GUI Agent · 1.00
-
We compare against previous methods that incorporate a world model for planning, including ViMo's image-based world model [ref] and MobileWorld's text-based world model [ref] .
MobileWorldBench: Towards Semantic World Modeling For Mobile Agents · Experiments on mobile GUI Agent · 1.00
Cites as background 11 papers
-
UI-TARS [ref] introduces a native agent approach with long-term memory and reflection mechanisms for iterative improvement.
UI-TARS: Pioneering Automated GUI Interaction with Native Agents · GUI Agents · 0.87
-
MobileWorld [ref] represents transitions as natural language triplets.
MobileWorldBench: Towards Semantic World Modeling For Mobile Agents · World Models · 0.82
-
WMA [ref] predicts important state changes through transition-focused abstraction.
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation · World Models · 0.90
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment