Pith.

Artificial Intelligence · arXiv:2601.04035 v2 · submitted 7 Jan 2026

MobileDreamer: Generative Sketch World Model for GUI Agent

Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Siran Dai, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi

show 1 more author

Wenji Mao

The paper says

“Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%.”

from the abstract · picked out as the main claim by a classifier

Pith says

Pith has not reviewed this paper.

Get a Pith Review, $5

Reviews

Reviews by people

No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23 · no language-model review

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.12 · human participants 0.05 · controlled comparison 0.31 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.80 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Mobile GUI agents have shown strong potential in real-world automation and practical applications. background 1.00 0.06 0.02 asserted 0.99
However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. gap 0.66 0.06 0.02 asserted 0.99
Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. background 0.90 0.09 0.09 asserted 0.87
This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. background 0.54 0.06 0.02 asserted 0.99
In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. contribution 0.99 0.87 0.19 asserted 0.99
It consists of textual sketch world model and rollout imagination for GUI agent. method 0.83 0.65 0.04 asserted 1.00
Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. method 0.98 0.90 0.05 asserted 1.00
The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. method 0.95 0.73 0.06 asserted 1.00
Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. result 1.00 0.96 0.65 asserted 1.00
World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements. result 0.99 0.95 0.12 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,406 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 6 Pith papers
How the literature uses it

11 arXiv papers cite this in 15 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 8 sentences without a confident label

Uses its method or tool 1 paper
  • When a visual state predictor is available (, MobileDreamer [ref] ), the agent generates $N$ candidate actions and VisCritic scores each based on the predicted post-action state $ŝ_t+1^(a)$, rendered as simplified layout images.

    VisCritic: Visual State Comparison as Process Reward for GUI Agents · Best-of-N Selection (Pre-Execution, Optional). · 0.94

Cites it as support 1 paper
  • This abstraction discards critical GUI information, including fine-grained spatial layout and visual attributes (e.g., iconography, typography, and color) [ref] .

    Generative Visual Code Mobile World Models · Introduction · 0.96

Cites it as background 5 papers

What it cites · 21 sentences without a confident label

Uses the method or tool of 2 papers
  • To convert raw screenshots into structured textual sketches, we annotate each screenshot using PaddleOCR-VL [ref] and represent the GUI state as a set of UI elements, each associated with its label, recognized text, and position (bounding box).

    2510.14528 · Experiments on World Model · 1.00

  • We also use Qwen3-8B [ref] as the backbone of our textual sketch world model, and report results with and without the proposed order-invariant learning objective.

    Qwen3 Technical Report · Experiments on World Model · 0.97

Compares with 4 papers
  • To isolate the benefit of MobileDreamer, we also evaluate several backbone models as direct GUI agents, including Qwen3-VL-Plus [ref] , GPT-4.1 [ref] , GPT-4o [ref] , Gemini-3-Flash [ref] , and Claude-Sonnet-4.5 [ref] .

    GPT-4o System Card · Experiments on mobile GUI Agent · 0.97

  • We compare against previous methods that incorporate a world model for planning, including ViMo's image-based world model [ref] and MobileWorld's text-based world model [ref] .

    ViMo: A Generative Visual GUI World Model for App Agents · Experiments on mobile GUI Agent · 1.00

  • We compare against previous methods that incorporate a world model for planning, including ViMo's image-based world model [ref] and MobileWorld's text-based world model [ref] .

    MobileWorldBench: Towards Semantic World Modeling For Mobile Agents · Experiments on mobile GUI Agent · 1.00

Cites as background 11 papers

Challenges and discussion

Know something about this paper? Leave a comment.

No comments yet.

Sign in to comment