Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

A single vision-language model that predicts the next mobile screen as renderable web code outperforms general-purpose models over 50 times larger on mobile GUI world modeling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:36 UTC pith:6ITMBXYH

load-bearing objection Solid, useful mobile-GUI world-model paper, but the headline '50.25x larger' ratio is inflated by counting inactive MoE parameters and needs correction. the 4 major comments →

arxiv 2602.01576 v2 pith:6ITMBXYH submitted 2026-02-02 cs.LG cs.AIcs.CV

Generative Visual Code Mobile World Models

Woosung Koh , Sungjun Han , Segyu Lee , Se-Young Yun , Jamin Shin This is my paper
classification cs.LG cs.AIcs.CV
keywords mobile GUI world modelsrenderable code generationvision-language modelnext-state predictionweb code state representationbenchmarkdata scalinginstruction accuracy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the right way to build a visual world model for mobile interfaces is to generate renderable web code, not pixels. The authors argue that text-only world models discard layout and visual detail, while image-generation models cannot render legible text or respect UI dynamics. Instead, a single vision-language model is trained to take a screenshot and an action and output the HTML/CSS of the next screen, which is then rendered to pixels. On six benchmarks, the resulting models (8B and 32B parameters) top models up to 50.25 times larger, and the gains transfer to downstream action selection. If true, this gives mobile agents a cheap, open, single-model way to imagine the future state of a screen.

Core claim

gWorld is the first open-weight visual mobile GUI world model built on renderable code generation. Given a screenshot St and a coordinate-space action At, a single VLM outputs a reasoning trace and executable web code for the next state St+1, which renders to an image. The training data is synthesized automatically: offline policy trajectories are repurposed into (St, At, St+1) triplets, the ground-truth next screen is re-labeled from pixels to web code by a frontier VLM, and reasoning traces are generated with free look-ahead to the true next state. The authors claim this yields a new accuracy-per-parameter frontier: gWorld 8B and 32B outperform 8 frontier open-weight image- and code-genera

What carries the argument

Renderable web code is the central object: the model never predicts raw pixels, only HTML/CSS that can be rendered to a screen image. This lets the VLM exploit its linguistic priors for precise text and its pretraining on structured code for layout fidelity. The supporting machinery is a data-generation pipeline that (1) repurposes existing mobile-agent trajectories into state–action–next-state triplets, (2) re-labels the next-state image into web code through a frontier VLM, and (3) adds look-ahead reasoning traces that decompose the task into 'what changes' then 'what it looks like.'

Load-bearing premise

The load-bearing premise is that web code can faithfully represent every action-relevant visual detail of a mobile screen; when code serialization loses information — as the paper concedes for photo-rich content such as video or camera imagery — the world model cannot represent those transitions, and the reported accuracy describes the code, not the true next screen.

What would settle it

Take 100 transitions whose next screen contains a large photographic region (camera viewfinder, video player, map tiles), render gWorld's generated HTML, and compare those regions pixel-wise to ground truth. If the code model's image regions fail to reproduce the photographic content while a pixel-diffusion model with OCR-postprocessing succeeds, then 'world modeling via renderable code' is incomplete exactly where visual information matters most.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The resulting models set a new pareto frontier in accuracy versus parameter count, beating frontier open-weight models up to 50.25× larger on all six benchmark splits.
  • Instruction accuracy rises with training-data size following a power law (R²≈0.95); only about 7% of the available repurposed trajectories have been used so far.
  • Code-based generation cuts structural render failures to below 1% on average, where frontier VLM baselines reach 30–40%.
  • Plugging the 8B world model into a policy agent's value estimation improves step-wise action selection by roughly 22 absolute points, and world-model accuracy correlates with downstream gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The code-as-state idea likely transfers beyond mobile: any UI domain with regular structure (web, desktop, automotive, smart-home) where pixel generation is brittle could use the same image-to-code relabeling recipe.
  • Because rendered code is inspectable and executable, gWorld could act as a simulator for training GUI agents without a persistent Android emulator, removing the device–GPU bottleneck in rollout collection — an application the paper mentions but does not test.
  • The reported IAcc. metric is scored by VLM judges on rendered images, so it may be insensitive to photo-realistic content that web code cannot express (e.g., video frames, camera feeds); a human or pixel-level evaluation on imagery-heavy screens would test this blind spot.
  • The projected 3.7M-transition dataset, if realized, could push the current power-law trend considerably further, likely shifting the pareto frontier and making world-model-guided RL more practical at 8B scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes gWorld, a visual mobile GUI world model that predicts the next screen state as executable web code rather than pixels. The method fine-tunes Qwen3-VL 8B and 32B on a large SFT dataset (260K samples) assembled by repurposing existing mobile-agent trajectories, converting ground-truth next screens into HTML via Gemini 3 Flash (cross-modal re-labeling), and adding look-ahead reasoning traces. The authors introduce MWMBENCH, a benchmark with four in-distribution and two out-of-distribution test sets, and report that gWorld sets a new accuracy-vs-size Pareto frontier, outperforming eight frontier open-weight models that are up to 50.25× larger. Additional experiments show scaling trends with dataset size, ablations supporting each pipeline component, and downstream policy gains when the world model is used for value estimation.

Significance. If the results hold, this is a meaningful step: a single, open-weight VLM can generate renderable code that preserves both textual and structural GUI fidelity, avoiding the multi-stage external-model pipelines of prior visual world models. The paper ships open data and benchmark assets, uses three independent VLM judges with per-judge results, and includes downstream policy evaluation. The code-based representation is a plausible and practically valuable middle ground between text-only and pixel-generation world models. The main significance depends on the integrity of the evaluation and on whether the efficiency claim is not overstated.

major comments (4)
  1. [§3 (MWMBENCH) and §2.5] The in-distribution test sets are described as 'randomly sample 500 world modeling instances from trajectories' in AitW, GUIO, AC, and AMEX, which are the same datasets used to build the 260K training set. No episode-level hold-out is stated. If the random sampling is at the transition level, training examples and test examples can come from the same episode or even the same state-action pairs, inflating ID accuracy in Table 2. Please either adopt episode-level splitting or provide explicit evidence that no transition overlap exists between training and test.
  2. [Abstract and §4.2 (Table 2, Fig. 1)] The headline 'over 50.25× larger' is based on total parameter count for Llama 4 402B-A17B, which is a MoE model with only 17B active parameters, as the table itself labels it. Using active parameters, the ratio for gWorld 8B vs Llama 4 is about 2.1×, and gWorld 32B has more active parameters than Llama 4 402B-A17B (32B vs 17B). The Pareto frontier may survive, but the central efficiency claim and Figure 1's x-axis (labeled 'Parameter Size') should transparently report both total and active parameter counts, or use active parameters for MoE baselines.
  3. [§4.1 (Instruction Accuracy metric)] IAcc is computed as the mean of three VLM judges, one of which is Gemini 3 Flash — the same model family used to generate the training targets (π* in §2.4–2.5). Although per-judge results and inter-judge agreement (Fig. 11–12) provide some mitigation, judge-family bias is not fully ruled out. Please report the primary results with Gemini 3 Flash excluded, or add a human-agreement study on a subset, so the reader can judge the metric's independence.
  4. [§2.3 and Appendix E] The core assumption — that web code faithfully captures the action-relevant visual content of a mobile GUI — is acknowledged to be lossy for photo-rich screens (video players, natural images). Since the evaluation renders the generated code back to images, IAcc and Similarity on such screens measure fidelity to the code representation, not to the original screen. Please quantify the fraction of MWMBENCH transitions that involve photo/video-heavy content and report whether gWorld's advantage persists on those cases. This would clarify the scope of the reported accuracy.
minor comments (6)
  1. [§4.4 (Fig. 5)] The caption and text claim monotonic gains, but the plotted points near the right end appear non-monotonic in the figure. Please clarify which values correspond to which dataset and whether the reported 'monotonic' statement refers to the power-law fit rather than the observed points. Also specify the number of random seeds.
  2. [Fig. 1] The x-axis is 'Parameter Size (log10)', which for MoE models is ambiguous. Please add an axis note that total parameters are used, and consider a second axis for active parameters.
  3. [Table 2] Minor inconsistencies: 'MWMBench-AitW' vs 'MWMBENCH-AITW' and 'MWMBench-Korean' vs 'MWMBENCH-KAPPS' should be harmonized. The header 'Similarly' appears to be a typo for 'Similarity'.
  4. [§5] The comparison with VIMO ('VIMO achieves 74%') would benefit from specifying that this is the same evaluation subset and the same similarity metric version; otherwise the reader cannot assess comparability.
  5. [Appendix C] Please clarify whether the learning rate 2e-7 is applied to both the LLM and the MLP projector, and whether any layer-wise decay is used. Also, the statement 'max image pixels 4,233,600' could briefly explain how this handles high-resolution mobile screens.
  6. [Global] Typos and grammar issues: 'Futhermore' (§2.3), 'GPU sit idle' (§5), and inconsistent capitalization of 'MWMBench' vs 'MWMBENCH'. A thorough proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity: the SFT/render/evaluate chain is self-contained; a minor judge-model overlap does not reduce the central claim.

full rationale

The paper's derivation chain is not circular. Training data are built by repurposing offline trajectories into (St, At, St+1) pairs, converting the ground-truth next-state image into renderable code with Gemini 3 Flash, and SFT-ing Qwen3-VL to predict (reasoning, code) from (St, At). At inference the model has not seen St+1, so no reported accuracy is a fitted parameter renamed as a prediction. The only overlap is that Gemini 3 Flash both generated the code targets (Sec. 2.5) and is one of three IAcc judges (Sec. 4.1). However, IAcc is averaged with GPT-5 Mini and Claude 4.5 Haiku, per-judge scores are reported in Tab. 9 with high inter-judge agreement (Figs. 11-12), and the rendered outputs are also scored against ground-truth images via DINO similarity and on external OOD benchmarks (AndroidWorld, KApps). Thus the pareto-frontier claim does not reduce to agreement with the training-label generator. There is no load-bearing self-citation chain: the human-correlation claim for IAcc cites external VIMO work, and the scaling-law references are standard external results. The abstract's '50.25x larger' claim uses total parameters for MoE baselines rather than active parameters, which is a correctness/metric issue, not a circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The pipeline introduces no new physical entities. It relies on existing VLMs, a new serialization target (web code), and a hand-tuned evaluation prompt. The main external dependencies are Gemini 3 Flash as data generator and three frontier VLM judges.

free parameters (3)
  • Power-law exponent b = 0.074–0.166 per benchmark
    Fit to 4 data points in Fig. 5; used to project gains to a 3.7M-sample dataset. With only 4 points the fit is over-parameterized for extrapolation.
  • Similarity threshold for deduplication = 0.997
    Hand-chosen threshold in §B.1 for identifying duplicate transitions in AndroidWorld; affects the composition of the OOD benchmark.
  • Evaluation prompt P_WM = curated for Qwen3 VL 235B-A22B
    The evaluation prompt was optimized on a specific model (§C), introducing a model-dependent factor into all baseline comparisons.
axioms (4)
  • domain assumption Gemini 3 Flash's image-to-code conversion is a faithful surrogate for the true next-state image content.
    §2.4(2) builds all training targets on this conversion; any systematic error (e.g., dropped icons, mangled text) becomes a bias in the learned world model.
  • domain assumption VLM-as-a-judge IAcc. is a valid proxy for human judgment of next-state correctness.
    §4.1 relies on the correlation reported in Luo et al. (2025) but does not re-validate on MWMBENCH with human ratings.
  • domain assumption Web code rendering in a mobile viewport captures the perceptual and semantic content of GUI screenshots.
    This is the core representation assumption of §2.3 and is applied in every rendered evaluation image; the authors themselves note in Appendix E that it fails for photo-realistic content.
  • domain assumption The four offline datasets (AitW, GUIO, AC, AMEX) provide transitions that are independent of the ID test samples.
    §2.4 and §3: the paper does not describe an episode-level partition between the 260K training samples and the 500-sample ID test sets, so this independence is not established.

pith-pipeline@v1.3.0-alltime-deepseek · 27996 in / 14079 out tokens · 143347 ms · 2026-08-03T05:36:23.789755+00:00 · methodology

0 comments
read the original abstract

Mobile Graphical User Interface (GUI) World Models (WMs) offer a promising path for improving mobile GUI agent performance at train- and inference-time. However, current approaches face a critical trade-off: text-based WMs sacrifice visual fidelity, while the inability of visual WMs in precise text rendering led to their reliance on slow, complex pipelines dependent on numerous external models. We propose a novel paradigm: visual world modeling via renderable code generation, where a single Vision-Language Model (VLM) predicts the next GUI state as executable web code that renders to pixels, rather than generating pixels directly. This combines the strengths of both approaches: VLMs retain their linguistic priors for precise text rendering while their pre-training on structured web code enables high-fidelity visual generation. We introduce gWorld (8B, 32B), the first open-weight visual mobile GUI WMs built on this paradigm, along with a data generation framework (gWorld) that automatically synthesizes code-based training data. In extensive evaluation across 4 in- and 2 out-of-distribution benchmarks, gWorld sets a new pareto frontier in accuracy versus model size, outperforming 8 frontier open-weight models over 50.25x larger. Further analyses show that (1) scaling training data via gWorld yields meaningful gains, (2) each component of our pipeline improves data quality, and (3) stronger world modeling improves downstream mobile GUI policy performance.

Figures

Figures reproduced from arXiv: 2602.01576 by Jamin Shin, Segyu Lee, Se-Young Yun, Sungjun Han, Woosung Koh.

Figure 1
Figure 1. Figure 1: Average Instruction Accuracy (IAcc.) across all six benchmarks. gWorld 8B and 32B achieve a new pareto frontier in terms of model size (log10 scaled). The existing pareto frontier was defined by Qwen3 VL 8B, 32B, and GLM 4.6V 106B. Notably, extremely large models (e.g., Llama 4 402B) do not reach this pareto frontier, while text-image-to-image models (e.g., Emu3.5 34B) struggle with mobile GUI dynamics. ar… view at source ↗
Figure 2
Figure 2. Figure 2: Mobile GUI world modeling via renderable code. Given an image state St and action At, the model predicts the next state St+1. Our model, gWorld, generates renderable web code to ensure pixel-perfect text and structurally accurate layouts. In contrast, image-gen baselines (e.g., Qwen-Image-Edit 20B) struggle with the discrete nature of GUIs, frequently producing illegible text and distorted layouts. See App… view at source ↗
Figure 3
Figure 3. Figure 3: Schematic diagram of our data generation pipeline. We construct VLM world modeling data via three steps: (1) Repurposing offline policy trajectories into transition triplets; (2) Cross-modal relabeling of the ground-truth next state from pixels (S image t+1 ) to renderable web code (S code t+1 ); and (3) Synthesizing reasoning traces (Rt) using look-ahead access to the target state. The final training obje… view at source ↗
Figure 4
Figure 4. Figure 4: Correlation between input-output similarity and model performance. Top: Pearson correlation ρ between Sim(St, St+1) and Sim(Sˆt+1, St+1). Image generation models show strong posi￾tive correlations (ρ > 0.7), suggesting output quality largely de￾pends on how similar St and St+1 already are. Bottom: Sim(Sˆt+1, St+1) − Sim(St, St+1) vs. Sim(St, St+1), with the gray line indicating the score ceiling. Emu3.5 34… view at source ↗
Figure 5
Figure 5. Figure 5: Data scaling laws for mobile world modeling at 8B. We fit power-law curves (y = axb ) to the test performance across five distinct benchmarks as a function of training dataset size. The high coefficients of determination (R 2 ≥ 0.94 for most splits) indicate a predictable and non-saturating relationship between data scale and performance. This suggests that our data generation pipeline has not yet reached … view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on Rt train data quality. Our method con￾sistently outperforms the naïve alternative in terms of IAcc. Both models are trained on 37K samples on top the base model Qwen3 VL 8B. 4.5. Further Analysis: Ablation Ablation Analysis on Generating S code t+1 . For our first ablative study, we examine whether the synthetic cross￾model policy trajectory repurposed training (see Step 1 and 2 in [PITH_FULL_… view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of transitions in MWMBench’s OOD splits. Left: per-app transition counts (colors indicate domain). Right: domain-wise distribution. AndroidWorld is dominated by productivity apps (55.4%), while KApps shows a more balanced distribution across food & shopping, communication, and productivity domains. This process reduces the dataset from 1,094 to 686 transitions (37% reduction), removing redunda… view at source ↗
Figure 8
Figure 8. Figure 8: Data collection interface software built for KApps. Prompt P WM: World Model Evaluation You are an expert mobile UI World Model that can accurately predict the next state given an action. Given a screenshot of a mobile interface and an action, you must generate clean, responsive HTML code that represents the state of the interface AFTER the action is performed. First generate reasoning about what the next … view at source ↗
Figure 9
Figure 9. Figure 9: Data scaling analysis. We report average Instruction Accuracy (IAcc.) across the four in-distribution test splits as we scale the repurposed training data from 37K to 240K examples. The results demonstrate strong positive scaling. significantly enhance performance by scaling the training data to utilize the full set of available offline trajectories. Second, there are fundamental limitations to rendering p… view at source ↗
Figure 10
Figure 10. Figure 10: Data scaling analysis. We report average Instruction Accuracy (IAcc.) in MWMBENCH-ANDROIDWORLD. as we scale the repurposed training data from 37K to 240K examples. The results demonstrate strong positive scaling. 0 20 40 60 80 Gemini 3 Flash 0 20 40 60 80 GPT-5 Mini = 0.95 = 0.82 0 20 40 60 80 Gemini 3 Flash 0 20 40 60 80 Claude Haiku 4.5 = 0.91 = 0.76 0 20 40 60 80 GPT-5 Mini 0 20 40 60 80 Claude Haiku 4… view at source ↗
Figure 11
Figure 11. Figure 11: Inter-judge agreement on world model instruction-following. Each scatter plot compares Instruction-following Accuracy (IAcc., %) scored by two different VLM-as-a-Judge models (Gemini 3 Flash, GPT-5 Mini, Claude Haiku 4.5) across MWMBENCH datasets (AitW, GUIOdyssey, AndroidControl, AMEX, AndroidWorld, KApps). Each point corresponds to a (WM model, dataset) result, colors denote datasets, and the dashed lin… view at source ↗
Figure 12
Figure 12. Figure 12: Consistency of model rankings across VLM-as-a-Judge choices. Bump chart shows the relative rank (1 = best) of each world model under different judges (Gemini 3 Flash, GPT-5 Mini, Claude Haiku 4.5), where ranks are determined by average IAcc. over MWMBENCH datasets. Rankings remain largely stable across judges, with gWorld consistently achieving top performance compared to other code-generation and image-g… view at source ↗
Figure 13
Figure 13. Figure 13: Additional qualitative example 1. Input “Swipe Up” gWorld (ours) Qwen-Image-Edit Ground Truth [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Additional qualitative example 2. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Additional qualitative example 3. Input Ground Truth Llama4 402B-A17B Emu 3.5 Ours (32B) [PITH_FULL_IMAGE:figures/full_fig_p028_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Additional qualitative example 4. The red marker on the input is for visualization only and was not provided to the model. Action: click at normalized coordinates (802, 394). 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Additional qualitative example 6 (Android World). The red marker on the input is for visualization only and was not provided to the model. Action: click at normalized coordinates (756, 685). 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Additional qualitative example 7 (Android World). The red marker on the input is for visualization only and was not provided to the model. Action: click at normalized coordinates (819, 549). Input Ground Truth Llama4 402B-A17B Emu 3.5 Ours (32B) [PITH_FULL_IMAGE:figures/full_fig_p030_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Additional qualitative example 8. The red marker on the input is for visualization only and was not provided to the model. Action: TAP at normalized coordinates (857, 421). 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Same analysis as [PITH_FULL_IMAGE:figures/full_fig_p033_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond

    cs.AI 2026-04 unverdicted novelty 7.0

    Proposes a levels x laws taxonomy for world models in AI agents, defining L1-L3 capabilities across physical, digital, social, and scientific regimes while reviewing over 400 works to outline a roadmap for advanced ag...

  2. Scaling GUI Agents with Visual State Transitions

    cs.AI 2026-07 conditional novelty 6.0

    A joint inverse-forward pretraining stage on visual screen transitions improves GUI-agent fine-tuning by 0.6–6.2 percentage points across three benchmarks.

  3. Qwen-AgentWorld: Language World Models for General Agents

    cs.CL 2026-06 unverdicted novelty 6.0

    Qwen-AgentWorld are language world models that simulate multi-domain agent environments and boost general agent capabilities via decoupled RL simulation and unified foundation model training.

  4. How Mobile World Model Guides GUI Agents?

    cs.AI 2026-05 unverdicted novelty 6.0

    Mobile world models in text, image, and code modalities reach state-of-the-art on their benchmarks and improve downstream GUI agent performance, with code best for in-distribution accuracy and text more robust for out...

  5. Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection

    cs.CR 2026-04 unverdicted novelty 6.0

    Semantic-level UI Element Injection distracts GUI agents by overlaying safety-aligned UI elements, achieving up to 4.4x higher attack success rates that transfer across models and create persistent attractors.

  6. How Mobile World Model Guides GUI Agents?

    cs.AI 2026-05 unverdicted novelty 4.0

    World models trained on delta text, full text, diffusion images, and renderable code achieve SoTA on two benchmarks and improve downstream GUI agent performance on three mobile datasets with modality-specific strengths.

  7. Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond

    cs.AI 2026-04 conditional novelty 4.0

    A survey proposing a three-level capability taxonomy (L1 Predictor, L2 Simulator, L3 Evolver) for world models across physical, digital, social, and scientific domains.

Reference graph

Works this paper leans on

41 extracted references · 3 linked inside Pith · cited by 5 Pith papers

  1. [1]

    REASONING: Analyze the screenshot and plan the HTML structure

  2. [4]

    For images, use inline SVG placeholders with explicit width and height

  3. [5]

    Make it visually as close to the provided screenshot

  4. [6]

    inferable

    Each button’s function must be "inferable" / "differentiable"

  5. [7]

    reasoning

    All text content from the screenshot must be well-represented. Return ONLY a JSON object with this exact structure: { "reasoning": "Your detailed analysis and planning here", "html": "Your complete HTML code here" } 15 Split #Trans #Apps Lang Example Apps ANDROIDWORLD686 18 EN Productivity:Joplin, Markor, Tasks, Simple Calendar Media:Retro Music, Simple G...

  6. [8]

    Nguyen, D., Chen, J., Wang, Y ., Wu, G., Park, N., Hu, Z., Lyu, H., Wu, J., Aponte, R., Xia, Y ., Li, X., Shi, J., Chen, H., Lai, V

    Accessed: 2026-01-20. Nguyen, D., Chen, J., Wang, Y ., Wu, G., Park, N., Hu, Z., Lyu, H., Wu, J., Aponte, R., Xia, Y ., Li, X., Shi, J., Chen, H., Lai, V . D., Xie, Z., Kim, S., Zhang, R., Yu, T., Tanjim, M., Ahmed, N. K., Mathur, P., Yoon, S., Yao, L., Kve- ton, B., Kil, J., Nguyen, T. H., Bui, T., Zhou, T., Rossi, R. A., and Dernoncourt, F. GUI agents: ...

    2026

  7. [9]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl

    doi:10.18653/v1/2025.findings-acl 2025

  8. [11]

    naacl-long.11/

    URL https://aclanthology.org/2024. naacl-long.11/. Chen, Z., Zhou, Q., Shen, Y ., Hong, Y ., Sun, Z., Gutfre- und, D., and Gan, C. Visual chain-of-thought prompt- ing for knowledge-based visual reasoning.Proceedings of the AAAI Conference on Artificial Intelligence, 38 (2):1254–1262, Mar. 2024c. doi: 10.1609/aaai.v38i2. 27888. URL https://ojs.aaai.org/ind...

    arXiv 2024

  9. [12]

    ISBN 979-8-89176-189-6

    Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long

    doi:10.18653/v1/2025.naacl-long 2025

  10. [15]

    Focus on these two critical criteria:

    HTML GENERATION: Create the HTML code based on your analysis. Focus on these two critical criteria:

  11. [16]

    inferable

    Each button’s function should be "inferable" / "differentiable" - users must be able to understand what each button does

  12. [17]

    Each text content should be well-represented in the HTML output - all visible text must be accurately captured. In your REASONING, address: • The overall structure and layout of the screen (header, main content, footer, etc.) • Important UI elements and their hierarchy (buttons, text, images, icons, etc.) • Which parts of the screen are most important for...

  13. [19]

    Choose between using inline CSS and utility classes from Bootstrap, Tailwind CSS, or MUI for styling

  14. [25]

    Clicks: Red circle with crosshair + yellow center dot

  15. [26]

    Focus only on the changes that can be predicted from the current screenshots

    Scrolls: Blue line with green start point + red end point or based on direction Generate reasoning on what this next state would look like as if you were only given the current screenshot. Focus only on the changes that can be predicted from the current screenshots. In the reasoning, do not mention the visual annotation of the action or the existence of t...

    2025

  16. [27]

    Provide reasoning about what the next state should look like based on the action

  17. [28]

    Generate complete, valid HTML5 code

  18. [29]

    Choose between using inline CSS and utility classes from Bootstrap, Tailwind CSS, or MUI for styling, depending on which option generates the closest code to the screenshot

  19. [30]

    Use mobile-first design principles matching screenshot dimensions

  20. [31]

    Matching the approximate color is also good

    For images, use inline SVG placeholders with explicit width and height attributes that match the approximate dimensions from the screenshot. Matching the approximate color is also good

  21. [32]

    Use modern web standards and best practices

  22. [33]

    Return ONLY the HTML code, no explanations or markdown formatting

  23. [34]

    The generated HTML should render properly in a mobile viewport

  24. [35]

    Thoughts

    Generated HTML should look like the screen that logically follows the current screen and the action. Action: {action} Output format: Next State Reasoning: <your reasoning about what the next state should look like> HTML: <valid_html_code > Generate the next state reasoning and the next state in html: 19 Metric: Instruction Accuracy.Instruction Accuracy is...

    2025

  25. [36]

    Each alternative MUST be a completely different action from the one already suggested above

  26. [37]

    Do NOT repeat or slightly modify the already-suggested action (e.g., if the suggested action is a TAP at (500, 300), do NOT suggest a TAP at (500, 301) or nearby coordinates)

  27. [38]

    For TAP actions, choose DIFFERENT UI elements to tap, not the same element with slightly different coordinates

  28. [39]

    action_type

    Each alternative should represent a meaningfully different approach to achieving the goal. For each action, explain the reasoning behind it. You must output exactly {num_alternatives} actions numbered 1 to {num_alternatives}: {{1: {{Reason: ..., Action: {{"action_type":...}}}}, ..., {num_alternatives}: {{Reason: ..., Action: {{"ac- tion_type":...}}}}}} 21...

  29. [40]

    Does the action target the correct UI element or area visible on screen?

  30. [41]

    Is the action type appropriate for the current context?

  31. [42]

    Reason":

    How directly does this action advance the goal vs. being a roundabout step? Respond in JSON format: {{"Reason": "Your explanation", "Judgement": "valid" or "invalid", "Confidence": <score>}} IMPORTANT: Use the FULL range of confidence scores to differentiate action quality: • 0.9–1.0: Clearly the optimal action, directly advances the goal • 0.7–0.8: Good ...

  32. [43]

    BEFORE screenshot: The current UI state before the action

  33. [44]

    action_type

    AFTER screenshot: The predicted UI state after performing the action Available action types: • TAP: Tap on a location. Format: {{"action_type": "TAP", "x": <x>, "y": <y>}} • SCROLL: Scroll in a direction. Format: {{"action_type": "SCROLL", "direction": "<up|down|left|right>"}} • TYPE: Type text. Format: {{"action_type": "TYPE", "text": "<text>"}} • BACK: ...

  34. [110]

    findings-acl.110/

    URL https://aclanthology.org/2025. findings-acl.110/. Chen, D., Chen, R., Zhang, S., Wang, Y ., Liu, Y ., Zhou, H., Zhang, Q., Wan, Y ., Zhou, P., and Sun, L. MLLM- as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. InForty-first International Conference on Machine Learning, 2024a. URL https: //openreview.net/forum?id=dbFEFHAD...

    2025

  35. [199]

    naacl-long.199/

    URL https://aclanthology.org/2025. naacl-long.199/. Team, V ., Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., Pan, L., Duan, S., Wang, W., Wang, Y ., Cheng, Y ., He, Z., Su, Z., Yang, Z., Pan, Z., Zeng, A., Wang, B., Chen, B., Shi, B., Pang, C., Zhang, C., Yin, D., Yang, F., Chen, G., Xu, J., Zhu, J., Chen, J., Chen, J.,...

    Pith/arXiv arXiv 2025

  36. [454]

    emnlp-main.454/

    URL https://aclanthology.org/2025. emnlp-main.454/. Feng, J., Zhang, Y ., Zhang, C., Lu, Y ., Liu, S., and Wang, M. Web world models, 2025. URL https://arxiv. org/abs/2512.23676. Gao, Y ., Ye, J., Wang, J., and Sang, J. Websynthesis: World- model-guided mcts for efficient webui-trajectory synthe- sis.arXiv preprint arXiv:2507.04370, 2025. Gou, B., Wang, R...

    arXiv 2025

  37. [1158]

    findings-acl.1158/

    URL https://aclanthology.org/2025. findings-acl.1158/. Niu, R., Ji, J., Chang, Y ., and Wang, Q. Screenexplorer: Training a vision-language model for diverse exploration in open gui world.arXiv preprint arXiv:2505.19095, 2025. Oquab, M., Darcet, T., Moutakanni, T., V o, H. V ., Szafraniec, M., Khalidov, V ., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby,...

    Pith/arXiv arXiv 2025

  38. [2023]

    cc/paper_files/paper/2023/file/ 6dcf277ea32ce3288914faf369fe6de0-Paper-Conference

    URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ 6dcf277ea32ce3288914faf369fe6de0-Paper-Conference. pdf. Lu, Q., Shao, W., Liu, Z., Du, L., Meng, F., Li, B., Chen, B., Huang, S., Zhang, K., and Luo, P. Guiodyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. InProceedings of the IEEE/CVF In- ternational Conferenc...

    Pith/arXiv arXiv 2023

  39. [2024]

    Zhang, Z., Zhang, A., Li, M., hai zhao, Karypis, G., and Smola, A

    URL https://openreview.net/forum? id=hFVpqkRRH1. Zhang, Z., Zhang, A., Li, M., hai zhao, Karypis, G., and Smola, A. Multimodal chain-of-thought reasoning in language models.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:// openreview.net/forum?id=y1pPWFVfvR. Zhang, Z., Lu, Y ., Fu, Y ., Huo, Y ., Yang, S., Wu, Y ., Si, H., Cong...

    2024

  40. [2025]

    Chai, Y ., Huang, S., Niu, Y ., Xiao, H., Liu, L., Wang, G., Zhang, D., Ren, S., and Li, H

    URL https://openreview.net/forum? id=moWiYJuSGF. Chai, Y ., Huang, S., Niu, Y ., Xiao, H., Liu, L., Wang, G., Zhang, D., Ren, S., and Li, H. AMEX: Android multi- annotation expo dataset for mobile GUI agents. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Findings of the Association for Computational Linguis- tics: ACL 2025, pp. 2138–21...

    2025

  41. [8856]

    Gui, Y ., Li, Z., Zhang, Z., Wang, G., Lv, T., Jiang, G., Liu, Y ., Chen, D., Wan, Y ., Zhang, H., Jiang, W., Shi, X., and Jin, H

    URL https://openreview.net/forum? id=c6l7yA0HSq. Gui, Y ., Li, Z., Zhang, Z., Wang, G., Lv, T., Jiang, G., Liu, Y ., Chen, D., Wan, Y ., Zhang, H., Jiang, W., Shi, X., and Jin, H. Latcoder: Converting webpage design to code with layout-as-thought. InProceed- ings of the 31st ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining V .2, KDD ’25, pp....

    arXiv 2025