Pith.

Artificial Intelligence · arXiv:2610.36877 v1 · submitted 27 Oct 2026 · 63 references

GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

The paper says

“Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text.”

more of the abstract

However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA

from the abstract

Pith says

Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A clean, well-executed paper that replaces VLM token decoding with JEPA latent scoring for GUI actions, achieving real speed gains at modest accuracy cost, with the a11y reliance honestly acknowledged. This line is wrong

3 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 63 references say.

The paper says, at Section 4.4 and Limitations (b124)

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The paper acknowledges that 'Design choices other than the reranker's variant, weight and gate were made on test IDD and re-checked on validation only for the span ranker and overall base-policy accuracy.' This means that most architectural decisions (predictor form, structural head, text matcher design, training hyperparameters) were evaluated on the same IDD test set that reports the headline numbers.

more of objection 1

While the OOD results and mechanism ablations support internal consistency, the absolute IDD accuracy likely carries an optimistic bias from this implicit tuning. The authors should discuss the magnitude of this bias—for example, by reporting accuracy on a held-out mini-split or by showing that a design selected on the validation split gives similar results.

The paper says, at Section 4.3 (b124, b126)

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The same-protocol 7B reference is trained on only 4,000 of the 13,594 training episodes, while GUI-JEPA uses the full set. The paper mentions this but does not discuss how much the 7B baseline might improve with full data or why 4,000 episodes were chosen (presumably computational cost).

more of objection 2

This asymmetry weakens the claim that GUI-JEPA achieves '86–92% of its accuracy' because the 7B reference may be undertrained. The authors should provide evidence that the 7B agent's accuracy saturates at 4,000 episodes, or at least estimate the gap that additional data could close (e.g., by plotting accuracy against training subset size).

The paper says, at Section 4.1 and Table 1

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.64):

“Apart from LiMAC and its two baselines, discussed in the text, none is set against our numbers because input, judge or split differ (ours for reference: 77.64 / 52.85 Low / High on the full test and 83.09 / 60.12 on IDD as 3-seed means, 78.64 / 53.75 and 84.06 / 60.99 for seed 3, under the exact-match judge with a11y input).”

Pith says: Objection 3 of 3 · major

The paper introduces an exact-match judge that differs from the original AndroidControl protocol in four ways (no equivalent actions, normalised text match, accepting click for long-press, ground-truth box from enumerated candidates). While the same-protocol 7B reference uses this judge, the comparison with published numbers (e.g., PaLM 2S) is compromised because those numbers use the original relaxed judge.

more of objection 3

The authors clearly mark these differences, but the risk of misinterpretation is high (e.g., the sentence 'GUI-JEPA matches it on AC-Low (84.06 vs 83.2)' appears in the main text without reminding readers that the judges differ). I recommend adding a sentence in the main result paragraph (Section 4.2) that restates the judge discrepancy for clarity.

7 minor objections

The paper says, at Section 3 (first use of IDD)

Pith says: minor objection 1 of 7

The term 'IDD' is used without definition before Section 4.1. Define it at first occurrence (e.g., 'in-distribution (IDD)').

The paper says, at Figure 2 caption

Pith says: minor objection 2 of 7

'0.50B GUI-JEPA: score all candidatesin one pass' has a missing space before 'in'.

show 5 more

The paper says, at Section 5.1 (b102)

Pith says: minor objection 3 of 7

The claim 'The pathway is trained for the decision' is supported by the random-initialisation experiment, but the phrase could be misinterpreted as 'the training objective directly optimises the predictor for decision quality' (which it does, via the policy loss). The paper already explains this, but a slight rephrasing (e.g., 'the predictor learns what the policy needs') would avoid overstatement.

The paper says, at Table 3 footnote

Pith says: minor objection 4 of 7

The mem×ours column: '≥: vLLM pre-allocates at least the weights.' Clarify whether the reported number for vLLM is peak or allocated; the 7B agent's memory of ≥16 GB seems to be allocation, while GUI-JEPA is measured peak. This should be consistent.

The paper says, at Section 4.5 (b439)

Pith says: minor objection 5 of 7

All real-step timings are AC-High; AC-Low would have an additional CLIP call per step. Mention in the main text that AC-Low latency will be higher, and roughly how much (the appendix gives 1.3 ms amortised, but the per-step cost is about 5.6 ms).

The paper says, at Appendix B (b300)

Pith says: minor objection 6 of 7

The bootstrap intervals are useful, but the sentence 'the 95% t-interval over seeds covers training variance' could be clarified: for n=3, the t-interval is wide and the sample is small; the bootstrap intervals over episodes may be more informative. Consider merging the two uncertainty treatments in Table 4.

The paper says, at Section 4.3 (b80)

Pith says: minor objection 7 of 7

'The gap is 7.0 points when the step instruction is given and 9.9 with the goal alone.' These numbers refer to AC-Low and AC-High respectively. Specify which is which for readers scanning.

What would settle it

Run an experiment in which the a11y candidate text is systematically corrupted (e.g., 50% random character substitution) on a held-out IDD split. If the accuracy drops by more than 5 points, the text matcher's robustness to noisy a11y text is weaker than assumed; if less than 2 points, the matcher is more resilient than the limitation suggests. This directly tests the boundary of the acknowledged a11y reliance.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 28 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 37 s · 38,800 tokens read, 4,114 written
  3. Editor wrote the verdict above 11 s · 39,568 tokens read, 932 written
  4. Referee minor revision: 3 major objections, 7 minor 31 s · 39,071 tokens read, 3,340 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here is the honest take on GUI-JEPA.

What is new and good: this is the first GUI agent that uses a JEPA as the policy itself, scoring candidate actions in latent space without ever decoding a token. The key design—candidate-conditioned latent prediction with a frozen encoder, a residual predictor, and lightweight scoring heads—is novel and practically motivated. The speed numbers are real: 51–74 ms per step on a single H20 versus 207–507 ms for a 7B VLM, at 86–92% of its IDD accuracy with 6% of the parameters. The ablations are thorough. The mechanism analysis showing that the predictor learns a decision-oriented representation (not a good forward model) and that a random initialisation works as well as dynamics pretraining is genuinely interesting. The limitations section is honest.

Soft spots: the elephant in the room is the a11y tree. It is a privileged input that provides candidate elements with exact bounding boxes and text, which drastically simplifies grounding. The paper acknowledges this, and it is a fair design choice for on-device scenarios where the a11y tree is available, but it means the method is not directly comparable to pixel-only agents. The same-protocol comparison to a 7B VLM that also receives the a11y tree is fair, but the gap to pixel-only published numbers is larger. Design choices (except the click reranker) were made on test IDD; the authors flag this. The evaluation is step-level, not task-level, and excludes screen capture and a11y retrieval from latency. All of these are stated as limitations, so no hidden flaws.

For whom: readers working on efficient GUI agents or on-device deployment. The paper delivers a practical speed-accuracy trade-off under well-defined conditions. I would bring it to a reading group, and I would cite it for the JEPA-as-policy idea and the predictor randomisation result.

Recommendation: the paper deserves a serious referee. Send it to peer review. The central claim is substantiated; the a11y reliance is a scope condition rather than a flaw. A minor revision could ask for a pixel-only variant or a clearer discussion of when a11y is unavailable, but the paper is publishable as is.

The strongest objection

The paper is transparent about its core limitation (b120): GUI-JEPA depends on the a11y tree for candidate enumeration. This is already identified as the weakest assumption. All other potential concerns—the training data asymmetry (13,594 episodes vs 4,000 for the 7B reference), the exclusion of screen capture and a11y retrieval from latency, and the step-level (not task-level) evaluation—are stated in the limitations (b121, b125, b126). The mechanism analyses are thorough and internally consistent: the candidate-conditioned latent pathway is essential (lesions cost 10.8–23.4 High points), but the predictor need not be a forward model. The speed and accuracy claims are substantiated under the stated conditions. No hidden assumption undermines the core architectural contribution.

The reader correctly identifies the reliance on the a11y tree as the weakest assumption. The paper already acknowledges this limitation (b120). No additional load-bearing concern emerges from careful reading. The speed comparison is fair (same GPU, same inputs, same judge). The accuracy ratio is reported relative to a controlled reference; the training data asymmetry is disclosed. The mechanism evidence is robust. Therefore the CONDITIONAL verdict is appropriate and no change is needed.

The referee's report

If the results hold under a broader range of conditions, GUI-JEPA offers a fundamentally different efficiency trade-off for GUI agents: by replacing token-by-token decoding with a single pass over candidate-conditioned latents, it eliminates the computational bottleneck of generative VLM policies. The finding that the predictor need not be a forward model—it is trained for the decision itself—is a non-obvious insight that could simplify the design of future latent-space agents. The paper provides thorough mechanism analyses and is transparent about limitations (a11y reliance, offline evaluation, latency boundaries). The architectural contribution (candidate-conditioned residual prediction for scoring) and the empirical speed-accuracy comparison under a same-protocol baseline are valuable. The extension to task-level planning remains future work, but the step-level result is a clear advance within the stated scope.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that a decoder-free GUI agent built on a Joint-Embedding Predictive Architecture (JEPA) can replace the token-by-token action generation of 7B-scale VLMs with a single batched scoring pass over candidate-conditioned latents. This approach dramatically reduces per-step latency and memory footprint while retaining most of the accuracy. The authors show that the scoring relies on the latent pathway, and that the predictor can be trained directly for the decision without needing dynamics pretraining. If correct, this opens a route to lightweight, fast GUI agents deployable on devices.

What carries the argument. The residual latent predictor, conditioned on each candidate's action vector and region-of-interest (ROI) pooled encoder features, transforms the shared screen latent into one candidate-conditioned latent per candidate. The structural head then scores these conditioned latents via cosine similarity against the text condition, complemented by a MiniLM text matcher. All candidates are processed in one batch without any autoregressive decoding.

Core claim. GUI-JEPA is the first GUI agent policy whose action selection is entirely decoder-free, using a JEPA encoder to embed the screenshot once, a residual predictor to produce one candidate-conditioned latent per candidate action, and lightweight heads to score all candidates against the instruction in a single batched pass. On AndroidControl, under the same inputs and judge, it reaches 92.4% / 86.1% of the in-distribution Low / High accuracy of a LoRA-tuned Qwen2.5-VL-7B agent while using only 6.0% of its parameters (0.50B vs 8.29B) and being 2.8-4.0x faster on an H20 GPU. The authors further establish that the scorer depends on the JEPA latent pathway and that the predictor learns what the decision needs, not next-latent prediction: a randomly initialised predictor matches a dynamics-pretrained one.

The premise it rests on. GUI-JEPA relies on the accessibility tree (a11y) to provide candidate actions with bounding boxes and text, which is a privileged input that pixel-only agents do not have; without it, the candidate enumeration fails and performance would degrade substantially.

If the paper is right (Pith's inferences, not the authors')

GUI agents can be made orders of magnitude smaller and faster, enabling deployment on mobile devices and real-time interactive applications.

The finding that a randomly initialised predictor works as well as a dynamics-pretrained one suggests that JEPA-based policies do not require expensive forward-model pretraining for GUI action selection.

The speed and memory savings make it practical to run GUI agents at scale, e.g., for automated testing or accessibility assistance.

The cascade strategy that uses confidence margins to route uncertain steps to a larger model can recover 98% of the 7B agent's accuracy at reduced average latency.

The approach may generalise to other structured interfaces (web DOM, desktop UI trees) by adapting the candidate representation and scoring heads.

If the method can be extended to pixel-only inputs (without a11y tree), it would address a key limitation and broaden applicability to unmodified apps.

The result that predictability and action relevance come apart on GUIs (the predictor is a poor forward model) strengthens the case for training world models directly for the decision task rather than for reconstruction.

The confidence-based cascade suggests a practical deployment pattern where a cheap first-stage JEPA agent handles routine steps and a larger VLM handles the hard cases, balancing speed and accuracy.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-28

The authors' main claim, as the classifier picked it out 0.97

“On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.95 · states limitation 0.64 · human participants 0.04 · controlled comparison 0.90 · randomized assignment 0.04 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. background 1.00 0.06 0.02 asserted 1.00
However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. background 0.77 0.08 0.03 asserted 1.00
To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. contribution 0.90 0.80 0.22 asserted 0.99
Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. method 0.99 0.81 0.13 asserted 1.00
We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. result 0.87 0.95 0.08 asserted 1.00
On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. result 1.00 0.96 0.52 asserted 1.00
Code is available at https://github.com/bcwang-sjtu/GUI-JEPA contribution 0.72 0.36 0.02 asserted 0.90

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 5,838 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (11)
Figure 1 of the paper
Figure 1. Same inputs, same GPU: GUI-JEPA reaches 86–92% of the 7B agent’s accuracy at a fraction of its cost. Bars are relative to our same￾protocol Qwen2.5-VL-7B SFT agent, fine-tuned with LoRA (IDD AC-Low / AC-High accuracy; latency: me￾dians over 100 shared real AC-High steps on one H20 at batch 1; peak memory under HF serving;
Figure 2 of the paper
Figure 2. Overview of GUI-JEPA. Top: on a real AndroidControl step, the frozen JEPA encoder embeds the screenshot once, the residual predictor turns it into one latent per candidate in a single batch, and the structural head and the text matcher score every candidate (bars: a trained base policy’s probabilities on this step; grids are schematic). Bottom: the training stages, and scoring in one pass versus decoding token by token (measured in Section 4.5). Circled numbers mark the stage that trains each module (1: JEPA pretraining, 2: policy training, 3: text rankers on the fixed policy);…
Figure 3 of the paper
Figure 3. Where a step’s time goes (segment means, 100 shared real steps, one H20, batch 1; medians: Ta￾ble 3). Decoding takes half or more of the 7B step, and its prefill alone outlasts a whole GUI-JEPA step. goal embedding (5.6 ms) are excluded. Speed. The final GUI-JEPA pipeline (base pol￾icy, span ranker and click reranker) needs a median 74 ms per step in fp32 and 51.5 ms with bf16 auto￾cast, against 207 ms for the 7B agent with vLLM and 507 ms with HF transformers (merged LoRA;
Figure 4 of the paper
Figure 4. The scorer relies on the latent pathway, which needs no dynamics pretraining. (a) Evaluation-time lesions (IDD; OOD and retrained controls in Tables 5 and 8). (b) Pilot AC-Low (an early configuration without the text matcher): only the structural head uses the pathway (orange box: deployed design). (c) Change in IDD accuracy against the jointly fine-tuned, dynamics-pretrained predictor when the predictor is kept frozen or randomly initialised (identical recipe). (a) Element: picks the menu entry Goal: Go to the Google Play Books app & bookmark the currently opened page click “Add…
Figure 5 of the paper
Figure 5. GUI-JEPA on three real AC-High IDD steps, one per action kind: (a) an element click, (b) a global action, (c) a typed argument that the span ranker copies from the goal. Each step takes one scoring pass over all its candidates (pt: probability of the chosen action; boxes: ground truth in green, GUI-JEPA’s choice in purple). points; at full scale, joint fine-tuning beats a frozen predictor by 3.8 AC-Low / 1.9 AC-High IDD points (Figure 4c; alternatives
Figure 6 of the paper
Figure 6. The text rankers’ gains transfer across subsplits. AC-High step accuracy (3 training seeds, mean ± sd) of the base policy, after the span ranker and after the click reranker, on IDD, the three overlapping OOD subsplits and the full test set; increments are computed on identical base checkpoints. Numbers
Figure 7 of the paper
Figure 7. Lenient frozen-encoder localisation probe across pretraining recipes. ScreenSpot-Pro hit@8 of a soft-argmax localisation probe on frozen patch tokens (our encoders: mean ± sd over 5 probe seeds); hit@8 accepts errors up to 57% of the image side. Points, left to right: a web anchor (≈77K web screenshots, 60 epochs); mobile-only (119K GUI-Odyssey screenshots, 2 epochs); desktop-only (70K VideoCUA screenshots, 2 epochs); the policy encoder (blue; the mixed 189K mobile and desktop recipe, 2 epochs); a replicate pretraining of the same recipe after 2 epochs and, continued, after 4 and 6…
Figure 8 of the paper
Figure 8. Same-hardware per-step latency (median, one H20, batch 1). Stars: the deployed GUI-JEPA pipeline on the 100 real AC-High IDD steps timed for both systems, placed at their median of 28 scored actions (18 a11y candidates plus global actions). Lines: the syn￾thetic fixed-N micro-benchmark with random inputs and N scored actions (
Figure 9 of the paper
Figure 9. Latency decomposition in the synthetic fixed-N micro-benchmark (one NVIDIA H20, batch 1, fp32 eager, N=57 scored actions, random image tensors and candidate vectors, 100-iteration median). The click-reranker segment is amortised over the action mix; a click step takes 98.4 ms. The benchmark excludes JPEG decoding, CPU ROI pooling and the span ranker and is a scaling illustration only; real-step medians are in
Figure 10 of the paper
Figure 10. Complementary errors behind the cascade on two real AC-High IDD steps (GUI-JEPA with both rankers and the same-protocol 7B SFT agent, given the same candidates). (a) GUI-JEPA is right with a margin above the 50%-coverage threshold τ (the median margin), so the cascade keeps its answer, while the 7B agent presses the wrong key. (b) GUI-JEPA picks the wrong destination with a margin below τ , so the cascade escalates the step to the 7B agent, which is right. Boxes: ground truth (green), GUI-JEPA (purple), 7B (rose). 50 100 150 200 250 mean ms per step (one H20, batch 1) 60 65 70 75…
Figure 11 of the paper
Figure 11. GUI-JEPA as the cheap first stage of a cascade (AC-High, IDD). Steps with a large top-two margin are answered by GUI-JEPA, the rest by the 7B SFT agent; labels: share of steps kept by GUI-JEPA. Latency: the mean GUI-JEPA step plus, on escalated steps, the mean vLLM 7B step (same 100 real steps). 26

Figures are the authors', reproduced from the paper.

References (63)
  1. [1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern R... on Pith
  2. [2] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Lab... on PithPith's reviewarXiv
  3. [3] Bizhe Bai, Jiakang Yuan, Hongming Wu, Xinyue Wang, Jie Ren, Siyao Chen, Yuchen Ya, Fan Bai, Pai Peng, Huafeng Qin, and Tao Chen. 2026. https://arxiv.org/abs/2609.02309 Efficient gui agents: A systems survey of observation, memory, action, and runtime optimization . Preprint, a... on PithPith's reviewarXiv
  4. [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13... on PithPith's reviewarXiv
  5. [5] Randall Balestriero and Yann LeCun. 2025. https://arxiv.org/abs/2511.08544 Lejepa: Provable and scalable self-supervised learning without the heuristics . Preprint, arXiv:2511.08544 on PithPith's reviewarXiv
  6. [6] Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Siran Dai, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, and Wenji Mao. 2026. https://arxiv.org/abs/2601.04035 Mobiledreamer: Generative sketch world model for gui agent . Preprint, arXiv:2601.04035 on PithPith's reviewarXiv
  7. [7] Hyungjoo Chae, Namyoung Kim, Kai Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. 2025. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In International Conference on Learning Representations, ... on Pith
  8. [8] Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, and Tian Lan. 2026. https://arxiv.org/abs/2604.05157 Intentscore: Intent-conditioned action evaluation for computer-use agents . Preprint, arXiv:2604.05157 on PithPith's reviewarXiv
  9. [9] Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. 2025. Lightweight neural app control. In International Conference on Learning Representations, volume 2025, pages 74161--74178 on Pith
  10. [10] Taoyong Cui, Pinxian Zeng, Wanghan Xu, Hao Xiao, Shuhan Xue, Qiang Gao, Wanli Ouyang, Pheng Ann Heng, Zhenfei Yin, Yingcheng Wu, and Ling Yang. 2026. https://arxiv.org/abs/2610.13912 Rsi-anything: Unlocking recursive self-improvement with world models . Preprint, arXiv:2610.13912 on PithPith's reviewarXiv
  11. [11] Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. 2026. V-droid: Advancing mobile gui agent through generative verifiers on Pith
  12. [12] Zihan Dong, Rui Qian, Qishi Zhan, Dongshen Peng, Kaixin Li, and Yu Li. 2026. https://arxiv.org/abs/2607.28399 Why are gui agents correct but late? decode on the decision-time critical path, tested with pre-compiled policy trees . Preprint, arXiv:2607.28399 on PithPith's reviewarXiv
  13. [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognit... on Pith
  14. [14] Yicheng Fu, Raviteja Anantha, Prabal Vashisht, Jianpeng Cheng, and Etai Littwin. 2025. Ui-jepa: Towards active perception of user intent through onscreen user activity. In Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization, pages 224--233 on Pith
  15. [15] Zhenfeng Gan, ZiTong Zeng, Jiajun Cheng, Yeke Song, Yongyi Tang, and Xueqian Wang. 2026. https://arxiv.org/abs/2607.26712 Actswm: Action-sensitive world models for long-horizon planning in open-world games . Preprint, arXiv:2607.26712 on PithPith's reviewarXiv
  16. [16] Qiwen Gu, Jifan Li, Bingjie Gao, Rui Chen, Jing Tang, Xiangxiang Chu, and Junqiao Zhao. 2026. https://arxiv.org/abs/2609.23369 The right future for action: Learning action-relevant predictive states in world action models . Preprint, arXiv:2609.23369 on PithPith's reviewarXiv
  17. [17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR . OpenReview.net on Pith
  18. [18] Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, and Lin Ma. 2025. https://arxiv.org/abs/2505.00416 Scaletrack: Scaling and back-tracking automated gui agents . Preprint, arXiv:2505.00416 on PithPith's reviewarXiv
  19. [19] Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, Kaixin Li, Patrice Bechard, Spandana Gella, and Sai Rajeswar. 2026. https://arxiv.org/abs/2603.24440 Cua-suite: Massive human-annotated video demonstrations for computer-use agents . Preprint, arXiv:2603.24440 on PitharXiv
  20. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In SOSP , pages 611--626. ACM on Pith
show 43 more
  1. [21] Kaixin Li, Zi-Yang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025. https://api.semanticscholar.org/CorpusID:277740982 Screenspot-pro: Gui grounding for professional high-resolution computer use . Proceedings of the 33rd ACM Interna... on Pith
  2. [22] Kelvin Li, Dhruv Pendharkar, Anish Pahilajani, Chuyi Shang, Leon Oks, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, and Roei Herzig. 2026. https://arxiv.org/abs/2609.02885 Discriminative world models for web agents . Preprint, arXiv:2609.02885 on PithPith's reviewarXiv
  3. [23] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130--92154 on Pith
  4. [24] Jiaping Lin, Fei Shen, Junzhe Li, Ping Nie, Fei Yu, Ming Li, and Haizhou Li. 2026 a . https://arxiv.org/abs/2605.12549 What happens before decoding? prefill determines gui grounding in vlms . Preprint, arXiv:2605.12549 on PithPith's reviewarXiv
  5. [25] Yihan Lin, Jiawei He, Shifeng Bao, Chen Zhao, Yang Li, Xiaobo Wang, Yan Wang, Cheng Chi, and Jing Zhang. 2026 b . https://arxiv.org/abs/2608.09381 Jepa-wam: Learning vision-language-action policies with joint-embedding world modeling . Preprint, arXiv:2608.09381 on PithPith's reviewarXiv
  6. [26] Shuaijun Liu, Chengyu Wu, Qifu Wen, Feiyang You, Chenglong Zhang, Shuyang Hao, Xi Lin, and Ningxin Su. 2026 a . https://arxiv.org/abs/2609.24749 D-jepa: A decision-aligned latent world model . Preprint, arXiv:2609.24749 on PithPith's reviewarXiv
  7. [27] Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, and Tianyu Pang. 2026 b . https://arxiv.org/abs/2607.24112 Scaling gui agents with visual state transitions . Preprint, arXiv:2607.24112 on PithPith's reviewarXiv
  8. [28] Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. 2025. Guiodyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. In Proceedings of the IEEE/CVF International Conference o... on Pith
  9. [29] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Pengxiang Zhao, Guangyi Liu, Guanjing Xiong, and Hongsheng Li. 2026. https://doi.org/10.1609/aaai.v40i21.38816 Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement le... on Pithdoi:10.1609/aaai.v40i21.38816
  10. [30] Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye HAO, Jun Wang, and Kun Shao. 2026. https://openreview.net/forum?id=mWoMyDEfbM Vimo: A generative visual GUI world model for app agents . In The Fourteenth International Conference on Learni... on Pith
  11. [31] Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. 2026. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312 on PithPith's reviewarXiv
  12. [32] Kai Mei, Jiang Guo, Shuaichen Chang, Mingwen Dong, Dongkyu Lee, Xing Niu, and Jiarong Jiang. 2026. R-wom: Retrieval-augmented world model for computer-use agents. In ICLR . proceedings.iclr.cc on Pith
  13. [33] Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, and Qi Wang. 2026. https://arxiv.org/abs/2610.05861 Imagine to act: High-fidelity data synthesis via image editing world model for scalable gui agent training . Preprint, arXiv:2610.05861 on PithPith's reviewarXiv
  14. [34] Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Micha... on PithPith's reviewarXiv
  15. [35] Taehwan Park, Changmin Lee, Hayeon Lee, and Taesik Gong. 2026. https://arxiv.org/abs/2609.32166 Pastforward: Faster on-device gui agents via computational experience reuse . Preprint, arXiv:2609.32166 on PithPith's reviewarXiv
  16. [36] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 on PithPith's reviewarXiv
  17. [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML , volu... on Pith
  18. [38] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP (1) , pages 3980--3990. Association for Computational Linguistics on Pith
  19. [39] Francisco-Javier Rodrigo-Gin \'e s. 2026. https://hal.science/hal-05546567 Agentic-JEPA: A Self-Supervised World Model for Planning in Text-Based Agent Environments . Working paper or preprint on Pith
  20. [40] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. 2020. https://doi.org/10.1038/s41586-020-03051-4 Mastering atari, go, chess ... on PithPith's reviewdoi:10.1038/s41586-020-03051-4
  21. [41] Oriane Sim \'e oni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seung Eun Yi, Michael Ramamonjisoa, Francisco Massa, Daniel HAZIZA, Luca Wehrstedt, Jianyuan Wang, Timoth \'e e Darcet, Th \'e o Moutakanni, Leone... on Pith
  22. [42] Yifan Sui, Xin Huang, Hongbing Li, Fang Xu, Jiahe Lv, Haolong Yan, Yeqing Shen, Litao Liu, Zhimin Fan, Ziyang Meng, Jia Wang, Junbo Qi, Kaijun Tan, Zheng Ge, Xiangyu Zhang, Daxin Jiang, and Osamu Yoshie. 2026. https://arxiv.org/abs/2605.27761 Androiddaily: A verifiable benchma... on PithPith's reviewarXiv
  23. [43] Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, and Zhibo Chen. 2026. Vla-jepa: Enhancing vision-language-action model with latent world model. In European Conference on Computer Vision, pages 478--497. Springer on Pith
  24. [44] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In NeurIPS on Pith
  25. [45] Ziwei Wang, Junjie Zheng, Leyang Yang, Sheng Zhou, Xiaoxuan Tang, Zhou Fang, Zhi-Wei Liu, Da-Jun Chen, Yong Li, and Jia-Jun Bu. 2026. https://api.semanticscholar.org/CorpusID:287479867 Towards scalable lightweight gui agents via multi-role orchestration . In Annual Meeting of ... on Pith
  26. [46] Jason Wu, Siyan Wang, Siman Shen, Yi - Hao Peng, Jeffrey Nichols, and Jeffrey P. Bigham. 2023. Webui: A dataset for enhancing visual UI understanding with web semantics. In CHI , pages 286:1--286:14. ACM on Pith
  27. [47] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2025. OS-ATLAS: foundation action model for generalist GUI agents. In ICLR . OpenReview.net on Pith
  28. [48] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. 2024. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR... on Pith
  29. [49] Yiping Xie, Song Chen, Jingxuan Xing, Wei Jiang, Zekun Zhu, Yingyao Wang, Pi Bu, Jun Song, Yuning Jiang, and Bo Zheng. 2026. https://arxiv.org/abs/2603.08533 Secagent: Efficient mobile gui agent with semantic context . Preprint, arXiv:2603.08533 on PitharXiv
  30. [50] Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao, Xiaolin Hu, Shuo Shang, and Bo An. 2026 a . https://arxiv.org/abs/2608.05891 Appdeltaworld: Transition-grounded delta code world model for mobile gui agents . Preprint, arXiv:2608.05891 on PithPith's reviewarXiv
  31. [51] Weikai Xu, Kun Huang, Yunren Feng, Jiaxing Li, Yuhan Chen, Yuxuan Liu, Zhizheng Jiang, Heng Qu, Pengzhi Gao, Wei Liu, Jian Luan, Xiaolin Hu, and Bo An. 2026 b . https://arxiv.org/abs/2605.10347 How mobile world model guides gui agents? Preprint, arXiv:2605.10347 on PithPith's reviewarXiv
  32. [52] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Aguvis: Unified pure vision agents for autonomous GUI interaction. In ICML , volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net on Pith
  33. [53] Rui Yang, Qianhui Wu, Zhaoyang Wang, Hanyang Chen, Ke Yang, Hao Cheng, Huaxiu Yao, Baolin Peng, Huan Zhang, Jianfeng Gao, and Tong Zhang. 2026 a . https://arxiv.org/abs/2602.22190 Gui-libra: Training native gui agents to reason and act with action-aware supervision and partial... on PithPith's reviewarXiv
  34. [54] Zhichao Yang, Yuanze Hu, Haojie Hao, Longkun Hao, Dongshuo Huang, Hongyu Lin, Gen Li, Lanqing Hong, Yihang Lou, and Yan Bai. 2026 b . Mirage: Mobile agents with implicit reasoning and generative world models. arXiv preprint arXiv:2606.04627 on PithPith's reviewarXiv
  35. [55] Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, and Wei Yang Bryan Lim. 2026 a . https://arxiv.org/abs/2610.11942 Right screen, wrong transition: World models as verifiers for gui agents . Preprint, arXiv:2610.11942 on PithPith's reviewarXiv
  36. [56] Linghua Zhang. 2026. Jev-mobile: Jev as an executor for mobile gui agents. arXiv preprint arXiv:2609.30186 on PithPith's reviewarXiv
  37. [57] Yang Zhang, Jiangyuan Zhao, Chenyou Fan, Jiayu Hu, Xiu Yuan, Chenjia Bai, and Xiu Li. 2026 b . https://arxiv.org/abs/2609.37250 V-jepa policy: Building effective world-action models on predictive visual latents . Preprint, arXiv:2609.37250 on PithPith's reviewarXiv
  38. [58] Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, et al. 2025. Agentcpm-gui: Building mobile-use agents with reinforcement fine-tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language ... on Pith
  39. [59] Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, and Kevin Qinghong Lin. 2026. https://arxiv.org/abs/2602.09856 Code2world: A gui world model via renderable code generation . Preprint, arXiv:2602.09856 on PitharXiv
  40. [60] Artem Zholus, Nicolas Beltran-Velez, Jianhao Yuan, Sarath Chandar, Tushar Nagarajan, Daniel Severo, Koustuv Sinha, Michal Drozdzal, Adriana Romero Soriano, Jeannette Bohg, Nicolas Ballas, and Mahmoud Assran. 2026. https://arxiv.org/abs/2610.10515 Robojepa: Scaling robotic late... on PithPith's reviewarXiv
  41. [61] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. 2025 a . DINO-WM: world models on pre-trained visual features enable zero-shot planning. In ICML , volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net on Pith
  42. [62] Hanzhang Zhou, Xu Zhang, Pan-Rong Tong, Jianan Zhang, Liangyu Chen, Qu-Yu Kong, Cheng-Lin Cai, Chen Liu, Yue Wang, Jing-Ren Zhou, and Steven Hoi. 2025 b . https://api.semanticscholar.org/CorpusID:284275280 Mai-ui technical report: Real-world centric foundation gui agents . ArX... on PitharXiv
  43. [63] Xurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li, Kaiwen Zhou, Shuai Wang, Shuo Yang, Zhuotao Tian, and Rui Shao. 2026. Hiconagent: History context-aware policy optimization for GUI agents. In CVPR , pages 13028--13038. Computer Vision Foundation on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment