REVIEW 3 major objections 5 minor 1 cited by
Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Self-evolving training lets a 7B GUI model beat a 72B one by learning to zoom in first.
desk verdict The training recipe is genuine and the gains are large, but the headline SoTA rests on an unreported inference-time min-pixels setting that Figure 4 shows swings accuracy from 30 to 47.5; fix that before trusting the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the focus-region action: the model emits crop(x,y,w,h) before click(x,y). Region preferences are constructed using two complementary rewards—Monte Carlo quality estimation, which measures how often a candidate crop leads to the correct click, and IoU-based diversity, which penalises two candidate crops that overlap too much. Together these define a self-generated preference signal for DPO, and the crop action is what turns a static single-step grounding model into a multi-step perceptual one.
What would settle it
Run LASER and the key baselines (GTA1-7B, GTA1-32B, Qwen2.5-VL-72B) on ScreenSpot-Pro at every min-pixels value shown in Figure 4 and compare each model's accuracy at a shared threshold. If, at the common optimal threshold, LASER no longer exceeds all baselines, the state-of-the-art claim is refuted; publishing the exact min-pixels used for every row of Table 1 would settle whether the comparison is fair.
Extended reading notes
Core claim
The central claim is that active perception—deciding which part of a screenshot to inspect—can be learned by a VLM through a self-evolving loop, rather than distilled from an expert or forced by reinforcement learning from scratch. LASER formalizes this as a two-action policy: first predict a crop box, then predict a click coordinate inside the cropped view. The training data is generated by rejection sampling: only trajectories whose click is correct are kept for supervised fine-tuning. To learn which crop regions are good, the model samples multiple candidate crops and scores them by Monte Carlo quality estimation (the success rate of the subsequent click across rollouts) and an IoU-based
Load-bearing premise
The central comparison assumes the test-time min-pixels (image resizing threshold) used for LASER is the same as, or fair relative to, the settings used for the baselines; Section 6.3 shows accuracy is highly sensitive to this value, but the main-table value is not stated.
Editorial extensions
If this is right
- If LASER's claims hold, a 7B GUI model can outperform a 72B general-purpose VLM on professional high-resolution grounding, so model size matters less than learned perception strategy.
- The method does not require teacher trajectories or imitation data, suggesting other high-resolution visual tasks could bootstrap their own zoom-in behavior.
- The self-evolving loop can be re-applied on top of an already strong RL-trained base (GTA1-7B) and still yield gains, indicating the two training paradigms are complementary.
- The adaptive step count observed (peak accuracy at two steps, with the model sometimes choosing more) implies small models can allocate reasoning budget by difficulty without explicit supervision.
Reading between the lines
- The reported 55.7 depends on an unstated test-time min-pixels value; Figure 4 shows accuracy varies from about 30% to 47.5% as this threshold changes, so a fair cross-model comparison would need the same threshold for all baselines.
- If min-pixels is standardized, the exact SoTA margin may narrow or shift, but the qualitative conclusion that crop-then-click helps likely survives because the ablation trend holds across thresholds.
- The same region-preference construction could transfer to other pixel-space reasoning tasks (document layout, medical images, satellite images) where an agent must first locate a region of interest before fine-grained prediction.
- A natural next test is extending the trajectory length beyond two steps in training; the paper's step analysis suggests longer rollouts are already possible, so deeper supervision may push accuracy further.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LASER, a multi-stage self-evolving training framework for GUI grounding. Given a user instruction and screenshot, the model first predicts a crop/focus region, then predicts the click coordinate in the cropped view. Training proceeds by: (1) rejection-sampling SFT on single-step crop-then-click trajectories, (2) DPO preference learning using a Monte-Carlo accuracy reward and an IoU-based diversity reward, and (3) iterative multi-step trajectory construction and further training to enable adaptive reasoning depth. Experiments on ScreenSpot-Pro and ScreenSpot-v2 show large gains over base models. In particular, LASER fine-tuned from GTA1-7B is reported to reach 55.7 on ScreenSpot-Pro, surpassing GTA1-32B (53.6) and Qwen2.5-VL-72B (53.3), which is claimed as a new state of the art among 7B-scale models.
Significance. If the reported results are reproducible and comparisons are fair, the work is significant: it demonstrates that a purely self-bootstrapped, preference-optimized training signal can elicit multi-step active perception (zooming/cropping) in 7B VLMs without distillation from larger teachers or explicit imitation of expert trajectories. The framework is clearly described, the code is released, and the ablations isolate the contribution of the two reward terms and the multi-step stage. The reasoning-step analysis (Figure 6) is a useful diagnostic. However, the headline SoTA claim is currently conditional on an unreported test-time resolution setting that the paper's own ablation shows can shift accuracy by more than 17 points, so the main contribution cannot be assessed as stated.
major comments (3)
- [Section 6.3 / Figure 4, and Table 1] The test-time min-pixels value is never stated for the main results. Figure 4 shows that LASER(Qwen2.5-VL) accuracy on ScreenSpot-Pro varies from 30.23% at min-pixels=3,136 to 47.50% at min-pixels=802,816, and Table 1 reports exactly 47.5 for LASER(Qwen2.5-VL). The paper does not state which min-pixels was used for LASER or for any baseline, nor the criterion for selecting the value. Since the margin over GTA1-32B is only 2.1 points while the min-pixels axis moves accuracy by >17 points, the reported ranking could reverse under a different but equally plausible setting. This is load-bearing for the central 'new SoTA' claim. The authors must report the exact inference settings for all models and justify the selection, ideally showing baselines under the same protocol.
- [Section 4.3, Eq. (2), and Section 5.1 / Table 1] The number of Monte Carlo rollouts N used to compute R_acc is never specified. Equation (2) defines the reward as an average over N rollouts, but Section 5.1 gives only dataset sizes and DPO hyperparameters. The reliability of the preference labels and the reproducibility of the method depend directly on N. Additionally, all experiments appear to be single runs with no error bars or multiple seeds; the 2.1-point gap over GTA1-32B in Table 1 is exactly the kind of difference that could be within run-to-run variance. Please report N, the sampling temperature, and multiple-seed statistics for at least the main tables.
- [Section 4.2 and Section 4.3] The 'self-evolving without external supervision' claim is overstated. Rejection sampling in Section 4.2 filters trajectories by comparing the model's click to the ground-truth label, and R_acc in Eq. (2) is computed against the same ground-truth click label. The method does not require a teacher model or handcrafted trajectory templates, but it does rely on ground-truth click annotations in the GTA1 training data. The authors should qualify the novelty statement to say 'without teacher models or imitation of expert trajectories' rather than implying no external labels at all.
minor comments (5)
- [Table 1] In the LASER(GTA1) row, the OS/Text entry appears as '57.0 37' — likely a missing decimal point ('37.0' or similar). Please correct.
- [Figure 4] The x-axis is labeled 'Image Pixel Size' and the caption calls it 'min-pixels threshold'. Clarify whether the x-axis is the threshold value or the resulting resized pixel count, and specify the original image resolution distribution so readers can interpret the values.
- [Section 6.2 / Figure 3] The stage-wise comparison in Figure 3 does not state whether the inference-time min-pixels is held constant across stages. If it is not, the gains could be confounded with the resolution effect studied in Section 6.3. Please state the setting.
- [Section 4.4] The multi-step trajectory collection is described as applying MDPO iteratively to 'single-step failure cases' where the crop contains the ground-truth box but the click is wrong. It would be helpful to report how many such cases existed and how many were successfully corrected, since this determines the difficulty distribution of D⟳.
- [Related Work] The distinction between 'self-evolving' and prior self-evolutionary RL methods such as SE-GUI could be sharpened; the term 'self-evolving' is used differently in the literature. A sentence explaining the difference would help readers.
Circularity Check
No circular derivation: LASER's rejection sampling, Monte Carlo rewards, and DPO pairs are all anchored to external ground-truth click labels. The unstated min-pixels inference threshold in Table 1 is a reproducibility gap, not a circular step.
full rationale
The paper's derivation chain is self-contained with respect to external labels at every stage. DSFT is filtered by comparing predicted clicks with ground-truth labels (Sec 4.2). R_acc in Eq (2) is the fraction of Monte Carlo rollouts whose click equals the ground-truth label; R_div in Eq (3) measures IoU between candidate crops. Preference pairs in DDPO are selected by thresholds on these rewards, and D⟳ is built from cases where the crop contains the ground-truth box and the final multi-step click is corrected to the ground-truth label (Sec 4.4). Thus no 'prediction' is defined in terms of the model's own output alone; every preference signal reduces to dataset labels, which is external supervision, not circularity. The self-evolving loop is a rejection-sampling/DPO bootstrap that can amplify initial biases but does not assume the result it claims to produce. There are no load-bearing self-citations or imported uniqueness theorems. The one notable flaw is reporting: Table 1 reports LASER(Qwen2.5-VL)=47.5 on ScreenSpot-Pro, which equals the peak of the min-pixels sweep in Fig. 4 (47.50 at 802,816), yet Section 6.3 does not state the min-pixels value used for the main comparison or for baselines. This is a test-set-selection/reproducibility concern that undermines the comparability of the headline SoTA, but it is not a circular derivation. Score 1 reflects this minor non-circular reporting gap.
Assumptions & free parameters
free parameters (6)
- δ (score margin filter) =
4
- τ (IoU overlap filter) =
0.8
- β (DPO scaling) =
0.1
- N (Monte Carlo rollouts) =
not reported
- min-pixels inference threshold =
not reported for main results
- training data sizes =
120K SFT, 36K multi-step, 26K + 8K DPO pairs
assumptions (4)
- domain assumption A focus region of approximately 20% of the original image is optimal for GUI grounding
- domain assumption Ground-truth click labels are a reliable supervision signal for region quality
- domain assumption The GTA-1 filtered training data (270K instances) is a suitable training distribution
- ad hoc to paper Self-generated multi-step trajectories that end in correct clicks are valid training data
Cite this review
Pith. "Pith review of Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding." pith.science (2026). https://pith.science/paper/JXHF26ZZ
@misc{pith2026250904243,
author = {Pith},
title = {Pith review of: Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding},
year = {2026},
howpublished = {\url{https://pith.science/paper/JXHF26ZZ}},
note = {Machine review of arXiv:2509.04243}
}
read the original abstract
Vision Language Models (VLMs) have recently achieved significant progress in bridging visual perception and linguistic reasoning. Recently, OpenAI o3 model introduced a zoom-in search strategy that effectively elicits active perception capabilities in VLMs, improving downstream task performance. However, enabling VLMs to reason effectively over appropriate image regions remains a core challenge in GUI grounding, particularly under high-resolution inputs and complex multi-element visual interactions. In this work, we propose LASER, a self-evolving framework that progressively endows VLMs with multi-step perception capabilities, enabling precise coordinate prediction. Specifically, our approach integrate Monte Carlo quality estimation with Intersection-over-Union (IoU)-based region quality evaluation to jointly encourage both accuracy and diversity in constructing high-quality preference data. This combination explicitly guides the model to focus on instruction-relevant key regions while adaptively allocating reasoning steps based on task complexity. Comprehensive experiments on the ScreenSpot Pro and ScreenSpot-v2 benchmarks demonstrate consistent performance gains, validating the effectiveness of our method. Furthermore, when fine-tuned on GTA1-7B, LASER achieves a score of 55.7 on the ScreenSpot-Pro benchmark, establishing a new state-of-the-art (SoTA) among 7B-scale models.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
DRS-GUI introduces a dynamic region search method with Focus/Shift/Scatter actions and MCTS-based planning that improves GUI grounding accuracy by 14% on ScreenSpot-Pro for both general and GUI-specific MLLMs without ...
Reference graph
Works this paper leans on
-
[3]
arXiv preprint arXiv:2505.20272
Ground-R1: Incentivizing Grounded Vi- sual Reasoning via Reinforcement Learning. arXiv preprint arXiv:2505.20272. Chen, Q.; Qin, L.; Liu, J.; Peng, D.; Guan, J.; Wang, P.; Hu, M.; Zhou, Y .; Gao, T.; and Che, W
-
[4]
arXiv preprint arXiv:2503.09567
Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567. Cheng, K.; Sun, Q.; Chu, Y .; Xu, F.; Li, Y .; Zhang, J.; and Wu, Z
-
[5]
arXiv preprint arXiv:2401.10935
Seeclick: Harnessing gui grounding for ad- vanced visual gui agents. arXiv preprint arXiv:2401.10935. Deng, X.; Gu, Y .; Zheng, B.; Chen, S.; Stevens, S.; Wang, B.; Sun, H.; and Su, Y
-
[6]
Hong, W.; Wang, W.; Lv, Q.; Xu, J.; Yu, W.; Ji, J.; Wang, Y .; Wang, Z.; Dong, Y .; Ding, M.; et al
Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243. Hong, W.; Wang, W.; Lv, Q.; Xu, J.; Yu, W.; Ji, J.; Wang, Y .; Wang, Z.; Dong, Y .; Ding, M.; et al
-
[7]
arXiv preprint arXiv:2402.02716
Understand- ing the planning of LLM agents: A survey. arXiv preprint arXiv:2402.02716. Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Rad- ford, A.; et al
-
[8]
arXiv preprint arXiv:2410.21276
Gpt-4o system card. arXiv preprint arXiv:2410.21276. Lai, H.; Liu, X.; Iong, I. L.; Yao, S.; Chen, Y .; Shen, P.; Yu, H.; Zhang, H.; Zhang, X.; Dong, Y .; et al
-
[9]
arXiv preprint arXiv:2504.07981
Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981. Lu, Z.; Chai, Y .; Guo, Y .; Yin, X.; Liu, L.; Wang, H.; Xiao, H.; Ren, S.; Xiong, G.; and Li, H
-
[10]
arXiv preprint arXiv:2503.21620
UI-R1: Enhanc- ing Efficient Action Prediction of GUI Agents by Reinforce- ment Learning. arXiv preprint arXiv:2503.21620. Luo, R.; Wang, L.; He, W.; and Xia, X
Show all 25 references
-
[11]
arXiv preprint arXiv:2504.10458
Gui-r1: A gen- eralist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458. OpenAI
-
[12]
https: //openai.com/index/o3-o4-mini-system-card/
OpenAI o3 and o4-mini System Card. https: //openai.com/index/o3-o4-mini-system-card/. Accessed: 2025-07-29. Qin, Y .; Hu, S.; Lin, Y .; Chen, W.; Ding, N.; Cui, G.; Zeng, Z.; Zhou, X.; Huang, Y .; Xiao, C.; et al
2025
-
[13]
arXiv preprint arXiv:2501.12326
Ui-tars: Pio- neering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326. Su, A.; Wang, H.; Ren, W.; Lin, F.; and Chen, W. 2025a. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2...
-
[14]
arXiv preprint arXiv:2504.07491
Kimi-vl technical report. arXiv preprint arXiv:2504.07491. Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y .; Chen, D.; Wu, Y .; and Sui, Z
-
[16]
arXiv preprint arXiv:2410.23218
Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218. Xi, Z.; Chen, W.; Guo, X.; He, W.; Ding, Y .; Hong, B.; Zhang, M.; Wang, J.; Jin, S.; Zhou, E.; et al
-
[17]
arXiv preprint arXiv:2505.13227
Scaling Computer- Use Grounding via User Interface Decomposition and Syn- thesis. arXiv preprint arXiv:2505.13227. Xu, Y .; Wang, Z.; Wang, J.; Lu, D.; Xie, T.; Saha, A.; Sa- hoo, D.; Yu, T.; and Xiong, C
-
[18]
arXiv preprint arXiv:2412.04454
Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. Yang, Y .; Li, D.; Dai, Y .; Yang, Y .; Luo, Z.; Zhao, Z.; Hu, Z.; Huang, J.; Saha, A.; Chen, Z.; et al
-
[19]
arXiv preprint arXiv:2507.05791
GTA1: GUI Test-time Scaling Agent. arXiv preprint arXiv:2507.05791. Yang, Y .; Wang, Y .; Li, D.; Luo, Z.; Chen, B.; Huang, C.; and Li, J
-
[20]
arXiv preprint arXiv:2412.16256
Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256. Yuan, X.; Zhang, J.; Li, K.; Cai, Z.; Yao, L.; Chen, J.; Wang, E.; Hou, Q.; Chen, J.; Jiang, P.-T.; et al
-
[21]
arXiv preprint arXiv:2505.12370
Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Re- inforcement Learning. arXiv preprint arXiv:2505.12370. Zhang, X.; Gao, Z.; Zhang, B.; Li, P.; Zhang, X.; Liu, Y .; Yuan, T.; Wu, Y .; Jia, Y .; Zhu, S.-C.; et al
-
[22]
arXiv preprint arXiv:2505.15436
Chain-of- Focus: Adaptive Visual Search and Zooming for Multimodal Reasoning via RL. arXiv preprint arXiv:2505.15436. Zheng, Z.; Yang, M.; Hong, J.; Zhao, C.; Xu, G.; Yang, L.; Shen, C.; and Yu, X
-
[23]
arXiv preprint arXiv:2505.14362
DeepEyes: Incentivizing” Thinking with Images” via Reinforcement Learning. arXiv preprint arXiv:2505.14362. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y .; Fried, D.; et al
-
[24]
arXiv preprint arXiv:2307.13854
Webarena: A realistic web environment for building au- tonomous agents. arXiv preprint arXiv:2307.13854. Zhu, M.; Zhong, H.; Zhao, C.; Du, Z.; Huang, Z.; Liu, M.; Chen, H.; Zou, C.; Chen, J.; Yang, M.; et al
-
[25]
arXiv preprint arXiv:2505.21457
Active-O3: Empowering Multimodal Large Language Mod- els with Active Perception via GRPO. arXiv preprint arXiv:2505.21457
-
[2023]
arXiv preprint arXiv:2312.08935
Math-shepherd: Verify and rein- force llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935. Wu, Z.; Wu, Z.; Xu, F.; Wang, Y .; Sun, Q.; Jia, C.; Cheng, K.; Ding, Z.; Chen, L.; Liang, P. P.; et al
-
[2024]
https: //www.anthropic.com/news/developing-computer-use
Developing a computer use model. https: //www.anthropic.com/news/developing-computer-use. Ac- cessed: 2025-04-12. Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al
2025
-
[2025]
5-vl technical report
Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cao, M.; Zhao, H.; Zhang, C.; Chang, X.; Reid, I.; and Liang, X
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.