Computer Vision and Pattern Recognition · arXiv:2610.36874 v1 · submitted 27 Oct 2026 · 64 references
AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
The paper says
“GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy.”
more of the abstract
However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround
from the abstract
Pith says
Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A cleanly engineered, training-free GUI grounding pipeline with real gains on small targets; the size-estimation robustness is better than the stress test suggests, but the method is verified only on two Qwen backbones and four benchmarks. This line is wrong
1 major objection · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 64 references say.
The paper says, at §3.2, Appendix B.3
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The central design of TNZ depends on the model's self-reported bounding-box size ˆs to set the zoom factor via Eq. (2) and crop size via Eq. (3). Appendix B.3 shows that the median ratio of ˆs to the true side is 1.2–1.4 on ScreenSpot-Pro and that accuracy changes by at most 0.6 points when the true size is used.
more of objection 1
However, this robustness test is performed on the same benchmarks used for development. The paper reports overall accuracy on held-out UI-Vision (Table 3) but does not analyze the quality of the size estimate or the per-sample behavior of TNZ on that distribution. Since UI-Vision has different screenshots and applications (83 desktop apps), the estimate could degrade. The authors should report, at least in the appendix, the distribution of ˆs errors on UI-Vision and confirm that the robustness result (max 0.6-point change) holds there, or discuss why the verifier and chain portfolio would absorb any degradation.
6 minor objections
The paper says, at §3.3
Pith says: minor objection 1 of 6
The verifier score v(p) is defined as the normalized first-token probability of 'Yes' vs 'No', summing over case variants and taking the 20 most likely first tokens. The exact procedure for selecting the 20 tokens and handling ties (e.g., when neither 'Yes' nor 'No' appears) should be stated more explicitly in the main text rather than only in Appendix A.1.
The paper says, at §3.3, Eq. (8)
Pith says: minor objection 2 of 6
The abstention rule uses θ_e=0.03. The paper states that this value was set for few false refusals (0.4–0.8%) but does not report sensitivity to this threshold. A brief sensitivity analysis (e.g., accuracy vs. θ_e ∈ [0.01, 0.2] as in Appendix B.2 but for both backbones) would help the reader understand how critical this choice is.
show 4 more
The paper says, at §4.1, 'Protocol'
Pith says: minor objection 3 of 6
The paper notes that outputs are not bit-exact across runs due to vLLM continuous batching. It should state explicitly whether the reported numbers are from a single run or averaged over multiple runs, and whether the confidence intervals in Table 13 account for this variance (they appear to be bootstrap intervals over screenshots, which do not capture inference randomness).
The paper says, at Table 5 and Algorithm 1
Pith says: minor objection 4 of 6
Table 5 lists many constants. While all are fixed across experiments, the paper would benefit from a compact notation in Algorithm 1 that makes it clear which constants are used at each step. Currently the procedure references 'the constants of Section 3.5' which is vague.
The paper says, at Figure 2
Pith says: minor objection 5 of 6
The overview figure is informative but the font in subfigures (a), (b), (c) is very small. In print, the text may be illegible. Consider enlarging the key panels or providing a separate high-resolution version.
The paper says, at §5, Limitations
Pith says: minor objection 6 of 6
The limitations paragraph on CSG states that it does not handle slider positions or text-cursor placement. The prompt in Appendix A.3 includes an 'along' operator and a 'text_edge' operator, but these are said not to be executed. This is confusing. Clarify whether these operators are reserved for future work or simply not evaluated here.
What would settle it
Compute the estimated-to-true size ratio on every sample of UI-Vision (the held-out benchmark) for both backbones, then group samples by ratio magnitude (e.g., <0.8, 0.8–1.2, 1.2–2, >2). Compare AdaGround's accuracy on each group to the fixed 4× crop. If AdaGround underperforms the fixed crop on the extreme groups (ratio >2 or <0.5), this would indicate that size-estimate degradation can cause failure, weakening the robustness claim.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 28 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
AdaGround does something genuinely useful. It solves five well-articulated problems in test-time zooming for GUI grounding—scale-agnostic zoom, open-loop acceptance, no second attempt, need for model internals, and no abstention—with three straightforward components: token-normalized adaptive zooming, a verifier that treats the same VLM as a black box, and compositional cell grounding. The results are consistently above strong baselines (MVP, ZoomClick) on three development benchmarks and hold on a held-out benchmark (UI-Vision). Paired tests with confidence intervals are provided, and the ablation study isolates where each component helps. The writing is clear and the limitations are honestly stated.
What the stress test calls a load-bearing flaw—the size estimate's accuracy—the paper already addresses in Appendix B.3: using the true target size changes accuracy by at most 0.6 points. That is a real empirical check, and the stress test's worry that it might degrade on a different distribution is speculative. The paper tests UI-Vision (83 desktop apps) as a held-out set and the gains carry over, so the concern is not acute.
The main soft spots are real but acknowledged. The method is evaluated only on Qwen3-VL-8B and -32B, and only on four benchmarks. The cost is high (3.5–4× visual tokens of a fixed crop). The comparison with MVP on OSWorld-G 32B is unpaired. CSG works only for spreadsheet cells. These limit the generality but do not undercut the central claim.
This is a solid, honest paper. I would bring it to a reading group and would accept a peer-review invitation. It deserves a serious referee.
The strongest objection
The paper's central claim depends on the ability of the model's self-reported bounding box to set the zoom factor via Eq. (2) and crop size via Eq. (3). Appendix B.3 shows that the median ratio of estimated to true side is 1.2–1.4 on ScreenSpot-Pro, causing under-magnification on average, and that accuracy changes by at most 0.6 points when using the true size. However, this robustness test is performed on the same benchmarks used for development. The paper does not evaluate how the size estimate behaves on a systematically different distribution—e.g., UI-Vision's 1,181 screenshots spanning 83 desktop applications—nor does it test the combined effect of estimation error on the verifier's arbitration and abstention decisions, which use scores derived from the same model. If the estimate were substantially worse on a new distribution, the margin of 0.6 points could be exceeded, reducing the claimed improvements.
The reader identified the size estimate as the weakest assumption, and I agree it is the most load-bearing component. However, the paper's robustness test (replacing estimated size with true size) is limited to the development benchmarks. A stronger stress test would examine performance on a held-out benchmark (UI-Vision) on samples where the estimate is most inaccurate. The results on UI-Vision overall are strong, suggesting the concern may not materialize, but the analysis is not currently disaggregated. This does not invalidate the central claim, but a targeted check would settle whether the weakest assumption actually holds beyond the development set. Hence the verdict remains unchanged.
The referee's report
If the claims hold, the paper makes a practical and principled contribution to GUI grounding. The insight that zoom geometry should be set by the target's token span rather than by fixed or heuristic ratios is novel and well motivated. The use of a black-box verifier (one Yes/No question) to replace attention map access and voting is a clean design that broadens applicability. The held-out evaluation on UI-Vision strengthens the claim that the method generalizes beyond the three development benchmarks. The fact that all constants are shared across backbones and benchmarks is a practical advantage. The gains are large (up to 20.6 points) and statistically significant in paired tests. The code release supports reproducibility.
Pith's reading of the paper (a machine's summary, not the authors')
AdaGround is a training-free framework that improves GUI grounding on high-resolution screens by solving five shortcomings of existing test-time zooming methods. It uses Token-Normalized Adaptive Zooming (TNZ) to derive zoom magnification, crop size, and depth from the model's own target-size estimate, so the target occupies a fixed number of visual tokens (3 tokens). A Verifier-in-the-Loop Refinement (VLR) replaces open-loop acceptance with a Yes/No question about a marked click, using that score to keep or reject each zoom, choose between a TNZ chain and a fixed 4× crop, trigger re-grounding when unresolved, and abstain when the target is absent. Compositional Structure Grounding (CSG) handles spreadsheet cells by composing separately grounded headers. On Qwen3-VL-8B and 32B, AdaGround improves accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, surpassing state-of-the-art training-free methods (MVP, ZoomClick) and many trained models including 72B ones; gains carry over to held-out UI-Vision. A single set of constants serves both backbones and all benchmarks.
What carries the argument. The central mechanism is the token span σ = s·k/π, where s is the target's shorter side in original pixels, k is the linear scale of the view, and π is the token pitch (32 pixels for Qwen3-VL). Token-Normalized Adaptive Zooming (TNZ) solves Eq. (2) to set the magnification k = clip(τ·π/ŝ, 1, kmax) so that the estimated target spans τ = 3 tokens, and Eq. (3) to set the crop side d = clip(max(γŝ, ε·π/kprev), dmin, dmax) to cover context and localization error. The Verifier-in-the-Loop Refinement (VLR) uses v(p) = P(YES | marked-click question) / (P(YES)+P(NO)) from the model's first-token probabilities to gate acceptance, arbitrate between chains (Eq. (7)), trigger re-grounding, and decide abstention (Eq. (8)). The key insight is that the model's own predictions drive both zoom and verification, eliminating the need for attention maps or training.
Core claim. The central claim is that using the model's own size estimate to set a zoom that renders the target at exactly three token spans, and replacing open-loop acceptance and voting with model-driven verification and chain arbitration, can significantly improve GUI grounding without any training or access to internal model states. The paper demonstrates this with Qwen3-VL, showing consistent gains on high-resolution and small-target benchmarks. It shows that two different zoom geometries (TNZ and a fixed 4× crop) fail on different samples, and that the model's own Yes/No question can pick the right answer over a third of the way from the better chain to the oracle. The framework recovers 39% and 36% of the headroom above the better chain on ScreenSpot-Pro with Qwen3-VL-8B and 32B, respectively.
The premise it rests on. The paper assumes that the model's self-reported bounding box is accurate enough to set the zoom factor via Eq. (2); the paper acknowledges this estimate is coarse (median ratio 1.2–1.4 on ScreenSpot-Pro) but shows empirical robustness on the tested benchmarks.
If the paper is right (Pith's inferences, not the authors')
GUI agents can achieve higher grounding accuracy without re-training, using the same VLM they already deploy, by simply modifying the inference procedure.
The verification-based chain portfolio can be extended to more than two zoom strategies or to non-zoom views (e.g., different rendering scales), potentially improving robustness further.
The token span criterion provides a principled way to decide when and how much to zoom across different screen resolutions and target sizes, which could generalize to other VLMs that tile images into tokens.
The abstention mechanism (Eq. (8)) reduces false positives on absent-target instructions and can be combined with other systems to make GUI agents safer in tasks requiring confidence.
The framework's success suggests that the VLM's own confidence on a marked-click Yes/No question is a reliable indicator of grounding quality, potentially applicable to other perception tasks like object detection or visual question answering.
The constant τ = 3 was chosen based on an empirical accuracy plateau near 3 tokens on Qwen3-VL; on a different VLM with different resolution handling (e.g., smaller or larger token pitch), the optimal τ may differ, so the method may require retuning the single parameter when switching backbones.
The reliance on the model's box estimate means that if the model fails to output a bounding box (e.g., returns only a point or refuses), TNZ will not zoom; the paper acknowledges this and falls back to the fixed 4× chain and re-grounding, but on models that rarely produce boxes the benefit of TNZ is lost.
The cost (3.5–3.9× visual tokens, 3.4–4.2× latency) is nontrivial for latency-sensitive agent loops; however, the paper shows that arbitration alone (about 10 calls) already beats the fixed crop on ScreenSpot-Pro, offering a potential efficiency–accuracy trade-off.
The verifier's single threshold θr = 0.99 and abstention threshold θe = 0.03 were set on OSWorld-G and may not transfer to domains with different answer distributions; the paper tests θr = 0.9 and finds no significant change on the benchmarks, but this may not hold in all scenarios.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-28
The authors' main claim, as the classifier picked it out 1.00
“On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.85 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. | background 1.00 | 0.11 | 0.03 | asserted 1.00 |
| However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. | gap 0.96 | 0.06 | 0.03 | asserted 1.00 |
| To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. | method 0.58 | 0.89 | 0.07 | asserted 0.99 |
| On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. | result 1.00 | 0.96 | 0.83 | asserted 1.00 |
| Code is available at https://github.com/bcwang-sjtu/AdaGround | contribution 0.70 | 0.44 | 0.02 | asserted 0.92 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 4,562 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (64)
- [1] Anthropic . 2024. https://www.anthropic.com/news/3-5-models-and-computer-use Introducing computer use, a new Claude 3.5 Sonnet , and Claude 3.5 Haiku . Anthropic News on Pith
- [2] Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. https://doi.org/10.24963/ijcai.2024/339 Screenai: A vision-language model for ui and infographics understanding . In ... on Pithdoi:10.24963/ijcai.2024/339
- [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025 a . Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 on PithPith's reviewarXiv
- [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025 b . https://arxiv.org/abs/2502... on PithPith's reviewarXiv
- [5] Liangyu Chen, Hanzhang Zhou, Jianan Zhang, Panrong Tong, Xu Zhang, Quyu Kong, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, et al. 2026. Ui-ins: Enhancing gui grounding with multi-perspective instruction as reasoning. In International Conference on Learning Representations, volu... on Pith
- [6] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap... on Pith
- [7] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic c... on PithPith's reviewarXiv
- [8] Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, and Yongliang Shen. 2026. Test-time reinforcement learning for gui grounding via region consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30593--30601 on Pith
- [9] Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, et al. 2026. Gui-lens: Coarse-to-fine cropping for gui grounding with general-purpose vlms. arXiv preprint arXiv:2608.03270 on PithPith's reviewarXiv
- [10] Boyu Gou, Demi Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/4ca0e369689dadb25a5345ba9755ad6f-Paper-Conference.pdf Navigating the digital world as humans do: Universal visual gr... on Pith
- [11] Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, et al. 2025. Ui-venus technical report: Building high-performance ui agents with rft. arXiv preprint arXiv:2508.10833 on PithPith's reviewarXiv
- [12] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. 2025. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062 on PithPith's reviewarXiv
- [13] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281--14... on Pith
- [14] Longtao Hu, Xiao Liang, and Linchao Zhu. 2026. From interaction traces to persistent skills: Online evolution for computer-use agents. CoRR, abs/2609.04869 on PithPith's reviewarXiv
- [15] Zheng Hui, Yinheng Li, Dan Zhao, Colby R. Banbury, Tianyi Chen, and Kazuhito Koishida. 2025. Winspot: GUI grounding benchmark with multimodal large language models. In ACL (2) , pages 1086--1096. Association for Computational Linguistics on Pith
- [16] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 on PithPith's reviewarXiv
- [17] Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, et al. 2025. Zoom in, click out: Unlocking and evaluating the potential of zooming for gui grounding. arXiv preprint arXiv:2512.05941 on PitharXiv
- [18] Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, and Amit Ranjan Trivedi. 2026. Uncertainty quantification for computer-use agents: A benchmark across vision-language models and gui grounding datasets. arXiv preprin... on PithPith's reviewarXiv
- [19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princip... on Pith
- [20] Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, and Caiwen Ding. 2026. https://proceedings.mlr.press/v306/lei26h.html GUI -spotlight: Adaptive iterative focus refinement for enhanced GUI visual grounding . In Proceedings of the 43rd International Conference on ... on Pith
show 44 more
- [21] Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, and Shilong Liu. 2025 a . Chain-of-ground: Improving gui grounding via iterative reasoning and reference feedback. arXiv preprint arXiv:2512.01979 on PitharXiv
- [22] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025 b . Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778--8786 on Pith
- [23] Jiaping Lin, Fei Shen, Junzhe Li, Ping Nie, Fei Yu, Ming Li, and Haizhou Li. 2026 a . What happens before decoding? prefill determines gui grounding in vlms. arXiv preprint arXiv:2605.12549 on PithPith's reviewarXiv
- [24] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. Showui: One vision-language-action model for gui visual agent. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP... on Pith
- [25] Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, and Deheng Ye. 2026 b . Adaptvision: Efficient vision-language models via adaptive visual acquisition. In CVPR , pages 11923--11932. Computer Vision Foundation on Pith
- [26] Yichao Liu, Huawen Shen, Liu Yu, Shiyu Liu, Zeyu Chen, and Yu Zhou. 2026 a . DRS-GUI: dynamic region search for training-free GUI grounding. In CVPR , pages 34606--34616. Computer Vision Foundation on Pith
- [27] Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, et al. 2026 b . Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization. In Proceedings of the AAAI Conference on Artificia... on Pith
- [28] Zhou Liu, Ligang Huang, Zeli Su, Zewei Pan, Zhaoyang Han, Xing Chen, Yuanfeng Song, and Wentao Zhang. 2026 c . Skilllens: Visual skill cards for retrieval-augmented GUI action prediction and on-policy distillation. CoRR, abs/2608.10775 on PithPith's reviewarXiv
- [29] Ziwei Liu, Tao Feng, Borui Kang, Yanbing Yang, and Jun Luo. 2026 d . Zoom to essence: Trainless gui grounding by inferring upon interface elements. arXiv preprint arXiv:2603.14448 on PitharXiv
- [30] Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. 2025. Visual test-time scaling for gui agent grounding. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19989--19998. IEEE on Pith
- [31] Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodr \' guez, Montek Kalsi, Nicolas Chapados, M. Tamer \" O zsu, Aishwarya Agrawal, David V \' a zquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. 2025. Ui-vision: A desktop-centric GUI benchmar... on Pith
- [32] Anthony Nguyen. 2024. Improved gui grounding via iterative narrowing. arXiv preprint arXiv:2411.13591 on PithPith's reviewarXiv
- [33] Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. 2025. Gui agents: A survey. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22522--22538 on Pith
- [34] OpenAI . 2025. https://openai.com/index/introducing-operator/ Introducing Operator . OpenAI Blog on Pith
- [35] Mingyu Ouyang, Kevin Qinghong Lin, Mike Zheng Shou, and Hwee Tou Ng. 2026. Focusui: Efficient ui grounding via position-preserving visual token selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20313--20323 on Pith
- [36] Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, et al. 2026. Adazoom-gui: Adaptive zoom-based gui grounding with instruction refinement. arXiv preprint arXiv:2603.17441 on PitharXiv
- [37] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 on PithPith's reviewarXiv
- [38] Xinyu Qiu, Yunzhu Zhang, Heng Jia, Shuheng Shen, Changhua Meng, and Linchao Zhu. 2026. Vista: View-consistent self-verified training for gui grounding. arXiv preprint arXiv:2606.14579 on PitharXiv
- [39] Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, et al. 2026 a . Ui-zoomer: Uncertainty-driven adaptive zoom-in for gui grounding. arXiv preprint arXiv:2604.14113 on PithPith's reviewarXiv
- [40] Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. 2026 b . Gui-g ^2 : Gaussian reward modeling for gui grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p... on Pith
- [41] Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al. 2025. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865 on PithPith's reviewarXiv
- [42] Qingni Wang, Yue Fan, and Xin Eric Wang. 2026 a . Safeground: Know when to trust gui grounding models via uncertainty calibration. arXiv preprint arXiv:2602.02419 on PitharXiv
- [43] Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu, and Shengyu Zhang. 2026 b . Measure twice, click once: Co-evolving proposer and visual critic via reinforcement learning for gui grounding. In Proceedings of the 64th Annual Meeting of the Associ... on Pith
- [44] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met... on Pith
- [45] Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang, and Yiwei Wang. 2025 a . Dimo-gui: Advancing test-time scaling in gui grounding via modality-aware visual reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process... on Pith
- [46] Penghao Wu and Saining Xie. 2024. V*: Guided visual search as a core mechanism in multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084--13094. IEEE on Pith
- [47] Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. 2026. Gui-actor: Coordinate-free visual grounding for gui agents. Advances in Neural Information Processing Systems, 38:15101--15128 on Pith
- [48] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. 2025 b . Os-atlas: Foundation action model for generalist gui agents. In International Conference on Learning Representations, volume 2025, pag... on Pith
- [49] Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, and Hua Wu. 2026. Omegause-sop: SOP engineering for professional computer use from human demonstrations. CoRR, abs/2609.02149 on PithPith's reviewarXiv
- [50] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. 2026. Scaling computer-use grounding via user interface decomposition and synthesis. Advances in Neural Information Processing Systems, 38 on Pith
- [51] Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. 2025. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8851--8859 on Pith
- [52] Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. 2026. Gta1: Gui test-time scaling agent. In International Conference on Learning Representations, volume 2026, pages 48295--48323 on Pith
- [53] Ruilin Yao, Shegnwu Xiong, Tianyu Zou, Shili Xiong, and Yi Rong. 2026. Autofocus: Uncertainty-aware active visual search for gui grounding. arXiv preprint arXiv:2605.02630 on PithPith's reviewarXiv
- [54] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision, pages 240--255. Springer on Pith
- [55] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. 2026. Se-gui: Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. Advances in Neural Information Processing ... on Pith
- [56] Borui Zhang, Bo Zhang, Bo Wang, Wenzhao Zheng, Yuhao Cheng, Liang Tang, Yiqiang Yan, Jie Zhou, and Jiwen Lu. 2026 a . BAMI: training-free bias mitigation in GUI grounding. In CVPR , pages 34596--34605. Computer Vision Foundation on Pith
- [57] Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. https://mlanthology.org/tmlr/2025/zhang2025tmlr-large/ Large Language Model-Brained GUI Agents: A Survey . Transa... on Pith
- [58] Renshan Zhang, Haoyang Meng, Yixiao He, Rui Shao, April Hua Liu, and Liqiang Nie. 2026 b . Lookagain: Closed-loop gui grounding with visually grounded reflection. arXiv preprint arXiv:2608.09723 on PithPith's reviewarXiv
- [59] Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, and Yu Zhou. 2026 c . Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv preprint arXiv:2605.00642 on PithPith's reviewarXiv
- [60] Yunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen, Changhua Meng, and Linchao Zhu. 2026 d . Mvp: Multiple view prediction improves gui grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27482--27492 on Pith
- [61] Yu Zhao, Wei-Ning Chen, Huseyin A Inan, Samuel Kessler, Lu Wang, Lukas Wutschitz, Fangkai Yang, Chaoyun Zhang, Pasquale Minervini, Saravan Rajmohan, and Robert Sim. 2026. https://proceedings.mlr.press/v306/zhao26c.html Learning GUI grounding with spatial reasoning from visual ... on Pith
- [62] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. In ICML , volume 235 of Proceedings of Machine Learning Research, pages 61349--61385. PMLR / OpenReview.net on Pith
- [63] Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. 2026. Deepeyes: Incentivizing "thinking with images" via reinforcement learning. In ICLR . proceedings.iclr.cc on Pith
- [64] Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. 2026. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents. Advances in Neural Information Processing Systems, 38:95683--95705 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment