Pith.

Computer Vision and Pattern Recognition · arXiv:2610.36874 v1 · submitted 27 Oct 2026 · 64 references

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

The paper says

“GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy.”

more of the abstract

However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround

from the abstract

Pith says

Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A cleanly engineered, training-free GUI grounding pipeline with real gains on small targets; the size-estimation robustness is better than the stress test suggests, but the method is verified only on two Qwen backbones and four benchmarks. This line is wrong

1 major objection · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 64 references say.

The paper says, at §3.2, Appendix B.3

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

The central design of TNZ depends on the model's self-reported bounding-box size ˆs to set the zoom factor via Eq. (2) and crop size via Eq. (3). Appendix B.3 shows that the median ratio of ˆs to the true side is 1.2–1.4 on ScreenSpot-Pro and that accuracy changes by at most 0.6 points when the true size is used.

more of objection 1

However, this robustness test is performed on the same benchmarks used for development. The paper reports overall accuracy on held-out UI-Vision (Table 3) but does not analyze the quality of the size estimate or the per-sample behavior of TNZ on that distribution. Since UI-Vision has different screenshots and applications (83 desktop apps), the estimate could degrade. The authors should report, at least in the appendix, the distribution of ˆs errors on UI-Vision and confirm that the robustness result (max 0.6-point change) holds there, or discuss why the verifier and chain portfolio would absorb any degradation.

6 minor objections

The paper says, at §3.3

Pith says: minor objection 1 of 6

The verifier score v(p) is defined as the normalized first-token probability of 'Yes' vs 'No', summing over case variants and taking the 20 most likely first tokens. The exact procedure for selecting the 20 tokens and handling ties (e.g., when neither 'Yes' nor 'No' appears) should be stated more explicitly in the main text rather than only in Appendix A.1.

The paper says, at §3.3, Eq. (8)

Pith says: minor objection 2 of 6

The abstention rule uses θ_e=0.03. The paper states that this value was set for few false refusals (0.4–0.8%) but does not report sensitivity to this threshold. A brief sensitivity analysis (e.g., accuracy vs. θ_e ∈ [0.01, 0.2] as in Appendix B.2 but for both backbones) would help the reader understand how critical this choice is.

show 4 more

The paper says, at §4.1, 'Protocol'

Pith says: minor objection 3 of 6

The paper notes that outputs are not bit-exact across runs due to vLLM continuous batching. It should state explicitly whether the reported numbers are from a single run or averaged over multiple runs, and whether the confidence intervals in Table 13 account for this variance (they appear to be bootstrap intervals over screenshots, which do not capture inference randomness).

The paper says, at Table 5 and Algorithm 1

Pith says: minor objection 4 of 6

Table 5 lists many constants. While all are fixed across experiments, the paper would benefit from a compact notation in Algorithm 1 that makes it clear which constants are used at each step. Currently the procedure references 'the constants of Section 3.5' which is vague.

The paper says, at Figure 2

Pith says: minor objection 5 of 6

The overview figure is informative but the font in subfigures (a), (b), (c) is very small. In print, the text may be illegible. Consider enlarging the key panels or providing a separate high-resolution version.

The paper says, at §5, Limitations

Pith says: minor objection 6 of 6

The limitations paragraph on CSG states that it does not handle slider positions or text-cursor placement. The prompt in Appendix A.3 includes an 'along' operator and a 'text_edge' operator, but these are said not to be executed. This is confusing. Clarify whether these operators are reserved for future work or simply not evaluated here.

What would settle it

Compute the estimated-to-true size ratio on every sample of UI-Vision (the held-out benchmark) for both backbones, then group samples by ratio magnitude (e.g., <0.8, 0.8–1.2, 1.2–2, >2). Compare AdaGround's accuracy on each group to the fixed 4× crop. If AdaGround underperforms the fixed crop on the extreme groups (ratio >2 or <0.5), this would indicate that size-estimate degradation can cause failure, weakening the robustness claim.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 28 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 57 s · 36,172 tokens read, 6,577 written
  3. Editor wrote the verdict above 11 s · 37,069 tokens read, 915 written
  4. Referee minor revision: 1 major objection, 6 minor 24 s · 36,599 tokens read, 2,440 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

AdaGround does something genuinely useful. It solves five well-articulated problems in test-time zooming for GUI grounding—scale-agnostic zoom, open-loop acceptance, no second attempt, need for model internals, and no abstention—with three straightforward components: token-normalized adaptive zooming, a verifier that treats the same VLM as a black box, and compositional cell grounding. The results are consistently above strong baselines (MVP, ZoomClick) on three development benchmarks and hold on a held-out benchmark (UI-Vision). Paired tests with confidence intervals are provided, and the ablation study isolates where each component helps. The writing is clear and the limitations are honestly stated.

What the stress test calls a load-bearing flaw—the size estimate's accuracy—the paper already addresses in Appendix B.3: using the true target size changes accuracy by at most 0.6 points. That is a real empirical check, and the stress test's worry that it might degrade on a different distribution is speculative. The paper tests UI-Vision (83 desktop apps) as a held-out set and the gains carry over, so the concern is not acute.

The main soft spots are real but acknowledged. The method is evaluated only on Qwen3-VL-8B and -32B, and only on four benchmarks. The cost is high (3.5–4× visual tokens of a fixed crop). The comparison with MVP on OSWorld-G 32B is unpaired. CSG works only for spreadsheet cells. These limit the generality but do not undercut the central claim.

This is a solid, honest paper. I would bring it to a reading group and would accept a peer-review invitation. It deserves a serious referee.

The strongest objection

The paper's central claim depends on the ability of the model's self-reported bounding box to set the zoom factor via Eq. (2) and crop size via Eq. (3). Appendix B.3 shows that the median ratio of estimated to true side is 1.2–1.4 on ScreenSpot-Pro, causing under-magnification on average, and that accuracy changes by at most 0.6 points when using the true size. However, this robustness test is performed on the same benchmarks used for development. The paper does not evaluate how the size estimate behaves on a systematically different distribution—e.g., UI-Vision's 1,181 screenshots spanning 83 desktop applications—nor does it test the combined effect of estimation error on the verifier's arbitration and abstention decisions, which use scores derived from the same model. If the estimate were substantially worse on a new distribution, the margin of 0.6 points could be exceeded, reducing the claimed improvements.

The reader identified the size estimate as the weakest assumption, and I agree it is the most load-bearing component. However, the paper's robustness test (replacing estimated size with true size) is limited to the development benchmarks. A stronger stress test would examine performance on a held-out benchmark (UI-Vision) on samples where the estimate is most inaccurate. The results on UI-Vision overall are strong, suggesting the concern may not materialize, but the analysis is not currently disaggregated. This does not invalidate the central claim, but a targeted check would settle whether the weakest assumption actually holds beyond the development set. Hence the verdict remains unchanged.

The referee's report

If the claims hold, the paper makes a practical and principled contribution to GUI grounding. The insight that zoom geometry should be set by the target's token span rather than by fixed or heuristic ratios is novel and well motivated. The use of a black-box verifier (one Yes/No question) to replace attention map access and voting is a clean design that broadens applicability. The held-out evaluation on UI-Vision strengthens the claim that the method generalizes beyond the three development benchmarks. The fact that all constants are shared across backbones and benchmarks is a practical advantage. The gains are large (up to 20.6 points) and statistically significant in paired tests. The code release supports reproducibility.

Pith's reading of the paper (a machine's summary, not the authors')

AdaGround is a training-free framework that improves GUI grounding on high-resolution screens by solving five shortcomings of existing test-time zooming methods. It uses Token-Normalized Adaptive Zooming (TNZ) to derive zoom magnification, crop size, and depth from the model's own target-size estimate, so the target occupies a fixed number of visual tokens (3 tokens). A Verifier-in-the-Loop Refinement (VLR) replaces open-loop acceptance with a Yes/No question about a marked click, using that score to keep or reject each zoom, choose between a TNZ chain and a fixed 4× crop, trigger re-grounding when unresolved, and abstain when the target is absent. Compositional Structure Grounding (CSG) handles spreadsheet cells by composing separately grounded headers. On Qwen3-VL-8B and 32B, AdaGround improves accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, surpassing state-of-the-art training-free methods (MVP, ZoomClick) and many trained models including 72B ones; gains carry over to held-out UI-Vision. A single set of constants serves both backbones and all benchmarks.

What carries the argument. The central mechanism is the token span σ = s·k/π, where s is the target's shorter side in original pixels, k is the linear scale of the view, and π is the token pitch (32 pixels for Qwen3-VL). Token-Normalized Adaptive Zooming (TNZ) solves Eq. (2) to set the magnification k = clip(τ·π/ŝ, 1, kmax) so that the estimated target spans τ = 3 tokens, and Eq. (3) to set the crop side d = clip(max(γŝ, ε·π/kprev), dmin, dmax) to cover context and localization error. The Verifier-in-the-Loop Refinement (VLR) uses v(p) = P(YES | marked-click question) / (P(YES)+P(NO)) from the model's first-token probabilities to gate acceptance, arbitrate between chains (Eq. (7)), trigger re-grounding, and decide abstention (Eq. (8)). The key insight is that the model's own predictions drive both zoom and verification, eliminating the need for attention maps or training.

Core claim. The central claim is that using the model's own size estimate to set a zoom that renders the target at exactly three token spans, and replacing open-loop acceptance and voting with model-driven verification and chain arbitration, can significantly improve GUI grounding without any training or access to internal model states. The paper demonstrates this with Qwen3-VL, showing consistent gains on high-resolution and small-target benchmarks. It shows that two different zoom geometries (TNZ and a fixed 4× crop) fail on different samples, and that the model's own Yes/No question can pick the right answer over a third of the way from the better chain to the oracle. The framework recovers 39% and 36% of the headroom above the better chain on ScreenSpot-Pro with Qwen3-VL-8B and 32B, respectively.

The premise it rests on. The paper assumes that the model's self-reported bounding box is accurate enough to set the zoom factor via Eq. (2); the paper acknowledges this estimate is coarse (median ratio 1.2–1.4 on ScreenSpot-Pro) but shows empirical robustness on the tested benchmarks.

If the paper is right (Pith's inferences, not the authors')

GUI agents can achieve higher grounding accuracy without re-training, using the same VLM they already deploy, by simply modifying the inference procedure.

The verification-based chain portfolio can be extended to more than two zoom strategies or to non-zoom views (e.g., different rendering scales), potentially improving robustness further.

The token span criterion provides a principled way to decide when and how much to zoom across different screen resolutions and target sizes, which could generalize to other VLMs that tile images into tokens.

The abstention mechanism (Eq. (8)) reduces false positives on absent-target instructions and can be combined with other systems to make GUI agents safer in tasks requiring confidence.

The framework's success suggests that the VLM's own confidence on a marked-click Yes/No question is a reliable indicator of grounding quality, potentially applicable to other perception tasks like object detection or visual question answering.

The constant τ = 3 was chosen based on an empirical accuracy plateau near 3 tokens on Qwen3-VL; on a different VLM with different resolution handling (e.g., smaller or larger token pitch), the optimal τ may differ, so the method may require retuning the single parameter when switching backbones.

The reliance on the model's box estimate means that if the model fails to output a bounding box (e.g., returns only a point or refuses), TNZ will not zoom; the paper acknowledges this and falls back to the fixed 4× chain and re-grounding, but on models that rarely produce boxes the benefit of TNZ is lost.

The cost (3.5–3.9× visual tokens, 3.4–4.2× latency) is nontrivial for latency-sensitive agent loops; however, the paper shows that arbitration alone (about 10 calls) already beats the fixed crop on ScreenSpot-Pro, offering a potential efficiency–accuracy trade-off.

The verifier's single threshold θr = 0.99 and abstention threshold θe = 0.03 were set on OSWorld-G and may not transfer to domains with different answer distributions; the paper tests θr = 0.9 and finds no significant change on the benchmarks, but this may not hold in all scenarios.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-28

The authors' main claim, as the classifier picked it out 1.00

“On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.85 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. background 1.00 0.11 0.03 asserted 1.00
However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. gap 0.96 0.06 0.03 asserted 1.00
To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. method 0.58 0.89 0.07 asserted 0.99
On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. result 1.00 0.96 0.83 asserted 1.00
Code is available at https://github.com/bcwang-sjtu/AdaGround contribution 0.70 0.44 0.02 asserted 0.92

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 4,562 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Accuracy of AdaGround, the single pass and
Figure 2 of the paper
Figure 2. Overview of AdaGround. Compositional Structure Grounding (CSG) reads the instruction. Two zoom chains, Token-Normalized Adaptive Zooming (TNZ) and a fixed 4× crop, run on every sample. Verifier-in-the-Loop Refinement (VLR) uses one Yes/No verifier to keep a zoom, pick a chain, retry with the rejected click crossed out, and abstain. Top: the pipeline on one ScreenSpot-Pro sample with Qwen3-VL-8B (real model outputs), where v badges mark the model’s Yes/No checks. Bottom: (a) TNZ, (b) VLR and (c) CSG, of which (a) and (c) are schematic. revisit. (iv) White-box, vote-based aggregation.…
Figure 3 of the paper
Figure 3. Chain arbitration (counts: samples on which only that chain is right). The Yes/No check runs when￾ever the two clicks differ. With Qwen3-VL-32B the three bars read 72.4, 56.9 and 50.0%. tokens. TNZ as a gated chain (cTNZ) improves the single pass by 4.8 to 17.1 points on OSWorld-G and ScreenSpot-Pro and beats the fixed crop on held-out UI-Vision. On ScreenSpot-Pro the two chains disagree on 45 and 34% of samples, and chain arbitration (
Figure 4 of the paper
Figure 4. Accuracy versus token span per benchmark (bins with ≥ 30 views) and pooled (dashed), on views that contain the target (
Figure 5 of the paper
Figure 5. Within-sample change from the crop pass (

Figures are the authors', reproduced from the paper.

References (64)
  1. [1] Anthropic . 2024. https://www.anthropic.com/news/3-5-models-and-computer-use Introducing computer use, a new Claude 3.5 Sonnet , and Claude 3.5 Haiku . Anthropic News on Pith
  2. [2] Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. https://doi.org/10.24963/ijcai.2024/339 Screenai: A vision-language model for ui and infographics understanding . In ... on Pithdoi:10.24963/ijcai.2024/339
  3. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025 a . Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 on PithPith's reviewarXiv
  4. [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025 b . https://arxiv.org/abs/2502... on PithPith's reviewarXiv
  5. [5] Liangyu Chen, Hanzhang Zhou, Jianan Zhang, Panrong Tong, Xu Zhang, Quyu Kong, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, et al. 2026. Ui-ins: Enhancing gui grounding with multi-perspective instruction as reasoning. In International Conference on Learning Representations, volu... on Pith
  6. [6] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap... on Pith
  7. [7] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic c... on PithPith's reviewarXiv
  8. [8] Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, and Yongliang Shen. 2026. Test-time reinforcement learning for gui grounding via region consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30593--30601 on Pith
  9. [9] Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, et al. 2026. Gui-lens: Coarse-to-fine cropping for gui grounding with general-purpose vlms. arXiv preprint arXiv:2608.03270 on PithPith's reviewarXiv
  10. [10] Boyu Gou, Demi Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/4ca0e369689dadb25a5345ba9755ad6f-Paper-Conference.pdf Navigating the digital world as humans do: Universal visual gr... on Pith
  11. [11] Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, et al. 2025. Ui-venus technical report: Building high-performance ui agents with rft. arXiv preprint arXiv:2508.10833 on PithPith's reviewarXiv
  12. [12] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. 2025. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062 on PithPith's reviewarXiv
  13. [13] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281--14... on Pith
  14. [14] Longtao Hu, Xiao Liang, and Linchao Zhu. 2026. From interaction traces to persistent skills: Online evolution for computer-use agents. CoRR, abs/2609.04869 on PithPith's reviewarXiv
  15. [15] Zheng Hui, Yinheng Li, Dan Zhao, Colby R. Banbury, Tianyi Chen, and Kazuhito Koishida. 2025. Winspot: GUI grounding benchmark with multimodal large language models. In ACL (2) , pages 1086--1096. Association for Computational Linguistics on Pith
  16. [16] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 on PithPith's reviewarXiv
  17. [17] Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, et al. 2025. Zoom in, click out: Unlocking and evaluating the potential of zooming for gui grounding. arXiv preprint arXiv:2512.05941 on PitharXiv
  18. [18] Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, and Amit Ranjan Trivedi. 2026. Uncertainty quantification for computer-use agents: A benchmark across vision-language models and gui grounding datasets. arXiv preprin... on PithPith's reviewarXiv
  19. [19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princip... on Pith
  20. [20] Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, and Caiwen Ding. 2026. https://proceedings.mlr.press/v306/lei26h.html GUI -spotlight: Adaptive iterative focus refinement for enhanced GUI visual grounding . In Proceedings of the 43rd International Conference on ... on Pith
show 44 more
  1. [21] Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, and Shilong Liu. 2025 a . Chain-of-ground: Improving gui grounding via iterative reasoning and reference feedback. arXiv preprint arXiv:2512.01979 on PitharXiv
  2. [22] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025 b . Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778--8786 on Pith
  3. [23] Jiaping Lin, Fei Shen, Junzhe Li, Ping Nie, Fei Yu, Ming Li, and Haizhou Li. 2026 a . What happens before decoding? prefill determines gui grounding in vlms. arXiv preprint arXiv:2605.12549 on PithPith's reviewarXiv
  4. [24] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. Showui: One vision-language-action model for gui visual agent. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP... on Pith
  5. [25] Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, and Deheng Ye. 2026 b . Adaptvision: Efficient vision-language models via adaptive visual acquisition. In CVPR , pages 11923--11932. Computer Vision Foundation on Pith
  6. [26] Yichao Liu, Huawen Shen, Liu Yu, Shiyu Liu, Zeyu Chen, and Yu Zhou. 2026 a . DRS-GUI: dynamic region search for training-free GUI grounding. In CVPR , pages 34606--34616. Computer Vision Foundation on Pith
  7. [27] Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, et al. 2026 b . Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization. In Proceedings of the AAAI Conference on Artificia... on Pith
  8. [28] Zhou Liu, Ligang Huang, Zeli Su, Zewei Pan, Zhaoyang Han, Xing Chen, Yuanfeng Song, and Wentao Zhang. 2026 c . Skilllens: Visual skill cards for retrieval-augmented GUI action prediction and on-policy distillation. CoRR, abs/2608.10775 on PithPith's reviewarXiv
  9. [29] Ziwei Liu, Tao Feng, Borui Kang, Yanbing Yang, and Jun Luo. 2026 d . Zoom to essence: Trainless gui grounding by inferring upon interface elements. arXiv preprint arXiv:2603.14448 on PitharXiv
  10. [30] Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. 2025. Visual test-time scaling for gui agent grounding. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19989--19998. IEEE on Pith
  11. [31] Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodr \' guez, Montek Kalsi, Nicolas Chapados, M. Tamer \" O zsu, Aishwarya Agrawal, David V \' a zquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. 2025. Ui-vision: A desktop-centric GUI benchmar... on Pith
  12. [32] Anthony Nguyen. 2024. Improved gui grounding via iterative narrowing. arXiv preprint arXiv:2411.13591 on PithPith's reviewarXiv
  13. [33] Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. 2025. Gui agents: A survey. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22522--22538 on Pith
  14. [34] OpenAI . 2025. https://openai.com/index/introducing-operator/ Introducing Operator . OpenAI Blog on Pith
  15. [35] Mingyu Ouyang, Kevin Qinghong Lin, Mike Zheng Shou, and Hwee Tou Ng. 2026. Focusui: Efficient ui grounding via position-preserving visual token selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20313--20323 on Pith
  16. [36] Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, et al. 2026. Adazoom-gui: Adaptive zoom-based gui grounding with instruction refinement. arXiv preprint arXiv:2603.17441 on PitharXiv
  17. [37] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 on PithPith's reviewarXiv
  18. [38] Xinyu Qiu, Yunzhu Zhang, Heng Jia, Shuheng Shen, Changhua Meng, and Linchao Zhu. 2026. Vista: View-consistent self-verified training for gui grounding. arXiv preprint arXiv:2606.14579 on PitharXiv
  19. [39] Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, et al. 2026 a . Ui-zoomer: Uncertainty-driven adaptive zoom-in for gui grounding. arXiv preprint arXiv:2604.14113 on PithPith's reviewarXiv
  20. [40] Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. 2026 b . Gui-g ^2 : Gaussian reward modeling for gui grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p... on Pith
  21. [41] Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al. 2025. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865 on PithPith's reviewarXiv
  22. [42] Qingni Wang, Yue Fan, and Xin Eric Wang. 2026 a . Safeground: Know when to trust gui grounding models via uncertainty calibration. arXiv preprint arXiv:2602.02419 on PitharXiv
  23. [43] Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu, and Shengyu Zhang. 2026 b . Measure twice, click once: Co-evolving proposer and visual critic via reinforcement learning for gui grounding. In Proceedings of the 64th Annual Meeting of the Associ... on Pith
  24. [44] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met... on Pith
  25. [45] Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang, and Yiwei Wang. 2025 a . Dimo-gui: Advancing test-time scaling in gui grounding via modality-aware visual reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process... on Pith
  26. [46] Penghao Wu and Saining Xie. 2024. V*: Guided visual search as a core mechanism in multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084--13094. IEEE on Pith
  27. [47] Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. 2026. Gui-actor: Coordinate-free visual grounding for gui agents. Advances in Neural Information Processing Systems, 38:15101--15128 on Pith
  28. [48] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. 2025 b . Os-atlas: Foundation action model for generalist gui agents. In International Conference on Learning Representations, volume 2025, pag... on Pith
  29. [49] Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, and Hua Wu. 2026. Omegause-sop: SOP engineering for professional computer use from human demonstrations. CoRR, abs/2609.02149 on PithPith's reviewarXiv
  30. [50] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. 2026. Scaling computer-use grounding via user interface decomposition and synthesis. Advances in Neural Information Processing Systems, 38 on Pith
  31. [51] Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. 2025. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8851--8859 on Pith
  32. [52] Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. 2026. Gta1: Gui test-time scaling agent. In International Conference on Learning Representations, volume 2026, pages 48295--48323 on Pith
  33. [53] Ruilin Yao, Shegnwu Xiong, Tianyu Zou, Shili Xiong, and Yi Rong. 2026. Autofocus: Uncertainty-aware active visual search for gui grounding. arXiv preprint arXiv:2605.02630 on PithPith's reviewarXiv
  34. [54] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision, pages 240--255. Springer on Pith
  35. [55] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. 2026. Se-gui: Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. Advances in Neural Information Processing ... on Pith
  36. [56] Borui Zhang, Bo Zhang, Bo Wang, Wenzhao Zheng, Yuhao Cheng, Liang Tang, Yiqiang Yan, Jie Zhou, and Jiwen Lu. 2026 a . BAMI: training-free bias mitigation in GUI grounding. In CVPR , pages 34596--34605. Computer Vision Foundation on Pith
  37. [57] Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. https://mlanthology.org/tmlr/2025/zhang2025tmlr-large/ Large Language Model-Brained GUI Agents: A Survey . Transa... on Pith
  38. [58] Renshan Zhang, Haoyang Meng, Yixiao He, Rui Shao, April Hua Liu, and Liqiang Nie. 2026 b . Lookagain: Closed-loop gui grounding with visually grounded reflection. arXiv preprint arXiv:2608.09723 on PithPith's reviewarXiv
  39. [59] Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, and Yu Zhou. 2026 c . Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv preprint arXiv:2605.00642 on PithPith's reviewarXiv
  40. [60] Yunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen, Changhua Meng, and Linchao Zhu. 2026 d . Mvp: Multiple view prediction improves gui grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27482--27492 on Pith
  41. [61] Yu Zhao, Wei-Ning Chen, Huseyin A Inan, Samuel Kessler, Lu Wang, Lukas Wutschitz, Fangkai Yang, Chaoyun Zhang, Pasquale Minervini, Saravan Rajmohan, and Robert Sim. 2026. https://proceedings.mlr.press/v306/zhao26c.html Learning GUI grounding with spatial reasoning from visual ... on Pith
  42. [62] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. In ICML , volume 235 of Proceedings of Machine Learning Research, pages 61349--61385. PMLR / OpenReview.net on Pith
  43. [63] Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. 2026. Deepeyes: Incentivizing "thinking with images" via reinforcement learning. In ICLR . proceedings.iclr.cc on Pith
  44. [64] Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. 2026. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents. Advances in Neural Information Processing Systems, 38:95683--95705 on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment