Pith. sign in

REVIEW 4 major objections 4 minor 28 references

The paper argues that GUI grounding failures in VLMs are a coordinate-output bottleneck, not a spatial-understanding gap, and that zero-shot overlays of axes and grids can lift click accuracy from roughly 5% to about 72% on ScreenSpot-v2.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:44 UTC pith:CT4TDGWE

load-bearing objection Strong empirical finding on grid overlays for GUI grounding, but the latent-ability mechanism is built on an oracle and the headline numbers are tuned on the test set. the 4 major comments →

arxiv 2509.11548 v2 pith:CT4TDGWE submitted 2025-09-15 cs.CV

How Auxiliary Reasoning Unleashes GUI Grounding in VLMs

Weiming Li , Yan Shao , Jing Yang , Yujing Lu , Ling Zhong , Yuhan Wang , Min Yu , Tongxiao Ruan
show 1 more author
Manni Duan
This is my paper
classification cs.CV
keywords GUI groundingvision-language modelzero-shot auxiliary reasoningspatial overlaysPointing Gamecoordinate predictionattention mapScreenSpot benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Directly asking a vision-language model to output click coordinates on a GUI screenshot works poorly, even though the model often knows where the target is internally. The paper establishes that this is a communication gap, not a perceptual one: attention-map localization finds the target far more often than direct coordinate prediction does. To close the gap without retraining, the authors overlay spatial cues on the input—axes, grids, labeled intersections, and numbered grid cells—and ask the model to reason with those cues. The strongest variant, Mark-Grid Scaffold, converts continuous coordinate prediction into discrete grid-ID prediction followed by a zoomed-in second pass, lifting one proprietary VLM from roughly 5% to about 72% click accuracy on ScreenSpot-v2. If the claim holds, zero-shot inference-time overlays can capture much of the benefit of fine-tuned GUI grounding without the million-sample annotation cost.

Core claim

The paper's key observation is that VLMs possess latent GUI grounding ability: when the model's attention map (last text token to all image tokens, averaged over heads) is used to pick the highest-attention point, accuracy is far higher than when the model directly emits coordinates. The paper argues this gap shows the model 'knows' where to click but cannot articulate it in coordinate form. It then proposes three auxiliary reasoning methods; Mark-Grid Scaffold overlays an 8×8 grid with unique cell IDs, has the model predict grid IDs for the object's left, top, right, and bottom extremities, crops and magnifies that region, repeats the grid-ID prediction once, and derives the final click poi

What carries the argument

The central diagnostic is the Pointing Game: attention weights from the model's last text token to all image tokens are averaged over heads, reshaped into a spatial map, resized to the image, and the coordinate of maximum attention is checked against the ground-truth region, with a union across layers reporting the best observed accuracy. The central remedy is Mark-Grid Scaffold, which overlays an 8×8 grid of labeled cells on the screenshot, prompts the VLM to predict four grid IDs bounding the target, crops and enlarges that region, and repeats with a fresh 8×8 grid; the click point is the center of the final bounds. This converts unbounded coordinate regression into a sequence of discrete

Load-bearing premise

The load-bearing premise is that the overlay configuration chosen by ablations on ScreenSpot-v2—a 100-pixel axis interval, an 8×8 grid, and one zoom-in level—is not overfit to that benchmark, even though the headline accuracy is reported on that same benchmark.

What would settle it

Fix the overlay configuration without looking at the target benchmark's labels, then run Mark-Grid Scaffold on a fresh GUI split or a held-out benchmark; if the gain over direct prediction collapses to near the baseline, the claim that overlays unleash latent grounding rather than recycle benchmark-specific tuning would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Models without grounding-specific training show the largest improvements; a grounding-specialized VLM (Qwen2-VL-7B) degrades with the scaffolds, implying the method targets unused latent spatial ability.
  • Zero-shot GUI grounding can approach fine-tuned baselines on ScreenSpot and UI-I2E-Bench without annotated data or weight updates.
  • The Pointing Game gap can serve as a pre-inference diagnostic: if attention already localizes targets, an overlay is worth applying; if not, direct fine-tuning is likely needed.
  • The ablations indicate that effective overlays need explicit spatial information, integration of multiple cues, and balanced granularity—too sparse or too dense grids hurt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same overlay recipe may extend beyond GUI screenshots to other fine-grained localization tasks where VLMs have strong latent attention but weak coordinate output, such as document layout or medical imaging.
  • The inverse relationship between grid size and zoom depth suggests an adaptive zoom budget could reduce inference cost: spend more zoom levels when the model is uncertain, fewer when it is confident.
  • Because the headline configuration was selected on ScreenSpot-v2 and then reported on ScreenSpot-v2, a held-out transfer test would separate a general spatial-cue effect from benchmark-specific tuning; the relative gain over direct prediction is likely to survive, but the absolute 72% may shrink.
  • The Pointing Game diagnostic plus overlay performance gives a cheap model-selection rule for GUI agents, letting a system decide per model whether to apply a scaffold or rely on direct prediction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes three zero-shot auxiliary reasoning methods—Coordinate Scaffold, Axis-Grid Scaffold, and Mark-Grid Scaffold—that overlay spatial cues (axes, grids, labeled intersections, and grid-ID labels) on GUI screenshots to improve VLM coordinate prediction without fine-tuning. The authors first use Pointing Game to argue that VLMs have latent GUI-grounding ability that direct coordinate prediction fails to exploit, then evaluate the three methods on four GUI-grounding benchmarks across seven VLMs. The central quantitative claim is that Mark-Grid Scaffold improves Gemini-2.5-Flash on ScreenSpot-v2 from 5.50% to 72.09%, with consistent gains across most models and benchmarks.

Significance. If the empirical gains are robust, the paper demonstrates a surprisingly simple and inexpensive way to improve GUI grounding in VLMs, with direct practical relevance for GUI agents and zero-shot grounding. The paper's strengths include evaluation across seven models and four benchmarks, component-level ablations, and a public code release. However, the current text contains unsupported headline claims, a potential test-set-tuning issue, and an oracle-based diagnostic that weakens the proposed mechanistic explanation. These issues need to be addressed before the stated significance can be accepted.

major comments (4)
  1. [Abstract and Table 1] The abstract's headline result — Mark-Grid Scaffold boosts Gemini-3.1-Pro from 11.72% to 95.20% on ScreenSpot-v2 — is not supported anywhere in the manuscript. Table 1 contains no Gemini-3.1-Pro entry; the closest model, Gemini-2.5-Flash, goes from 5.66% to 72.09%. In addition, the abstract claims state-of-the-art performance on ScreenSpot and near fine-tuned performance on ScreenSpot-v2/UI-I2E-Bench, but Table 1 includes only the proposed methods, Direct Prediction, Grid-Augmented Vision, and Scaffold Prompting; no fine-tuned or prior GUI-grounding systems are compared. Please reconcile the reported numbers and either add the comparisons or remove the unsupported SOTA claims.
  2. [§3.2.3 and Table 1] The default configuration of the proposed methods is selected using the ScreenSpot-v2 test set with Gemini-2.5-Flash: Figure 3(c) chooses a 100-pixel axis interval, and Figure 3(e) chooses an 8×8 grid with one zoom-in level over the best-performing but more expensive 5×5/two-level option. Table 1 then reports the headline 72.09% on the same benchmark and model. This is test-set tuning, so the reported ScreenSpot-v2 gains may be optimistically biased. The consistent gains on the other three benchmarks reduce this concern, but they use the same ScreenSpot-v2-selected hyperparameters. Please provide a validation protocol that does not use the test set for model selection (e.g., a held-out split, nested cross-validation, or pre-registered defaults), or show that the conclusions are robust across the ablation configurations on all benchmarks.
  3. [§2.1 (Eq. 5), Figure 2, Conclusion] The Pointing Game success criterion in Eq. (5) is a per-sample oracle over attention layers: a sample is counted correct if any layer's arg-max falls inside the ground-truth box. The paper acknowledges this is an upper bound and not directly usable, but the abstract, Figure 2, and the conclusion use the gap between this oracle and Direct Prediction to conclude that VLMs possess considerable latent grounding capabilities and that the proposed scaffolds bridge this gap. Because no inference procedure can select the best layer without ground truth, the reported gap (e.g., Gemma-3-12B 47.56 vs 9.59 on ScreenSpot) may substantially overstate accessible spatial knowledge. Please report a non-oracle variant — fixed layer, attention aggregation across layers, or per-layer curves — and state how the gap changes. The empirical gains of the scaffolds are measured against Direct Prediction and remai
  4. [§3.1.2, Table 1] The comparison set is too narrow to support the abstract's 'state-of-the-art on ScreenSpot' and 'approaches the strongest fine-tuned methods' claims. The table includes only Direct Prediction, Grid-Augmented Vision, Scaffold Prompting, and the proposed methods. Existing GUI-grounding systems such as SeeClick, UGround, OS-ATLAS, CogAgent, and OmniParser, as well as recent inference-time methods (attention-driven grounding, iterative narrowing, ReGuide), are cited in the introduction but never evaluated on the four benchmarks. Please add these comparisons, or limit the claims to 'improves over the tested zero-shot baselines.'
minor comments (4)
  1. [§2.2] The method is called 'Mark-Grid Scaffold' throughout most of the paper but 'Marked-Grid Scaffold' in the bullet list of Section 2.2. Please unify the terminology.
  2. [§3.2.2] The discussion of Qwen2-VL-7B says 'we observe a performance decrease,' which understates the effect: on ScreenSpot, Mark-Grid drops from 50.24 to 26.34, and on UI-I2E-Bench from 26.61 to 15.44. Please quantify the drop and qualify the scope of the method to models without grounding-specific training.
  3. [Table 1] Pointing Game scores are listed in the same table as click accuracy, but they are attention-hit rates rather than coordinate-prediction accuracy. Please clarify in the caption that Pointing Game scores are not directly comparable to click-accuracy values.
  4. [§3.1.1] The benchmark is called 'ScreenSpot-Pro' in Section 3.1.1 but 'ScreenSpot-pro' in Table 1. Please standardize the capitalization.

Circularity Check

0 steps flagged

No significant circularity; empirical benchmark results are self-contained and the Pointing Game oracle is explicitly acknowledged as a diagnostic upper bound.

full rationale

The paper's central contributions are evaluated by direct measurements on four GUI benchmarks (Table 1). The proposed auxiliary methods (Coordinate, Axis-Grid, Mark-Grid) are applied at inference time and compared to baselines; none of the reported click-accuracy numbers is obtained by fitting a parameter and then re-predicting a closely related quantity. There is no self-citation chain: refs [13] and [14] are prior independent work, not the authors' own. The only potentially concerning step is §2.1/Fig. 2, where Pointing Game uses a per-sample union over attention layers (Eq. 5), making the measured 'latent grounding' an upper bound rather than an accessible inference procedure. However, the text explicitly states that this is a diagnostic indicator and not directly applicable for grounding inference, and the auxiliary methods do not use Pointing Game as an input. Thus the gap may be inflated, but it is not a circular derivation. Ablations on ScreenSpot-v2 followed by final evaluation on ScreenSpot-v2 raise benchmark-tuning concerns, but that is a validity/overfitting issue, not circularity under the stated criteria.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper adds no invented entities and uses no fitted mathematical constants. The free parameters are grid sizes and zoom levels selected on ScreenSpot-v2; the axioms are the oracle validity of attention-based Pointing Game, bounding-box containment as success, unaltered UI semantics under overlay, and stability of single-run VLM evaluations.

free parameters (3)
  • Axis-Grid interval = 100 pixels
    Chosen via ablation on ScreenSpot-v2 (Figure 3c); 100 beats 50, 150, and 200.
  • Mark-Grid grid size = 8x8
    Chosen as default after ablations over 5x5, 8x8, 10x10, and 13x13 to balance performance and cost; tuned on ScreenSpot-v2.
  • Mark-Grid zoom-in level = 1
    Chosen from 0, 1, and 2; 8x8 with one zoom gives the best balance and is selected as the default on ScreenSpot-v2.
axioms (4)
  • domain assumption Attention maps from the last text token to image tokens indicate spatial grounding, and union over layers with ground-truth selection is a valid measure of latent capability.
    Used in Section 2.1 to define Pointing Game and to claim a large gap versus Direct Prediction; this is an oracle metric, not a deployable inference method.
  • domain assumption Click accuracy measured by containment in the ground-truth bounding box is the correct success criterion.
    Borrowed from GLIP and used for all benchmarks; ignores whether a click inside a large box is a good interaction.
  • domain assumption Overlaying grids and labels does not materially alter the semantic content of the UI screenshot for the VLM.
    Central to all three methods; if overlays occlude or confuse small UI elements, the reported gains would not transfer.
  • domain assumption Single deterministic evaluations of proprietary and open-source VLMs are representative.
    Table 1 reports one accuracy number per condition with no repeated sampling; this assumes low output variance.

pith-pipeline@v1.3.0-alltime-deepseek · 7573 in / 12270 out tokens · 123787 ms · 2026-08-04T16:44:27.628596+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of How Auxiliary Reasoning Unleashes GUI Grounding in VLMs." pith.science (2026). https://pith.science/paper/CT4TDGWE

@misc{pith2026250911548,
  author       = {Pith},
  title        = {Pith review of: How Auxiliary Reasoning Unleashes GUI Grounding in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CT4TDGWE}},
  note         = {Machine review of arXiv:2509.11548}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Graphical user interface (GUI) grounding is a fundamental task for building GUI agents. However, general vision-language models (VLMs) struggle with this task due to a lack of specific optimization. We identify a key gap in this paper: while VLMs exhibit significant latent grounding potential, as demonstrated by their performance measured by Pointing Game, they underperform when tasked with outputting explicit coordinates. To address this discrepancy and bypass the high data and annotation costs of current fine-tuning approaches, we propose three zero-shot auxiliary reasoning methods. By providing explicit spatial cues such as axes, grids and labeled intersections as part of the input image, these methods enable VLMs to better articulate their implicit spatial understanding capabilities. We evaluate these methods on four GUI grounding benchmarks across seven open-source and proprietary VLMs. Experimental results show substantial gains from auxiliary reasoning. Mark-Grid Scaffold boosts Gemini-3.1-Pro from 11.72\% under direct inference to 95.20\% on ScreenSpot-v2, achieves state-of-the-art performance on ScreenSpot, and approaches the strongest fine-tuned methods on ScreenSpot-v2 and UI-I2E-Bench. Our code is available at https://github.com/liweim/AuxiliaryReasoning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 10 linked inside Pith

  1. [1]

    Beyond identification, this task requires the model to output precise screen coordinates, such as a click point or a bounding box

    INTRODUCTION Graphical user interface (GUI) grounding is a specialized form of visual grounding that involves locating specific screen components based on natural language instructions. Beyond identification, this task requires the model to output precise screen coordinates, such as a click point or a bounding box. This capability is essential for GUI age...

    Pith/arXiv arXiv 2025

  2. [2]

    Where should I click if I want to {instruction}?

    UNLEASHING THE GUI GROUNDING CAPABILITY In this section, we first detail the implementation of Pointing Game, a method we employ to investigate the latent grounding capability in- herent in VLMs. Then, we propose three auxiliary reasoning meth- ods that exploit this latent ability to enhance spatial reasoning and bridge the performance gap. 2.1. Implement...

  3. [3]

    Experimental Settings This section describes the experimental setup used to evaluate the performance of our proposed auxiliary reasoning methods

    EXPERIMENTS 3.1. Experimental Settings This section describes the experimental setup used to evaluate the performance of our proposed auxiliary reasoning methods. 3.1.1. Benchmarks We evaluate the methods on four GUI grounding benchmarks: •ScreenSpot[2]: A benchmark for GUI visual grounding, containing 1,272 single-step instructions with corresponding tar...

  4. [4]

    The diagnostic Pointing Game demonstrates this gap

    CONCLUSION We discover that general VLMs possess a strong latent potential for GUI grounding that is not fully realized by direct inference. The diagnostic Pointing Game demonstrates this gap. By simply adding spatial cues, we show that this gap can be effectively bridged. The proposed zero-shot auxiliary reasoning methods use simple visual cues, includin...

  5. [5]

    Gpt-4 technical report,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    SeeClick: Harnessing GUI grounding for advanced visual GUI agents,

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li Yan- Tao, Jianbing Zhang, and Zhiyong Wu, “SeeClick: Harnessing GUI grounding for advanced visual GUI agents,” inProceed- ings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). Aug. 2024, pp. 9313–9332, Association for Computational Linguistics

    2024

  7. [7]

    OS-ATLAS: Foundation action model for generalist GUI agents,

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao, “OS-ATLAS: Foundation action model for generalist GUI agents,” inThe Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  8. [8]

    Screenspot-pro: GUI grounding for professional high- resolution computer use,

    Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua, “Screenspot-pro: GUI grounding for professional high- resolution computer use,” inWorkshop on Reasoning and Planning for Large Language Models, 2025

    2025

  9. [9]

    Top-down neural attention by excitation backprop,

    Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff, “Top-down neural attention by excitation backprop,”International Journal of Computer Vision, vol. 126, pp. 1084–1102, 2018

    2018

  10. [10]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents,

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su, “Navigating the digital world as humans do: Universal visual grounding for GUI agents,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [11]

    Aguvis: Unified pure vision agents for autonomous GUI in- teraction,

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong, “Aguvis: Unified pure vision agents for autonomous GUI in- teraction,” inWorkshop on Reasoning and Planning for Large Language Models, 2025

    2025

  12. [12]

    Cogagent: A visual language model for gui agents,

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wen- meng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al., “Cogagent: A visual language model for gui agents,” inProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2024, pp. 14281–14290

    2024

  13. [13]

    Omniparser: A unified framework for text spotting key infor- mation extraction and table recognition,

    Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang, “Omniparser: A unified framework for text spotting key infor- mation extraction and table recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, 2024, pp. 15641–15653

    2024

  14. [14]

    Gui-actor: Coordinate-free visual ground- ing for gui agents,

    Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jian- wei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al., “Gui-actor: Coordinate-free visual ground- ing for gui agents,”arXiv preprint arXiv:2506.03143, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization,

    Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, et al., “Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization,”arXiv preprint arXiv:2508.05731, 2025

    arXiv 2025

  16. [16]

    Iris: Breaking gui complexity with adap- tive focus and self-refining,

    Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang, “Iris: Breaking gui complexity with adap- tive focus and self-refining,”arXiv preprint arXiv:2412.10342, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Grid-augmented vision: A simple yet effective approach for enhanced spatial understanding in multi-modal agents,

    Joongwon Chae, Zhenyu Wang, Lian Zhang, Dongmei Yu, and Peiwu Qin, “Grid-augmented vision: A simple yet effective approach for enhanced spatial understanding in multi-modal agents,”arXiv preprint arXiv:2411.18270, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Scaffolding coordinates to promote vision-language co- ordination in large multi-modal models,

    Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu, “Scaffolding coordinates to promote vision-language co- ordination in large multi-modal models,” inProceedings of the 31st International Conference on Computational Linguistics. 2025, pp. 2886–2903, Association for Computational Linguis- tics

    2025

  19. [19]

    Attention-driven gui grounding: Leveraging pretrained multi- modal large language models without fine-tuning,

    Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu, “Attention-driven gui grounding: Leveraging pretrained multi- modal large language models without fine-tuning,” inProceed- ings of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, pp. 8851–8859

    2025

  20. [20]

    Improved gui grounding via iterative nar- rowing,

    Anthony Nguyen, “Improved gui grounding via iterative nar- rowing,”arXiv preprint arXiv:2411.13591, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Reguide: Data efficient gui grounding via spatial reasoning and search,

    Hyunseok Lee, Jeonghoon Kim, Beomjun Kim, Jihoon Tack, Chansong Jo, Jaehong Lee, Cheonbok Park, Sookyo In, Jin- woo Shin, and Kang Min Yoo, “Reguide: Data efficient gui grounding via spatial reasoning and search,”arXiv preprint arXiv:2505.15259, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Grounded language- image pre-training,

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al., “Grounded language- image pre-training,” inProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 2022, pp. 10965–10975

    2022

  23. [23]

    Ui-e2i- synth: Advancing gui grounding with large-scale instruction synthesis,

    Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu, “Ui-e2i- synth: Advancing gui grounding with large-scale instruction synthesis,” inFindings of the Association for Computational Linguistics: ACL 2025. July 2025, pp. 15668–15684, Associa- tion for Computational Linguistics

    2025

  24. [24]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities,

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pa- supat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    The claude 3 model family: Opus, Sonnet, Haiku,

    Anthropic, “The claude 3 model family: Opus, Sonnet, Haiku,” March 2024

    2024

  26. [26]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Gemma 3 technical report,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram ´e, Rivi `ere, et al., “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lu- cas Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11975–11986

    2023