Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models ground GUI clicks 13 points better by zooming into their first guess and training with an IoU-aware loss.

desk verdict Solid, practical GUI-grounding work: the IoU-aware loss is well supported, but the zoom-in inference gains need a second-pass control before they fully convince. read the letter →

arxiv 2507.05673 v1 pith:UQQL5LOI submitted 2025-07-08 cs.CV

Joonhyung Park , Peng Tang , Sagnik Das , Srikar Appalaraju , Kunwar Yashraj Singh , R. Manmatha , Shabnam Ghadar This is my paper
classification cs.CV
keywords GUIgroundingvision-languagemodeltwo-stagezoom-inIoU-awarelossregionproposalagentScreenSpotMind2Web
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models that control computer interfaces struggle to point at the exact GUI element a user instruction names: they look at a full cluttered screenshot and are trained to maximize token-level cross-entropy, which does not reward boxes that overlap the target. The paper's central claim is that both problems can be fixed by borrowing two ideas from object detection. First, let the model's own first guess define a region proposal, crop and enlarge that region, and let the model predict again inside the zoomed view. Second, train with an IoU-aware weighted cross-entropy loss that gives higher weight to candidate boxes that overlap the ground truth more. Applying this recipe to an existing strong GUI-grounding model raises grounding accuracy by 13 percentage points absolute on ScreenSpot and AgentStudio, and improves GUI navigation success on AITW and Mind2Web by 3.2-9.7 points. The paper also shows that the zoom-in step alone improves even a much larger pretrained model with no retraining.

What carries the argument

The load-bearing machinery is the two-stage zoom-in grounding loop combined with the IoU-aware weighted cross-entropy loss. In the first stage the model outputs a box from the full screenshot; that box is enlarged by a factor k (typically 5-7x), clamped to the image, cropped, and resized back to the original resolution, and the same VLM is prompted to predict a finer box inside that zoomed view; coordinates are then inverted back to the original image. The second piece is a training objective: M pseudo boxes are sampled near the ground truth with GIoU above a threshold, each is weighted by w = 1 + 0.5 log(GIoU(b_pseudo, b_gt)) when its coordinate tokens contribute to cross-entropy, and all boxes are decoded in a single forward pass because the attention mask blocks pseudo boxes from attending to each other and the RoPE embeddings of the ground-truth box are reused for all pseudo boxes. A third, supporting piece is zoom-in instruction-tuning data: pseudo region proposals are generated by perturbing ground-truth boxes to plausible first-stage errors, paired with prompts asking for a refined box inside the zoomed view; ablation shows the gain grows from +8.3 to +12.9 when this tuning is added on top of inference-time zoom-in.

What would settle it

On a held-out GUI grounding set outside ScreenSpot and AgentStudio, compute how often the first-stage predicted box contains the ground-truth element. The paper's account predicts that most residual failures will have the target inside the proposal (61-76% on ScreenSpot) and that accuracy will track first-stage recall; a dataset where that inclusion rate is much lower should show the 13-point advantage shrink toward zero. If the advantage persists despite low inclusion, then the zoom-in is doing something other than refinement of a correct proposal.

Watch

Extended reading notes

Core claim

The discovery the paper sets out to establish is that VLM-based GUI grounding is leaving accuracy on the table because it ignores the geometry of the target, and that the same two mechanisms that made object detection accurate—region-based refinement and IoU-aware training—transfer to coordinate prediction in language space. Concretely, R-VLM takes the model's initial bounding-box prediction as a region proposal, crops a region around it scaled to the predicted object size, resizes that crop back to full resolution, and runs the same VLM again to get a refined box, which is mapped back to original coordinates. For training, it generates pseudo bounding boxes by perturbing the ground truth with a GIoU threshold, weights each pseudo box's cross-entropy contribution by a log function of its GIoU with the ground truth, and packs all boxes into one forward pass by masking attention between pseudo boxes and reusing the ground-truth box's rotary positional embeddings. On the SeeClick baseline with identical architecture and pretraining data, this yields a 12.9-13.0 point absolute accuracy gain on ScreenSpot and GroundUI-1K, and 3.2-9.7 point gains on AITW and Mind2Web navigation; the two-stage zoom-in also improves a stronger 10M-pretrained model, UGround, without any fine-tuning.

Load-bearing premise

The method only works when the model's first guess already contains the target element; if the initial box misses it, zooming in cannot recover it, so the reported gains depend on first-stage recall being high.

Editorial extensions

If this is right

  • Grounding accuracy for vision-only GUI agents can be raised by about 13 points on two grounding benchmarks without new data collection, new model architectures, or changing the VLM backbone.
  • Because the zoom-in step is applied at inference only, it transfers to other pretrained VLMs: applying it to UGround improves ScreenSpot average accuracy by about 3.5 points and planner-guided Multimodal-Mind2Web element accuracy by 3.2 points, all without fine-tuning.
  • Training with the IoU-aware loss shifts the distribution of predicted boxes toward high IoU with the target, which is the metric used to judge grounding, so accuracy gains are not merely fitting specific coordinate tokens.
  • Improvements in grounding translate to downstream task performance: step success on AITW and Mind2Web rises by 3.2-9.7 points absolute, making precise coordinate prediction a practical lever for GUI agent success.
  • Adding more zoom-in stages gives diminishing returns (1.1 points from 2 to 4 stages) at roughly 2x latency per extra stage, so the two-stage design sits at the practical trade-off point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the paper's failure analysis shows 61-76% of residual errors already have the ground truth inside the first-stage proposal, a testable extension is to add a second or third candidate proposal at stage one with a confidence ranking, which should recover a substantial share of those failures directly.
  • The loss weighting is written in terms of GIoU but the mechanism is generic: any similarity between the pseudo label and the target (the paper itself adapts it to Euclidean distance for point prediction) could replace GIoU, suggesting the recipe transfers to other coordinate-output tasks such as referring expression comprehension or document layout understanding.
  • The cost-efficient packing of pseudo boxes (attention masking plus RoPE reuse) makes many pseudo labels nearly as cheap as one, so scaling to more pseudo boxes—the paper shows accuracy rising from 2 to 8 boxes—might push grounding accuracy beyond the tested regime.
  • Because the two-stage pipeline is capped by first-stage recall, a natural line of work is to make the first stage explicitly recall-oriented (e.g., generate multiple proposals and select by confidence), which the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes R-VLM, a modification of the SeeClick GUI-grounding VLM that combines two components: a two-stage zoom-in grounding procedure, in which the model first predicts a bounding box, crops and enlarges the region around it, and then re-predicts in the zoomed view; and an IoU-aware weighted cross-entropy loss, in which pseudo bounding boxes are generated around the ground truth and used as soft labels with weights based on their GIoU with the ground truth. The authors report substantial improvements over SeeClick on ScreenSpot and GroundUI-1K (AgentStudio), as well as gains on the AITW and Mind2Web navigation benchmarks, and they show that the two-stage zoom-in procedure can be applied in a training-free manner to another VLM, UGround. They also include ablations, a failure analysis, pseudocode, and a discussion of limitations.

Significance. If the results hold, the paper would provide a simple and broadly applicable recipe: using the model's own first prediction as a region proposal and adding an IoU-shaped training signal appear to yield large grounding gains on standard benchmarks while keeping the base architecture and optimizer fixed. The paper has real strengths: the component-wise ablation in Table 7 isolates the IoU-aware loss under otherwise similar conditions; Appendix D provides concrete pseudocode; and the Limitations section plus Appendix A.5 give a transparent analysis of first-stage-recall failure modes. The central weakness is that the causal role of the region-proposal mechanism is not cleanly established: the zoom-in experiments change both the input image and the number of forward passes, so part of the reported gains could be a generic 'second look' effect rather than a consequence of zooming into the predicted region. The paper also overstates its state-of-the-art claim on ScreenSpot, since the trained R-VLM average (66.3) is below the UGround baseline (73.3) shown in Table 5.

major comments (4)
  1. [Abstract / Tables 1 and 5] The central claim that two-stage zoom-in region proposals are the active ingredient is not fully identified because every comparison involving zoom-in inference simultaneously adds a second forward pass and changes the input image from the full screenshot to a crop. Specifically, the training-free SeeClick gain (+8.3 in Table 9), the incremental zoom-in-inference gain in Table 7 (+2.4), and the UGround gains in Table 5 (+1.8 on ScreenSpot, +3.2 on Multimodal-Mind2Web) all compare one pass on the original image with two passes, the second on a zoomed crop centered on the first prediction. There is no control that spends the same additional compute on the unzoomed screenshot, such as re-querying the model with a 'verify your prediction' or paraphrase-refinement prompt, and no control that zooms into a fixed or random region of the same size. Please add these compute-matched controls; without them, the reported improvements could be partly a 'look again' effect that does not depend on the region proposal being accurate.
  2. [Section 4.1 / Table 1 text] The abstract's claim of 'improving the state-of-the-art grounding accuracy by 13%' is overstated. Table 1 shows R-VLM at 66.3 average on ScreenSpot, but Table 5 shows the strong UGround baseline at 73.3 average on the same benchmark. The 13% figure is an improvement over the previous SOTA model SeeClick, not a new state-of-the-art result on ScreenSpot. Please rephrase the claim (e.g., 'improving over the previous SOTA SeeClick') or include the comparison against the strongest available baseline at the time of submission. As written, the abstract invites a misreading that R-VLM itself sets a new SOTA on these benchmarks.
  3. [Appendix A.5 / Table 12] The sentence 'despite identical pretraining data and model architecture, with the only difference being the inclusion of zoomed-in data' is internally inconsistent. R-VLM is trained on the SeeClick pretraining data plus 600K additional zoomed-in samples (Section B.2) and with the IoU-aware weighted cross-entropy loss, while SeeClick uses neither. Thus the headline comparison changes training data, training loss, and inference procedure simultaneously. The ablation in Table 7 partially addresses this, but the text should state explicitly which factors are held fixed in each row; in particular, row 2 isolates the IoU-aware loss on the same base data, while row 3 adds both zoom-in instruction-tuning data and a different label distribution. Please revise the wording so the controlled experiments are described accurately.
  4. [Appendix A.5 / Table 12] The failure analysis shows that in 61-76% of failure cases (depending on element type) the ground-truth region is already inside the initial region proposal, meaning the two-stage process fails even though the target was in view at the second stage. This indicates that first-stage recall is not the only bottleneck: the refinement stage itself fails to correct the prediction in the majority of failure cases. The paper's Limitations section emphasizes first-stage recall, but the data in Table 12 imply that a substantial share of the remaining errors would persist even with a perfect first-stage proposal. Please add an analysis of why the second stage fails in these cases (e.g., crop scale, prompt sensitivity, model capacity) and quantify the best-case improvement available from a perfect second-stage refiner. This would sharpen the claimed limitation and help the reader understand where future gains can come from.
minor comments (5)
  1. [Section 3.3 / Eq. (1)] The text says the low-IoU prediction patterns were 'observed in Section 3.3', but the relevant analysis is in Section 3.1; please fix the cross-reference.
  2. [Tables 1-4 and 7] Equation (1) is called 'IoU-aware', but the weight is defined as 1 + (1/2) log(GIoU(...)). Throughout the paper the terms IoU and GIoU are used interchangeably. Please consistently use 'GIoU-aware' or define clearly when the loss uses GIoU rather than IoU, and note that the log is natural or specify the base.
  3. [Section D / Code 1] All reported numbers are single runs without error bars or multiple seeds. Given the small benchmark sizes (ScreenSpot has ~1.2k instructions), several of the smaller differences reported in the paper, especially the 1-3 point gains in Table 5 and the latency numbers in Table 6, would be more convincing with variance estimates or a statement about training instability. Please report at least one or two repeated runs for the main tables, or an explicit note that only one seed was used.
  4. [Appendix B.2] The zoom-in pseudocode hard-codes the scale factor as 5, but the paper says k is chosen from {5, 7} and the ablation in Table 10 uses High/Mid/Low zoom levels. Please make the code consistent with the experimental description, or add a comment that k is a hyperparameter.
  5. [Appendix B.2] The zoom-in instruction-tuning data uses GIoU greater than -0.2 with the ground truth, while the pseudo-box generation for the IoU-aware loss uses a threshold of 0.3. This distinction is not explained in the main text; a sentence clarifying why the two thresholds differ would help the reader reproduce the data recipe.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: R-VLM's reported gains are measured on external benchmarks and do not reduce to fitted inputs by construction.

full rationale

The paper's central claims are empirical and are evaluated on held-out, external benchmarks (ScreenSpot, GroundUI-1K from AgentStudio, AITW, and Mind2Web), using metrics such as click accuracy, element accuracy, action matching score, and step success rate. These metrics are not the quantities being fitted. The two proposed components are (i) a two-stage zoom-in inference procedure in which the model's own initial bounding box is used as the crop center, and (ii) a GIoU-weighted cross-entropy training loss (Eq. 1). Neither component is defined in terms of the reported evaluation quantities: the loss weights use synthetic pseudo-boxes generated from ground-truth boxes during training, and at inference the final coordinate is obtained by inverting the zoomed-in prediction back to the original image, not by reusing the initial box. The improvements are therefore measured, not derived by definition. The only self-citations in the paper (Gao et al. 2024; Patel et al. 2023; Zhu et al. 2025) appear in related-work enumerations and are not load-bearing for the method or the evaluation. The paper also honestly identifies a real limitation: the two-stage approach is upper-bounded by the recall of the first-stage proposal, and Appendix A.5 quantifies that in 61-76% of failure cases the ground truth is already inside the initial proposal. That stated limitation further confirms that the reported success is not forced by construction. A possible experimental attribution issue is that the training-free zoom-in comparison changes both the input crop and the number of forward passes simultaneously, so some of the gain could be a 'second look' effect rather than region-aware cropping specifically; however, this is a control/interpretation concern, not circularity in the derivation chain.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The pseudo boxes are augmented training labels generated from ground truths, not independent entities. The listed free parameters are the hand-chosen values that the method's performance depends on; all are empirically motivated and ablated, but they are not fitted in the sense of a data-fitting exercise.

free parameters (5)
  • Zoom-in scale factor k = 5 or 7 for main experiments; 3x/5x/7x in ablation
    Controls how much the initial proposal is enlarged before the second-stage grounding. Chosen from a small set and ablated in Appendix A.3.
  • Number of pseudo boxes M = 4
    Number of pseudo ground-truth boxes used in the IoU-aware weighted cross-entropy loss. Ablated from 2 to 8 in Appendix A.2.
  • GIoU threshold for pseudo boxes = 0.3
    Minimum GIoU between a pseudo box and the ground truth (Code 2), determining how far pseudo boxes can deviate from the label.
  • GIoU threshold for zoom-in instruction data = -0.2
    Used in Appendix B.2 when generating zoomed-in training samples; controls the noise level of simulated first-stage proposals.
  • IoU weight log coefficient = 1/2
    Multiplicative factor in the formula w = 1 + 0.5 * log(GIoU), chosen by design to penalize low-IoU pseudo boxes more strongly.
assumptions (4)
  • domain assumption Zoomed-in views of region proposals improve localization accuracy for VLMs on GUI screenshots.
    Core motivation of Section 3.2, inherited from object detection (Girshick et al.), validated only empirically on the tested benchmarks.
  • domain assumption IoU-aware weighted cross-entropy provides a better training signal for coordinate token prediction than plain cross-entropy.
    Assumed in Section 3.3 and supported by the ablation and IoU histogram shift, but not derived from first principles.
  • domain assumption The SeeClick GUI grounding pretraining data is representative and sufficient for training a GUI grounding model.
    Used as the sole pretraining source (Section 4.1); the method inherits any biases or coverage gaps of that dataset.
  • domain assumption Coordinate outputs are normalized bounding-box coordinates in token space and can be treated as continuous box predictions.
    Required for IoU computation and zoom-in cropping (Code 1); a standard convention in GUI grounding VLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding." pith.science (2026). https://pith.science/paper/UQQL5LOI

@misc{pith2026250705673,
  author       = {Pith},
  title        = {Pith review of: R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQQL5LOI}},
  note         = {Machine review of arXiv:2507.05673}
}
read the original abstract

Visual agent models for automating human activities on Graphical User Interfaces (GUIs) have emerged as a promising research direction, driven by advances in large Vision Language Models (VLMs). A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms. Existing vision-only GUI agents directly ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. In addition, these approaches typically employ basic cross-entropy loss for learning grounding objectives, which fails to effectively capture grounding quality compared to established object detection metrics like Intersection-over-Union (IoU). To address these issues, we introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization. We also propose an IoU-aware objective function that facilitates model convergence toward high IoU predictions. Our approach bridges the gap between VLMs and conventional object detection techniques, improving the state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. In addition, our R-VLM approach shows 3.2-9.7% absolute accuracy improvements in GUI navigation tasks on the AITW and Mind2Web benchmarks.

Figures

Figures reproduced from arXiv: 2507.05673 by the authors.

Figure 3
Figure 3. Grounding accuracy of GUI elements on the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Efficient IoU-aware weighted cross-entropy [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Qualitative examples of GUI grounding. Pre [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Comparison of IoU histograms for GUI el [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Results of the ablation study on IoU-aware [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Examples of a set of prompts for instruction tuning on zoomed-in data. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Qualitative examples of GUI grounding. Predictions from our R-VLM are shown in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Qualitative examples of GUI grounding. Predictions from our R-VLM are shown in [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BAMI: Training-Free Bias Mitigation in GUI Grounding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    BAMI mitigates precision and ambiguity biases in GUI grounding via coarse-to-fine focus and candidate selection, raising accuracy on ScreenSpot-Pro without training.

  2. DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    DRS-GUI introduces a dynamic region search method with Focus/Shift/Scatter actions and MCTS-based planning that improves GUI grounding accuracy by 14% on ScreenSpot-Pro for both general and GUI-specific MLLMs without ...

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3

    arXiv 2023

  3. [3]

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa g nak Ta s rlar. 2023. https://www.adept.ai/blog/fuyu-8b Introducing our multimodal models

    2023

  4. [4]

    Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. 2022. A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision, pages 312--328. Springer

    work page 2022

  5. [5]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. https://aclanthology.org/2024.acl-long.505 S ee C lick: Harnessing GUI grounding for advanced visual GUI agents . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313--9332, Bangkok, ...

    work page 2024

  6. [6]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  7. [7]

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. In The Twelfth International Conference on Learning Representations

    work page 2023

  8. [8]

    Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. 2023. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312.13108

    arXiv 2023

Show all 38 references
  1. [9]

    Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Zhuowen Tu, Vijay Mahadevan, and Stefano Soatto. 2024. Enhancing vision-language pre-training with rich supervisions. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2024

  2. [10]

    R Girshick. 2015. Fast r-cnn. arXiv preprint arXiv:1504.08083

    2015 arXiv

  3. [11]

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580--587

    2014

  4. [12]

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for gui agents. In International Conference on Learning Representations

    2025

  5. [13]

    Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations

    2023

  6. [14]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  7. [15]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14...

    2024

  8. [16]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2022

  9. [17]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. 2024. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553

    2024 arXiv

  10. [18]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  11. [19]

    Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016. Ssd: Single shot multibox detector. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part...

    2016

  12. [20]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  13. [21]

    Yash Patel, Yusheng Xie, Yi Zhu, Srikar Appalaraju, and R Manmatha. 2023. Simcon loss with multiple views for text supervised semantic segmentation. arXiv preprint arXiv:2302.03432

    2023 arXiv

  14. [22]

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2024. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137--1149

    2016

  16. [24]

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2019

  17. [25]

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. 2023. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Proces...

    2023

  18. [26]

    Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. 2024. Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration. arXiv preprint arXiv:2411.16044

    2024 arXiv

  19. [27]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  20. [28]

    Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. 2013. Selective search for object recognition. International journal of computer vision, 104:154--171

    2013

  21. [29]

    Penghao Wu and Saining Xie. 2024. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084--13094

    2024

  22. [30]

    Zhuosheng Zhang and Aston Zhang. 2023. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436

    2023 arXiv

  23. [31]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024 a . Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614

    2024 arXiv

  24. [32]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024 b . https://openreview.net/forum?id=piecKJ2DlB GPT -4v(ision) is a generalist web agent, if grounded . In Forty-first International Conference on Machine Learning

    2024

  25. [33]

    Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. 2024 c . Agentstudio: A toolkit for building general virtual agents. arXiv preprint arXiv:2403.17918

    2024 arXiv

  26. [34]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2024. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations

    2024

  27. [35]

    Xiyue Zhu, Peng Tang, Haofu Liao, and Srikar Appalaraju. 2025. Turbocharging web automation: The impact of compressed history states. In Findings of ACL

    2025

  28. [36]

    C Lawrence Zitnick and Piotr Doll \'a r. 2014. Edge boxes: Locating object proposals from edges. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 391--405. Springer

    2014

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.