Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A vision-language model fine-tuned on synthetic automotive UI captions can point to interface elements and judge whether expected results pass or fail, and it keeps this skill on desktop, mobile, and web screens.

desk verdict A useful automotive UI benchmark and a plausible fine-tuning recipe, but the in-domain gains are unproven until the train/eval image split is disclosed. read the letter →

arxiv 2505.05895 v3 pith:XNFEBYDP submitted 2025-05-09 cs.CV cs.AI

Benjamin Raphael Ernhofer , Daniil Prokhorov , Jannica Langner , Dominik Bollmann This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelautomotiveUIvisualgroundingevaluationsyntheticdatagenerationlargeactionparameter-efficientfine-tuninginfotainmenttesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a relatively small vision-language model can learn to both locate and judge automotive infotainment screens from synthetic training data alone. The authors fine-tune a 7-billion-parameter open model with low-rank adaptation on a pipeline that generates, for each marked UI element, a reasoning trace plus either a test action or an expected result with a passed/failed verdict. On their new benchmark of 998 automotive screens with 4,208 annotations, the resulting model, ELAM-7B, improves test-action grounding by 16.3 percentage points and expected-result evaluation by 11.3 points over the base model. It also scores 80.8% average accuracy on the cross-domain ScreenSpot benchmark, matching or beating specialized mobile and web UI models despite being trained mostly on car screens. If true, this would let automotive UI validation move beyond fragile OCR and template matching toward a local, cost-efficient model that can be updated as interfaces change.

What carries the argument

The load-bearing machinery is the synthetic data pipeline and the dual-label annotation format it produces. For each UI element, a teacher model sees an image with marked bounding boxes, a technique called Set-of-Mark prompting, and writes a test action or an expected result together with a reasoning trace and a passed/failed conclusion; a smaller model then rephrases the text so the student does not overfit one teacher style. The student is Molmo-7B-D-0924, whose native pointing output, a normalized coordinate in a point tag, provides the grounding mechanism. Low-rank adaptation, or LoRA, is the parameter-efficient fine-tuning method that shifts the model into the automotive domain while preserving its general abilities. Ablations in the paper attribute part of the gain to the reasoning traces and part to rephrasing, with a rank-64 LoRA configuration giving the best results.

What would settle it

Audit a random sample of the 17,708 synthetic annotations against the original images and count mislabeled control states, such as toggle switches marked on when they are off. If the mislabel rate is high, or if ELAM-7B's expected-result accuracy on a new test set of state-based checks like 'seat heating is on' or 'temperature is set to 22 degrees' is no better than the base model's roughly 67%, the claimed evaluation capability is not real.

Watch

Extended reading notes

Core claim

The central claim is that domain shift and missing evaluation capability can be solved together by fine-tuning a general grounding-capable vision-language model on synthetic automotive data that explicitly pairs each localization with an evaluative label. Concretely, ELAM-7B, built from Molmo-7B-D-0924 with LoRA, reaches 87.6% accuracy on pointing to the target of a test action and 78.2% on classifying whether an expected result passes or fails, against 71.3% and 66.9% for the base model on AutomotiveUI-Bench-4K. The same model reaches 80.8% average accuracy on ScreenSpot, 5.6 points above the base and 2.2 points above the 72-billion-parameter variant, roughly matching specialized UI grounding systems. The paper argues this shows the synthetic pipeline transfers across domains and that evaluation capability, not just interaction, can be instilled in a small model trainable on consumer hardware.

Load-bearing premise

The whole approach rests on the synthetic captions generated by teacher models, and then rephrased, being correct and varied enough to teach genuine UI-state evaluation; the paper reports that some training samples mislabel basic element states and that expectations overemphasize presence and visibility, which is exactly where that assumption is weakest.

Editorial extensions

If this is right

  • An automotive UI validation model can run on a single consumer GPU with 24 GB of VRAM, with measured invoke times around 2.4 seconds for a test action and 3.4 seconds for an expected result, making in-house deployment practical.
  • A single model now performs both interaction grounding and pass/fail state evaluation, two capabilities that previous small UI models kept separate, so one inference pass can cover both test-action and expected-result checks.
  • Training primarily on automotive screens does not hurt general UI grounding; on ScreenSpot the fine-tuned model improves over its 7B base and even over a 72B variant, so synthetic domain adaptation appears to transfer to other UI domains.
  • The evaluation capability on expected results is the weaker and more persistent limitation: the paper itself cautions against using the model as the sole authority for UI verification, especially for safety-critical functions, and recommends keeping a human in the loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same pipeline logic should generalize to other narrow UI domains with scarce data, such as medical-device screens, industrial HMIs, or kiosks, where a teacher model can generate reasoning-labeled synthetic annotations and a 7B student can be fine-tuned locally.
  • Editorial inference: the paper's own error statistics suggest a cheap, testable improvement: fix the undersized and mis-centered bounding boxes, which account for roughly a quarter of expected-result errors, and add annotations that target control state rather than mere visibility; this could close much of the 15-point gap to the human expert's 93.2% evaluation accuracy.
  • Editorial inference: a stricter test would separate the two learned skills by evaluating the model on state-based expectations only, such as whether a toggle is on or off or a value is selected; if accuracy falls back to the baseline's roughly 67%, the apparent evaluation gain may come mostly from presence and visibility patterns rather than genuine state reasoning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ELAM-7B, a Molmo-7B model fine-tuned with LoRA on synthetic automotive UI data for visual grounding and pass/fail evaluation of UI states. It releases AutomotiveUI-Bench-4K, a manually curated benchmark of 998 infotainment images with 4,208 annotations, and describes a synthetic data pipeline that uses Gemini 2.0 Flash Thinking as a teacher and GPT-4o-mini for rephrasing. The reported in-domain results are large gains over the Molmo baseline on AutomotiveUI-Bench-4K: +16.3 points for test-action visual grounding (TA_vg), +6.1 points for expected-result grounding (ER_vg), and +11.3 points for expected-result evaluation (ER_evl). On the external ScreenSpot benchmark, the model reaches 80.8% average accuracy, a +5.6% improvement over Molmo-7B-D-0924. The paper also includes ablations over reasoning, rephrasing, and LoRA rank, a t-SNE-based failure analysis, and a detailed manual error taxonomy.

Significance. The contribution is potentially valuable: the released dataset fills a clear gap in automotive UI grounding and evaluation, the training pipeline is cost-efficient and open-sourced, and the ScreenSpot result provides an external generalization check that speaks positively of the approach. The paper is unusually candid about its limitations, explicitly acknowledging training-data misclassifications and an overemphasis on presence/visibility in expected results. If the in-domain numbers survive a clean image-disjoint split, the work would be a solid engineering contribution with a useful benchmark. As it stands, the headline in-domain claims rest on an unstated train/eval image split and on hyperparameters selected on the evaluation set, both of which are load-bearing for the paper's central thesis.

major comments (4)
  1. [§5.1, §5.2, Appendix A.1] The paper never states that the 998 AutomotiveUI-Bench-4K evaluation images are disjoint from the 6,230 images used to generate the synthetic training data. Section 3.2 describes generating captions from bounding boxes 'from the dataset,' and Section 5.1 reports a synthetic set of 6,230 images without excluding the benchmark images. If any of the 998 evaluation images (or crops/augmentations) overlap the training set, the reported +16.3 TA_vg, +6.1 ER_vg, and +11.3 ERevl gains in Table 2 can be inflated by memorization of icon positions, layouts, and passed/failed labels. The ScreenSpot result mitigates the cross-domain generalization concern but does not validate the in-domain evaluation-capability claim, which is the paper's main contribution. The authors should state the exact image-level split and report metrics on a held-out subset that is provably disjoint from synthetic training images.
  2. [§5.1, Table 5] The LoRA rank was selected on the evaluation benchmark itself: Section 5.1 states 'An optimal LoRa rank r = 64 was selected based on an ablation study (Table 5)' with Table 5 reporting performance on AutomotiveUI-Bench-4K. This selection-on-test-set introduces optimism bias that is not accounted for in the reported numbers. A validation split (or nested selection) is needed for the final in-domain metrics.
  3. [§5.6.2, §6] The paper's own error analysis and limitation paragraph report that the training data overemphasizes presence/visibility of elements and contains misclassified UI element states (e.g., toggle switches). Since ERevl is the paper's claimed evaluation capability, this acknowledged data bias means the 78.2% evaluation accuracy on AutomotiveUI-Bench-4K may not measure state-based evaluation as the paper intends. The authors should quantify how many expected-result ground truths in AutomotiveUI-Bench-4K require state evaluation rather than simple presence/visibility, and report ERevl separately for these two categories.
  4. [§5.2, Tables 2, 4, 5] All reported percentages are single-run results with no standard deviations, seed counts, or confidence intervals. Given the small benchmark (4,208 annotations) and the hyperparameter selection issue in Table 5, differences of 1–5 points in the ablation tables may not be statistically meaningful. The authors should report variance across multiple training seeds or at least bootstrap confidence intervals for the main metrics.
minor comments (5)
  1. [Throughout] The acronym is spelled inconsistently as 'LoRa' and 'LoRA' across Sections 3.3, 5.1, and 5.4; please unify the spelling.
  2. [Table 2 caption] The word 'superscipts' should be 'superscripts'.
  3. [Figure 20 caption] There is a duplicated definite article in 'The the medium sensitivity icon'; please correct the typo.
  4. [Abstract and Table 3] The abstract says ELAM 'closely matching or surpassing specialized models for desktop, mobile, and web,' but in Table 3, UGround-V1-7B (86.3) and OS-Atlas-Base-7B (81.0) outperform ELAM on average; the wording should be qualified to avoid overstating the comparison.
  5. [Table 4] The 'Baseline' row in Table 4 is not explicitly defined; please state whether it corresponds to no reasoning and no rephrasing, and how it differs from the 'Rephrasing' and 'Reasoning' rows.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims rest on an external benchmark and on synthetic training data generated by separate teacher models, not on the quantities being predicted.

full rationale

The paper's derivation chain is self-contained against external evidence. The in-domain improvements on AutomotiveUI-Bench-4K are measured against a newly released, human-annotated benchmark, while the training captions are synthesized by Gemini (gemini-2.0-flash-thinking-exp-01-21) and rephrased by GPT-4o-mini, i.e., by teacher models separate from the evaluated Molmo-7B backbone. The cross-domain claim is validated on ScreenSpot, an external benchmark with independently published baselines, so that claim is falsifiable outside the paper's own fitted values and training distribution. No self-citation is load-bearing; the cited UGround inspiration and Molmo/PixMo foundation are external prior work, and no uniqueness theorem or ansatz is imported from the authors' own prior publications. The only notable concern is that the paper does not explicitly state an image-level split between the synthetic training images and the 998 AutomotiveUI-Bench-4K evaluation images; if the same images were used for both training and evaluation, the in-domain gains could be inflated. However, that is a data-hygiene and soundness issue, not a demonstrated circular reduction: the paper never defines a predicted quantity in terms of its target, and no equation or fitted parameter is shown to reproduce the benchmark result by construction. Under the hard rule that circularity requires quoting a specific reduction, this concern does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claims are empirical and rest on the quality of the synthetic supervision, the comparability of external baselines, the validity of the point-in-box metric, and the unstated assumption that training and evaluation images are disjoint. No new physical or mathematical entities are introduced.

free parameters (4)
  • LoRA rank r = 64
    Selected via ablation on the evaluation set (Table 5). Lower ranks (16, 32) reduce TA_vg/ER_vg; rank 128 does not improve.
  • Learning rate = 1e-4
    Fixed by hand for all runs; no sweep is reported, but it controls the fine-tuning update scale.
  • Training epochs = 2
    Fixed for all runs; no early stopping or epoch ablation reported.
  • LoRA dropout = 0.05
    Fixed for all LoRA experiments; the paper does not vary it.
assumptions (6)
  • domain assumption Base Molmo-7B-D-0924 provides a robust pointing and language foundation that LoRA fine-tuning can adapt without destroying existing capabilities.
    The method assumes the pretrained model's capabilities transfer to automotive UI; ScreenSpot results are used as evidence but no forgetting analysis is provided.
  • domain assumption Synthetic captions generated by Gemini-2.0-flash-thinking and rephrased by gpt-4o-mini are sufficiently accurate to serve as training supervision for grounding and evaluation.
    The entire pipeline depends on teacher-model output quality; the paper reports training data errors such as misclassified toggle states.
  • domain assumption Point-in-box containment is a valid proxy for correct UI grounding.
    Grounding accuracy is measured by whether the predicted point or box centroid lies inside the annotated box; Sections 5.6 and A.5 note that many boxes are drawn too small, making the metric noisy.
  • domain assumption External ScreenSpot numbers for other models in Table 3 are comparable to ELAM's evaluation.
    Baseline accuracies are quoted from UGround [11] rather than re-run; prompt and preprocessing differences can affect comparability.
  • domain assumption The training and evaluation image sets are disjoint.
    Required for the AutomotiveUI-Bench-4K results to be valid; the paper does not explicitly state this split.
  • domain assumption The AutomotiveUI-Bench-4K images represent modern automotive infotainment systems.
    The dataset covers touchscreens from 15 brands in German and English; generalization to other brands, languages, or non-touchscreen systems is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI." pith.science (2026). https://pith.science/paper/XNFEBYDP

@misc{pith2026250505895,
  author       = {Pith},
  title        = {Pith review of: Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNFEBYDP}},
  note         = {Machine review of arXiv:2505.05895}
}
read the original abstract

Modern automotive infotainment systems necessitate intelligent and adaptive solutions to manage frequent User Interface (UI) updates and diverse design variations. This work introduces a vision-language framework to facilitate the understanding of and interaction with automotive UIs, enabling seamless adaptation across different UI designs. To support research in this field, AutomotiveUI-Bench-4K, an open-source dataset comprising 998 images with 4,208 annotations, is also released. Additionally, a data pipeline for generating training data is presented. A Molmo-7B-based model is fine-tuned using Low-Rank Adaptation (LoRa), incorporating generated reasoning along with visual grounding and evaluation capabilities. The fine-tuned Evaluative Large Action Model (ELAM) achieves strong performance on AutomotiveUI-Bench-4K (model and dataset are available on Hugging Face). The approach demonstrates strong cross-domain generalization, including a +5.6% improvement on ScreenSpot over the baseline model. An average accuracy of 80.8% is achieved on ScreenSpot, closely matching or surpassing specialized models for desktop, mobile, and web, despite being trained primarily on the automotive domain. This research investigates how data collection and subsequent fine-tuning can lead to AI-driven advancements in automotive UI understanding and interaction. The applied method is cost-efficient, and fine-tuned models can be deployed on consumer-grade GPUs.

Figures

Figures reproduced from arXiv: 2505.05895 by the authors.

Figure 1
Figure 1. Synthetic data generation pipeline result (Appendix A.3.3). After manually reviewing a small set of samples for GPT-4o, Gemini 2.0 Flash Thinking and Claude 3.5 Sonnet, Gemini (gemini-2.0-flash-thinking-exp-01-21) was selected to gener￾ate the training data. It was chosen for its better adherence to the required output structure, which was necessary for automatic parsing. To mitigate potential overfitting on the spe… view at source ↗
Figure 2
Figure 2. Examples of synthetic data for training, illustrating (a) a test action [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Example for a Test Action (red) “set A/C to max” and for an Expected Result (green) “Passen￾ger’s climate zone is synced to driver” (Passed) 5 Experiments This section details the experimental procedure for fine-tuning and evaluating ELAM. The following subsections cover the specific configuration used for training, the performance of the fine-tuned model against several baselines on the AutomotiveUI-Bench-4K and Sc… view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Confusion matrices for the Expected Results evaluation classification in AutomotiveUI-Bench￾4K, with values normalized to percentages. 5.3 Evaluating Generalizability with ScreenSpot To determine if the fine-tuning resulted in overfitting to the automotive UI domain, E…
Figure 5
Figure 5. Figure 5: t-SNE plot of the base Molmo-7B model for the [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: t-SNE plot of the ELAM-7B model for the Test Action task 5.5.2 Expected Result Utterances: Evaluation and Grounding Performance Baseline Molmo-7B-D-0924: The Expected Result utterances, representing a more complex task in￾volving visual grounding and subsequent evaluat…
Figure 7
Figure 7. Figure 7: t-SNE visualizations of the base Molmo-7B model for two [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: t-SNE visualizations of the ELAM-7B model for two [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Box too small: “select kWh/100mi as electric consumption unit” [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Ambiguous or poor description of the test action: [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Special icons from the vehicle domain: “turn steering wheel heating on” [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Clear description, but very similar elements confused the model: [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Connection between elements could not be established: [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Driver/passenger, left/right, up/down distinctions: [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Elements must be counted: “Activate the first weekly item from the charging list” [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Miscellaneous: “Go to main menu” [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: Box too small: “Audio quality is set to low” - expected result: failed, ELAM’s prediction: failed [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]
Figure 18
Figure 18. Figure 18: Multiple areas display the expected result: [PITH_FULL_IMAGE:figures/full_fig_p034_18.png]
Figure 19
Figure 19. Figure 19: Box not around option, when asked if options are shown: [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]
Figure 20
Figure 20. Figure 20: Special icons from the vehicle domain: “The medium sensitivity of the distance control was selected” - expected result: passed, ELAM’s prediction: failed [PITH_FULL_IMAGE:figures/full_fig_p036_20.png]
Figure 21
Figure 21. Figure 21: Clear description, but very similar elements confused the model: [PITH_FULL_IMAGE:figures/full_fig_p037_21.png]
Figure 22
Figure 22. Figure 22: Miscellaneous: “E-Call settings are shown” - expected result: failed, ELAM’s prediction: failed [PITH_FULL_IMAGE:figures/full_fig_p038_22.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    A LAYRAC , J.-B., D ONAHUE , J., L UC, P., M IECH , A., B ARR , I., H ASSON , Y., L ENC , K., MENSCH , A., M ILLICAN , K., R EYNOLDS , M., ET AL . Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    work page 2022

  2. [2]

    Amex: Android multi-annotation expo dataset for mobile gui agents

    C HAI , Y., H UANG , S., N IU, Y., X IAO, H., L IU, L., Z HANG , D., G AO, P., R EN, S., AND LI, H. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.17490 (2024)

    arXiv 2024

  3. [3]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    C HEN , Z., W ANG , W., C AO, Y., L IU, Y., G AO, Z., C UI, E., Z HU, J., Y E, S., T IAN , H., L IU, Z., G U, L., W ANG , X., L I, Q., R EN, Y., C HEN , Z., L UO, J., W ANG , J., J IANG , T., WANG , B., HE, C., S HI, B., Z HANG , X., L V, H., W ANG , Y., S HAO, W., C HU, P., T U, Z., H E, T., W U, Z., D ENG , H., G E, J., C HEN , K., Z HANG , K., W ANG , ...

    work page 2025

  4. [4]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    C HENG , K., S UN, Q., C HU, Y., X U, F., L I, Y., Z HANG , J., AND WU, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935 (2024)

    arXiv 2024

  5. [5]

    S., S ALEHI , M., M UEN - NIGHOFF , N., L O, K., S OLDAINI , L., ET AL

    D EITKE , M., C LARK , C., L EE, S., T RIPATHI , R., Y ANG , Y., PARK , J. S., S ALEHI , M., M UEN - NIGHOFF , N., L O, K., S OLDAINI , L., ET AL . Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146 (2024)

    arXiv 2024

  6. [6]

    Website screenshots dataset

    D WYER , B. Website screenshots dataset. https://universe.roboflow.com/ roboflow-gw7yv/website-screenshots, aug 2022. visited on 2024-07-02. 18

    work page 2022

  7. [7]

    Understanding mobile gui: From pixel-words to screen-sentences

    F U, J., Z HANG , X., W ANG , Y., Z ENG , W., AND ZHENG , N. Understanding mobile gui: From pixel-words to screen-sentences. Neurocomputing 601 (2024), 128200

    work page 2024

  8. [8]

    Mobileviews: A large-scale mobile gui dataset

    G AO, L., Z HANG , L., W ANG , S., W ANG , S., L I, Y., AND XU, M. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337 (2024)

    arXiv 2024

Show all 57 references
  1. [9]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions

    G HOSH , A., A CHARYA , A., S AHA , S., J AIN , V., AND CHADHA , A. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214 (2024)

    2024

  2. [10]

    Navi- gating the digital world as humans do: Universal visual grounding for GUI agents

    G OU, B., W ANG , R., Z HENG , B., X IE, Y., C HANG , C., S HU, Y., S UN, H., AND SU, Y. Navi- gating the digital world as humans do: Universal visual grounding for GUI agents

  3. [11]

    UGround, Feb

    G OU, B., W ANG , R., Z HENG , B., X IE, Y., C HANG , C., S HU, Y., S UN, H., AND SU, Y. UGround, Feb. 2025

    2025

  4. [12]

    J., S HEN , Y., WALLIS , P., A LLEN -ZHU, Z., L I, Y., WANG , S., W ANG , L., C HEN , W., ET AL

    H U, E. J., S HEN , Y., WALLIS , P., A LLEN -ZHU, Z., L I, Y., WANG , S., W ANG , L., C HEN , W., ET AL . Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  5. [13]

    Ultralytics YOLO, Jan

    J OCHER , G., Q IU, J., AND CHAURASIA , A. Ultralytics YOLO, Jan. 2023

    2023

  6. [14]

    E., AND KHAN , F

    K HAN , M. E., AND KHAN , F. A comparative study of white box, black box and grey box testing techniques. International Journal of Advanced Computer Science and Applications 3, 6 (2012)

    2012

  7. [15]

    Hardware in the loop for automotive vehicle control systems develop- ment

    K ING , P., AND COPP, D. Hardware in the loop for automotive vehicle control systems develop- ment. In UKACC Control 2004 Mini Symposia (2004), IET, pp. 75–78

    2004

  8. [16]

    J., S AXENA , M., AND KUMAR , A

    K UMAR , R., A HUJA , N. J., S AXENA , M., AND KUMAR , A. Automotive power window commu- nication with dtc algorithm and hardware-in-the loop testing. Wireless Personal Communications 114, 4 (2020), 3351–3366

    2020

  9. [17]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    L I, J., L I, D., S AVARESE , S., AND HOI, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning (2023), PMLR, pp. 19730–19742

    2023

  10. [18]

    E., L I, A., R AWLES , C., C AMPBELL -AJALA , F., T YAMAGUNDLU , D., AND RIVA, O

    L I, W., B ISHOP , W. E., L I, A., R AWLES , C., C AMPBELL -AJALA , F., T YAMAGUNDLU , D., AND RIVA, O. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems 37 (2024), 92130–92154

    2024

  11. [19]

    Benchmark evaluations, applications, and challenges of large vision language models: A survey

    L I, Z., W U, X., D U, H., N GHIEM , H., AND SHI, G. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189 (2025). Submitted on 4 Jan 2025, last revised 29 Jan 2025

    2025 arXiv

  12. [20]

    Q., L I, L., G AO, D., Y ANG , Z., W U, S., B AI, Z., L EI, W., W ANG , L., AND SHOU , M

    L IN, K. Q., L I, L., G AO, D., Y ANG , Z., W U, S., B AI, Z., L EI, W., W ANG , L., AND SHOU , M. Z. Showui: One vision-language-action model for gui visual agent, 2024

    2024

  13. [21]

    L IU, H., L I, C., W U, Q., AND LEE, Y. J. Visual instruction tuning, 2023

    2023

  14. [22]

    Omniparser for pure vision based gui agent, 2024

    L U, Y., YANG , J., S HEN , Y., AND AWADALLAH , A. Omniparser for pure vision based gui agent, 2024

    2024

  15. [23]

    Automotive User Interfaces: Creating Interactive Experiences in the Car

    M EIXNER , G., AND MUELLER , C. Automotive User Interfaces: Creating Interactive Experiences in the Car. Springer, 2017

    2017

  16. [24]

    Functional gui testing of in-vehicle infotainment systems in virtual and real environments

    M EZGER , S., AND DENG , M. Functional gui testing of in-vehicle infotainment systems in virtual and real environments. Hanser Automotive, 5-6 (2017). Translation of a German publication in Hanser Automotive, issue 5-6/2017. 19

    2017

  17. [25]

    Gui agents: A survey

    N GUYEN , D., C HEN , J., W ANG , Y., W U, G., P ARK , N., H U, Z., L YU, H., W U, J., A PONTE , R., X IA, Y., ET AL . Gui agents: A survey. arXiv preprint arXiv:2412.13501 (2024)

    2024

  18. [26]

    X., D UDERSTADT , B., AND MULYAR, A

    N USSBAUM , Z., M ORRIS , J. X., D UDERSTADT , B., AND MULYAR, A. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613 (2024)

    2024 arXiv

  19. [27]

    Tinyclick: Single-turn agent for empowering gui automation

    P AWLOWSKI , P., ZAWISTOWSKI , K., L APACZ , W., SKORUPA , M., W IACEK , A., P OSTANSQUE , S., AND HOSCILOWICZ , J. Tinyclick: Single-turn agent for empowering gui automation. arXiv preprint arXiv:2410.11871 (2024)

    2024 arXiv

  20. [28]

    W., H ALLACY , C., R AMESH , A., G OH, G., A GARWAL , S., S ASTRY, G., A SKELL , A., M ISHKIN , P., C LARK , J., ET AL

    R ADFORD , A., K IM, J. W., H ALLACY , C., R AMESH , A., G OH, G., A GARWAL , S., S ASTRY, G., A SKELL , A., M ISHKIN , P., C LARK , J., ET AL . Learning transferable visual models from natural language supervision. In International conference on machine learning (2021), PmLR,...

    2021

  21. [29]

    Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems 36 (2023), 59708–59728

    R AWLES , C., L I, A., R ODRIGUEZ , D., R IVA, O., AND LILLICRAP , T. Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems 36 (2023), 59708–59728

    2023

  22. [30]

    An overview of the tesseract ocr engine

    S MITH , R. An overview of the tesseract ocr engine. InNinth international conference on document analysis and recognition (ICDAR 2007) (2007), vol. 2, IEEE, pp. 629–633

    2007

  23. [31]

    Visualizing data using t-sne

    VAN DER MAATEN , L., AND HINTON , G. Visualizing data using t-sne. Journal of Machine Learning Research 9, 86 (2008), 2579–2605

    2008

  24. [32]

    Canoe — hil & sil tools across multiple industries — simulation, 2025

    V ECTOR . Canoe — hil & sil tools across multiple industries — simulation, 2025. Accessed: 2025-03-10

    2025

  25. [33]

    Large action models: From inception to implementation

    W ANG , L., Y ANG , F., Z HANG , C., L U, J., Q IAN , J., H E, S., Z HAO, P., Q IAO, B., H UANG , R., Q IN, S., ET AL . Large action models: From inception to implementation. arXiv preprint arXiv:2412.10047 (2024)

    2024 arXiv

  26. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    W ANG , P., B AI, S., T AN, S., W ANG , S., F AN, Z., B AI, J., C HEN , K., L IU, X., W ANG , J., GE, W., ET AL . Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  27. [35]

    Gui agents with foundation models: A comprehensive survey

    W ANG , S., L IU, W., C HEN , J., Z HOU , Y., G AN, W., Z ENG , X., C HE, Y., Y U, S., H AO, X., SHAO, K., ET AL . Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890 (2024)

    2024 arXiv

  28. [36]

    Webui: A dataset for enhancing visual ui understanding with web semantics

    W U, J., W ANG , S., S HEN , S., P ENG , Y.-H., N ICHOLS , J., AND BIGHAM , J. Webui: A dataset for enhancing visual ui understanding with web semantics. ACM Conference on Human Factors in Computing Systems (CHI) (2023)

    2023

  29. [37]

    P., ET AL

    W U, Z., W U, Z., X U, F., W ANG , Y., S UN, Q., J IA, C., C HENG , K., D ING , Z., C HEN , L., LIANG , P. P., ET AL . Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218 (2024)

    2024 arXiv

  30. [38]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    X IAO, B., W U, H., X U, W., D AI, X., H U, H., L U, Y., Z ENG , M., L IU, C., AND YUAN, L. Florence-2: Advancing a unified representation for a variety of vision tasks. arXiv preprint arXiv:2311.06242 (2023)

    2023 arXiv

  31. [39]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Y ANG , J., Z HANG , H., L I, F., Z OU, X., L I, C., AND GAO, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023)

    2023 arXiv

  32. [40]

    Automated testing for automotive infotainment systems

    Y IN, N. Automated testing for automotive infotainment systems. Master’s thesis, Chalmers Uni- versity of Technology, 2018. 20

    2018

  33. [41]

    Large language model-brained gui agents: A survey

    Z HANG , C., H E, S., Q IAN , J., L I, B., L I, L., Q IN, S., K ANG , Y., M A, M., L IU, G., L IN, Q., ET AL . Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279 (2024)

    2024 arXiv

  34. [42]

    Swift:a scalable lightweight infrastructure for fine-tuning, 2024

    Z HAO, Y., H UANG , J., H U, J., W ANG , X., M AO, Y., Z HANG , D., J IANG , Z., W U, Z., A I, B., WANG , A., Z HOU , W., AND CHEN , Y. Swift:a scalable lightweight infrastructure for fine-tuning, 2024

    2024

  35. [43]

    ""\ Identify and point to the UI element that corresponds to this test action: {test_action}

    Z HENG , L., H UANG , Z., X UE, Z., W ANG , X., A N, B., AND YAN, S. Agentstudio: A toolkit for building general virtual agents. arXiv preprint arXiv:2403.17918 (2024). 21 A Appendix A.1 Code and Dataset Availability For reproducibility and broader utility, our resources are o...

    2024 arXiv

  36. [44]

    select kWh/100mi as electric consumption unit

    Box too small: “select kWh/100mi as electric consumption unit” • Figure 9 shows the electric consumption menu of a BMW iX2 in English, featuring a dark background with light text. This menu allows you to change the electric consumption unit. Currently, the unit is set to kWh/1...

  37. [45]

    Select the burger button

    Ambiguous or poor description of the test action: “Select the burger button” • Figure 10 shows the navigation screen of a Toyota Yaris in German, featuring a dark back- ground with blue buttons and text. A small popup with the text “Guten Morgen... ”(good morning) is visible i...

  38. [46]

    turn steering wheel heating on

    Special icons from the vehicle domain: “turn steering wheel heating on” • Figure 11 shows the navigation screen of a Ford Mustang Mach-E with a white background. The area of interest in this test is the lower quarter of the screen featuring buttons related to climate control. ...

  39. [47]

    deactivate the reminder signal for mobile phone

    Clear description, but very similar elements confused the model: “deactivate the reminder signal for mobile phone” • Figure 12 shows the settings menu of an Audi e-tron GT, featuring a black background with white text. • The test action is to deactivate the reminder signal for...

  40. [48]

    add phone to Favourites

    Connection between elements could not be established: “add phone to Favourites” • Figure 13 shows the apps menu of a Maserati Grecale with a black background and white icons and text. Eight apps are currently visible. Each app features a star icon to add it to favourites. • Th...

  41. [49]

    increase the right temperature setting with the plus button

    Driver/passenger, left/right, up/down distinctions: “increase the right temperature setting with the plus button” • Figure 14 shows the climate menu of an Opel Astra in German. The background color is a very dark red and text and icons are white. Some icons/texts are highlight...

  42. [50]

    Activate the first weekly item from the charging list

    Elements must be counted: “Activate the first weekly item from the charging list” • Figure 15 shows the charging menu of a Mini Cooper in German. The round screen is unusual for a car’s infotainment system. The menu allows the user to set several timers for charging. • The tas...

  43. [51]

    Go to main menu

    Miscellaneous: “Go to main menu” • Figure 16 shows the camera settings menu of a Kia. This infotainment system has a light background with dark text and bluish highlights. • ELAM is asked to go to the main menu. • The red box, situated around the house icon that is visible at ...

  44. [52]

    Audio quality is set to low

    Box too small: “Audio quality is set to low” • Figure 17 shows the audio settings menu of a Cupra Leon in German, featuring a dark background with light text. Selected texts are highlighted in orange. • ELAM is asked to check if audio quality is set to low. The expected answer...

  45. [53]

    Sound settings are displayed

    Multiple areas display the expected result: “Sound settings are displayed” • Figure 18 shows the sound settings menu of a BMW iX2 in English, featuring a dark back- ground with white icons and text. Selected items are highlighted in light yellow. • The utterance to verify is: ...

  46. [54]

    Driver assistance options are dis- played

    Box not around option, when asked if options are shown: “Driver assistance options are dis- played” • Figure 19 shows the driver assistance menu of a Kia with a white background. The forward safety settings are currently selected. • ELAM needs to check if the driver assistance...

  47. [55]

    The medium sensitivity of the distance control was se- lected

    Special icons from the vehicle domain: “The medium sensitivity of the distance control was se- lected” • Figure 20 shows the distance control menu of a BMW iX2 in German with a black back- ground and light text. • The expected results utterance is: The medium sensitivity of th...

  48. [56]

    Noise reduction is disabled

    Clear description, but very similar elements confused the model: “Noise reduction is disabled” • Figure 21 shows the audio settings menu of a Kia in German, featuring a light background with gray text. • The utterance to verify is: Noise cancellation is disabled. The expected ...

  49. [57]

    E-Call settings are shown

    Miscellaneous: “E-Call settings are shown” • Figure 22 shows the Wi-Fi menu of a VW ID.4. This infotainment system has a black background with white text. • ELAM is asked to check if emergency call (E-Call) settings are currently displayed. ELAM answers correctly with failed. ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.