Pith. sign in

REVIEW 2 major objections 5 minor 73 references

TargetFinder shows that a computer-vision pipeline can detect desktop widgets in real time purely from pixels, enabling target-aware pointing techniques such as Bubble Cursor and Semantic Pointing to run system-wide without application coop

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:19 UTC pith:TVKO5Q4H

load-bearing objection Worth reviewing for the dataset and detection evaluation; the target-aware pointing feasibility claim is a prototype demonstration, not yet evidence it works. the 2 major comments →

arxiv 2607.19907 v1 pith:TVKO5Q4H submitted 2026-07-22 cs.HC

TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces

Ahmed Ben Akouche (ISIR) , G\'ery Casiez (LOOP) , Mathieu Nancel (LOOP) , Julien Gori (ISIR) This is my paper
classification cs.HC
keywords GUI widget detectioncomputer visionYOLOtarget-aware pointingBubble CursorSemantic Pointingdesktop datasetreal-time object detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that the interactive elements of a desktop interface—buttons, text fields, sliders, and the like—can be identified fast enough and accurately enough from screen pixels alone, without any help from the applications. If that holds, two classic 'target-aware' pointing enhancements, Bubble Cursor (which expands the clickable area around the nearest widget) and Semantic Pointing (which enlarges targets in motor space), can work everywhere on a desktop, not just inside a custom demo or a web browser. The authors support the claim with a new dataset of 520 fully annotated desktop screenshots across Windows, macOS, Ubuntu, and web interfaces, fine-tuned YOLO detection models, and a technical evaluation where their detector outperforms the compared baselines. They also provide working system-wide implementations of both techniques, plus open releases of the dataset, models, annotation tool, and library. The acknowledged remaining gap is detection of small widgets and the roughly 200 ms latency after interface changes.

Core claim

TargetFinder is a system that detects the interactive widgets in a desktop GUI by analyzing screenshots, without touching application internals. The authors built a dataset of 520 fully annotated screenshots (37,919 widget instances) spanning Windows, macOS, Ubuntu, and web interfaces, and fine-tuned YOLO models on it. In their evaluation, the best configuration (YOLO26n-640) reaches a mono-class F1 of 0.885 on the held-out test set, compared with 0.698 for OmniParser and 0.455 for REMAUI, and it maintains reasonable performance on legacy desktop software while transferring poorly to mobile UIs. They also implement system-wide Bubble Cursor and Semantic Pointing over this detector, arguing t

What carries the argument

The central object is a fine-tuned YOLO object detection network (YOLO26n-640, with variants at other sizes and input resolutions) that takes a full-resolution screenshot and outputs bounding boxes for six widget classes: button, text input, slider, toggle, text, and hyperlink. Around it sits a lightweight pipeline: continuous low-resolution screen capture at 30 Hz, a change-detection step that triggers full inference only when the screen changes, and a separate thread that publishes detection results for downstream use. The dataset—520 manually annotated screenshots (~38,000 instances) with documented bounding-box rules and measured annotator agreement—is what makes the detector work across

Load-bearing premise

The central claim rests on the assumption that the detector supplies a nearly complete and accurate set of widgets around the cursor quickly enough to steer target-aware pointing; the paper's own result of 0.303 recall on the smallest decile shows this premise is not yet guaranteed.

What would settle it

Run a controlled user study comparing TargetFinder-driven Bubble Cursor with the standard cursor on a dense small-widget interface (e.g., a toolbar). If selection time or error rate fails to improve, or a log shows the hovered small widget frequently absent from the detection stream, the practical claim collapses. A cheaper check: instrument a session and count the fraction of frames where the widget under the cursor is not in the detector output.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • System-wide target-aware pointing becomes feasible on unmodified third-party applications, since widget geometry no longer depends on accessibility APIs or browser layout engines.
  • A relatively small, manually curated desktop dataset can outperform a far larger automatically labeled web-only dataset on desktop layouts, indicating that annotation quality and platform diversity matter more than raw scale.
  • The same pixel-level geometry stream can support in-the-wild studies of pointing (e.g., Fitts' law), accessibility tools that work from pixels, and GUI-automation agents that need exhaustive widget lists.
  • The open release of dataset, models, annotation tool, and library lets independent groups reproduce the detector and build interaction techniques on top of it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If small-widget recall stays near the reported 0.303 for the smallest decile, target-aware techniques will likely falter on dense toolbars; a natural extension is to feed the detector's confidence or uncertainty into the pointing algorithm instead of treating detections as ground truth.
  • The ~200 ms end-to-end latency after interface changes is not a uniform cursor delay; it suggests system-wide techniques should either precompute geometry for likely next windows or tolerate a brief blind spot after changes.
  • The desktop-to-mobile transfer failure documented in the paper suggests the reverse direction (mobile-to-desktop) is also unlikely to hold; a testable route is fine-tuning on mixed data, which the paper implies but does not test.
  • Recursively running the detector inside detected widgets, which the paper floats as a future direction, would add hierarchy (e.g., close-tab within tab) and might also recover small widgets missed at the full-image scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. TargetFinder is a computer-vision system for real-time widget detection on desktop GUIs. The authors contribute a new dataset of 520 annotated screenshots (37,919 annotations) spanning Windows, macOS, Ubuntu, and web interfaces; multiple fine-tuned YOLO models; and an open-source Python library. On a held-out test set, their best configuration (YOLO26n-640) achieves precision 0.936, recall 0.840, and F1 0.885, outperforming all tested baselines (OmniParser, REMAUI, UIED, MobileSAM, and Martínez-Rojas et al.). Additional experiments examine generalization to legacy software, mobile interfaces, and cross-OS transfer, and the paper describes system-wide implementations of Bubble Cursor and Semantic Pointing as demonstrations.

Significance. If the reported detection accuracy is reproducible, this is a valuable contribution: it provides a cross-platform, accessibility-API-independent source of widget geometry with the potential to enable target-aware interaction techniques and other pixel-based GUI understanding tasks. The detection evaluation is methodologically careful: a stratified held-out test set, inter-annotator agreement of F1=0.933, comparisons against several baselines, explicit generalization tests, and 5-fold cross-validation for hyperparameter robustness. The planned public release of the dataset, models, annotation tool, and library is a community asset. The main weakness is that the paper's central interaction-level claim—that these detections make system-wide target-aware pointing feasible—is supported only by video demonstrations, with no user study or simulation quantifying the impact of detection errors and latency on pointing behavior.

major comments (2)
  1. [§6.3–6.4 and Abstract] The claim that 'system-wide implementations of Bubble Cursor and Semantic Pointing demonstrate the feasibility of deploying universal target-aware techniques' is not supported by the evidence presented. The implementations are described architecturally (PyQt overlay, pyautogui click redirection) and illustrated with videos; there is no user study, no simulation of selection accuracy, and no analysis of how detection errors affect the techniques. This is load-bearing because the abstract presents these demonstrations as a central outcome. Compounding the gap, Table 7 (§9.6) reports that for the smallest decile of widgets, YOLO26n-640 has recall 0.303 and F1 0.421; a bubble cursor using this detector would miss roughly 70% of small targets and may snap to unintended neighboring widgets, and semantic pointing's motor-space mapping would be computed on incomplete geometry. To support the fea
  2. [§5.3 and §6.3–6.4] The manuscript does not state which YOLO model is used in the Bubble Cursor and Semantic Pointing demonstrations. §5.3 suggests using the 's×1920' model for the presented use cases, but all small-widget performance results in Table 7 are reported for YOLO26n-640. If the demos use a different model, the 0.303 small-widget recall does not directly apply to the feasibility claim. The authors should specify the deployed model and report its performance on the small-widget subsets, so that the link between detector accuracy and pointing behavior can be assessed.
minor comments (5)
  1. [Abstract and §7] The abstract claims 'millisecond responsiveness suitable for interactive use,' but §7 reports an end-to-end latency of about 200 ms. This wording is inconsistent and overstates responsiveness; please reconcile the abstract and introduction with the actual measured latency.
  2. [Figure 1 caption] The caption of Figure 1 contains a long string of invalid Unicode escape sequences ('\uni00000025...'), which appears to be a text corruption. This should be fixed before publication.
  3. [§3.3 and Abstract] The abstract states 'We release the dataset, models, annotation tool, and an open-source library,' but §3.3 says the dataset 'will be made open-access and publicly hosted after the review period.' This discrepancy should be clarified.
  4. [Table 4] The inter-annotator agreement is reported as a mean F1 of 0.933, but per-image F1 values range as low as 0.772 (e.g., img_003, Ann3). A brief discussion of this range would strengthen confidence in the annotation protocol.
  5. [§4.3] The OmniParser comparison removes the OCR stage, reporting the YOLO11m-1280 detector alone as the 'best-performing configuration for object detection.' A one-sentence justification in the main text (beyond the supplementary) would help readers understand why this is a fair comparison.

Circularity Check

0 steps flagged

No circularity: empirical detection pipeline evaluated on held-out test set; limitations affect validity, not derivation.

full rationale

The paper makes no derivation claim that could be circular. Its central contributions are empirical: a new annotated dataset, fine-tuned YOLO detectors, a held-out evaluation against standard object-detection metrics, comparisons with published baselines, and system-wide demonstrations of Bubble Cursor and Semantic Pointing. The evaluation uses mAP, F1, and IoU computed on a held-out 52-image Test set (§4.1.1), with hyperparameters tuned only on Train0/Val0 and the final model retrained on Train0+Val0 before evaluation on Test; no fitted quantity is renamed as a prediction. The Bubble Cursor and Semantic Pointing implementations follow published algorithms ([26], [11]) and are demonstrations, not predictions derived from TargetFinder by construction. Self-citations (e.g., [25] for the widget-size distribution shape, [14] for pointing transfer functions, [37] for AutoGain) are contextual and non-load-bearing; none is invoked as a uniqueness theorem or to forbid alternatives. The paper honestly reports limitations in §7 and §9.6, including low recall for the smallest widgets (YOLO26n-640 recall 0.303 at the 10th percentile, Table 7) and roughly 200 ms end-to-end latency after interface changes. These are correctness/validity concerns about the strength of the feasibility claim, not circularity: the detection outputs are compared against independent ground truth, and the acknowledged gaps weaken but do not tautologically support any claim. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central contribution is empirical: a trained detector, a dataset, and a systems integration. The main input assumptions are the representativeness of the 520 screenshots, the sufficiency of bounding boxes for pointing, and the fairness of the baseline configurations. No new physical or conceptual entities are postulated.

free parameters (4)
  • YOLO hyperparameters (genetic-algorithm tuned) = not reported (supplementary results only)
    Augmentation settings and classification loss weight were tuned on Train0/Val0 (§4.2); the detection performance and all downstream claims depend on them.
  • Default confidence threshold = value maximizing F1 (not reported)
    Reported precision/recall/F1 values depend on this threshold, which is configurable in the library (§6.1).
  • Change-detection threshold = configurable (default not reported)
    Determines when YOLO re-runs; affects end-to-end latency and CPU usage (§6.1).
  • IoU / NMS threshold = configurable (default not reported)
    Affects duplicate suppression and hence mIoU and mAP scores (§6.1).
axioms (5)
  • domain assumption A bounding box with IoU≥0.5 is a sufficient representation of a GUI widget for target-aware pointing.
    All metrics and the Bubble Cursor/Semantic Pointing integrations consume boxes; §7 acknowledges class and hierarchy limitations but not the sufficiency of boxes.
  • domain assumption The 520 annotated screenshots sourced from OS-ATLAS and OmniACT are representative of real desktop GUIs.
    §3.2–3.3; the set covers Windows/macOS/Ubuntu/web, but the sample is small and web labels were partly inherited from OmniACT.
  • domain assumption Inter-annotator agreement measured on 15 of 520 images validates the annotation protocol.
    §3.2 and §9.4 report mean F1 0.933 and mIoU 0.892, but only a small subset was re-annotated.
  • ad hoc to paper The six-class taxonomy is adequate for the intended applications.
    §3.2 and §7; the authors state widget type is not critical for their target-aware use cases and that class sets vary across prior work.
  • domain assumption A ~200 ms post-change inference delay does not impede real-time interaction.
    §7 argues the delay only affects the brief moment after interface changes; no user study verifies this.

pith-pipeline@v1.3.0-alltime-deepseek · 24245 in / 14296 out tokens · 140428 ms · 2026-08-01T11:19:51.201411+00:00 · methodology

0 comments
read the original abstract

''Target-aware'' pointing techniques, like Bubble Cursor or Semantic Pointing, outperform traditional pointing by leveraging knowledge of target locations. Yet the lack of application-agnostic widget geometry information limits their adoption across the desktop. We present TargetFinder, a computer vision-based system for real-time detection of GUI widgets. TargetFinder leverages several fine-tuned YOLO networks trained on a new dataset of 520 annotated desktop screenshots (~38,000 annotations) spanning Windows, macOS, Ubuntu, and web interfaces. TargetFinder uses lightweight screen monitoring and low-latency detection, achieving millisecond responsiveness suitable for interactive use. Evaluations show that TargetFinder outperforms the baseline methods (OmniParser and REMAUI), while system-wide implementations of Bubble Cursor and Semantic Pointing demonstrate the feasibility of deploying universal target-aware techniques that work across applications. We release the dataset, models, annotation tool, and an open-source library for research and applications.

Figures

Figures reproduced from arXiv: 2607.19907 by Ahmed Ben Akouche (ISIR), G\'ery Casiez (LOOP), Julien Gori (ISIR), Mathieu Nancel (LOOP).

Figure 1
Figure 1. Figure 1: Example of hand-labeled screenshots. Button ToggleButton Hyperlink Text TextInput Slider Class 10 2 10 3 10 4 Number of examples per class (log scale) 11100 204 101 58 53 172 10449 243 94 63 144 97 5241 73 873 468 194 39 6650 682 575 18 248 80 33440 1202 1643 607 639 388 linux mac web windows total [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Class distribution by category. Note the logarithmic scale on [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visual comparison of widget detection outputs from our trained YOLO26n-640 model and OmniParser’s YOLO11m-1280 for three example [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of widget sizes in YOLO-pixels (logarithmic scale), [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison for different YOLO architectures [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: TargetFinder and system-wide implementations of BubbleCursor and SemanticPointing illustrated. 7 Limitations Limited number of widget classes. We considered 6 classes in the current implementation of TargetFinder, but prior work has generally used more, e.g. 27 in [45]. Considering that our intended applications do not actually require knowing the type of widget being targeted, and given that performance c… view at source ↗
Figure 9
Figure 9. Figure 9: Example of annotated TextInput fields. (3) The Slider class refers to sliding controls like volume adjusters or scrollbars; in such cases, we annotate the entire track along which the slider moves, excluding any side buttons, which are labeled as Button [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 7
Figure 7. Figure 7: Screenshot of the semi-automatic annotation tool WidgetAnno [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example of annotated hyperlinks and text blocks. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example of various annotated buttons. 9.3 Evaluation metrics for object detection The evaluation metrics commonly used in the object detection literature are: (1) Intersection over Union (IoU): This is the ratio between the number of pixels in the intersection of the ground truth object and the predicted object (true positive), and the number of pixels in their union. It measures the localization and size… view at source ↗
Figure 13
Figure 13. Figure 13: Intersection over Union. (2) Precision: The number of correct detections (true positives) divided by the total number of detections. It reflects the model’s ability to avoid false positives. Precision = True Positives True Positives + False Positives (3) Recall: The number of correct detections (true positives) divided by the total number of ground truth instances (i.e., the sum of true positives and fals… view at source ↗
Figure 14
Figure 14. Figure 14: Visual comparison of widget detection outputs from other methods and our trained YOLO26 model for three example screenshots: a Firefox [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 6 canonical work pages

  1. [1]

    Microsoft Active Accessibility

    2020. Microsoft Active Accessibility. https://learn.microsoft.com/en-us/windows/ win32/winauto/microsoft-active-accessibility. Accessed: 2025-09-10

    2020

  2. [2]

    WaltteriDesktopUIDataset

    2020. WaltteriDesktopUIDataset. https://github.com/waltteri/desktop-ui-dataset. Accessed: 2025-09-10

    2020

  3. [3]

    Microsoft UI Automation

    2021. Microsoft UI Automation. https://learn.microsoft.com/en-us/windows/ win32/winauto/entry-uiauto-win32. Accessed: 2025-09-10

    2021

  4. [4]

    Mehmet Dogan Altinbas and Tacha Serif. 2022. GUI Element Detection from Mobile UI Images Using YOLOv5. InMobile Web and Intelligent Information Systems: 18th International Conference, MobiWIS 2022, Rome, Italy, August 22–24, 2022, Proceedings(Rome, Italy). Springer-Verlag, Berlin, Heidelberg, 32–45. doi:10.1007/978-3-031-14391-5_3

    work page doi:10.1007/978-3-031-14391-5_3 2022

  5. [5]

    TakeshiAsano,EhudSharlin,YoshifumiKitamura,KazukiTakashima,andFumio Kishino. 2005. Predictive interaction using the delphian desktop. InProceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (Seattle,WA,USA)(UIST’05).AssociationforComputingMachinery,NewYork, NY, USA, 133–141. doi:10.1145/1095034.1095058

    arXiv 2005

  6. [6]

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. 2021. UIBert: Learning Generic Multimodal Representations for UI Understanding. InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Zhi-Hua Zhou (Ed.). International Joint Conferences on Artifici...

    doi:10.24963/ijcai.2021/235 2021

  7. [7]

    Patrick Baudisch, Edward Cutrell, Mary Czerwinski, Daniel C Robbins, Peter Tandler, Benjamin B Bederson, and Alex Zierlinger. 2003. Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content on Touch-and Pen-Operated Systems.. InInteract, Vol. 3. Citeseer, 57–64

    2003

  8. [8]

    Daniel Baulé, Christiane Gresse von Wangenheim, Aldo Von Wangenheim, and Jean Hauck. 2020. Recent Progress in Automated Code Generation from GUI Im- agesUsingMachineLearningTechniques.JUCS-JournalofUniversalComputer Science26, 9 (2020), 1095–1127. arXiv:https://doi.org/10.3897/jucs.2020.058 doi:10.3897/jucs.2020.058

    work page doi:10.3897/jucs.2020.058 2020

  9. [9]

    InProceedings of the ACM SIGCHI Symposium on Engineering Inter- active Computing Systems(Paris, France)(EICS ’18)

    TonyBeltramelli.2018.pix2code:GeneratingCodefromaGraphicalUserInterface Screenshot. InProceedings of the ACM SIGCHI Symposium on Engineering Inter- active Computing Systems(Paris, France)(EICS ’18). Association for Computing Machinery,NewYork,NY,USA,Article3,6pages. doi:10.1145/3220134.3220135

    arXiv 2018

  10. [10]

    Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2022. Accessibility in Software Practice: A Practitioner’s Perspective. ACM Trans. Softw. Eng. Methodol.31, 4, Article 66 (July 2022), 26 pages. doi:10.1145/3503508

    work page doi:10.1145/3503508 2022

  11. [11]

    Renaud Blanch, Yves Guiard, and Michel Beaudouin-Lafon. 2004. Semantic pointing: improving target acquisition with control-display ratio adaptation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria)(CHI ’04). Association for Computing Machinery, New York, NY, USA, 519–526. doi:10.1145/985692.985758

    arXiv 2004

  12. [12]

    Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedingsofthe2021CHIConferenceonHumanFactorsinComputingSystems (Yokohama, Japan)(CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 423, 14 pages. doi:10.1145/3411764.3445762

    arXiv 2021

  13. [13]

    Acomputationalapproachtoedgedetection.IEEETransactions on pattern analysis and machine intelligencePAMI-8 (12 1986), 679–698

    JohnCanny.1986. Acomputationalapproachtoedgedetection.IEEETransactions on pattern analysis and machine intelligencePAMI-8 (12 1986), 679–698. doi:10. 1109/TPAMI.1986.4767851

    arXiv 1986

  14. [14]

    Géry Casiez and Nicolas Roussel. 2011. No more bricolage! methods and tools to characterize, replicate and compare pointing transfer functions. InProceedings of the24thAnnualACMSymposiumonUserInterfaceSoftwareandTechnology(Santa Barbara, California, USA)(UIST ’11). Association for Computing Machinery, New York, NY, USA, 603–614. doi:10.1145/2047196.2047276

    arXiv 2011

  15. [15]

    Olivier Chapuis. 2005. Gestion des fenêtres : enregistrement et visualisation de l’interaction. InIHM ’05: Proceedings of the 17th international conference of the Association Francophone d’Interaction Homme-Machine. Toulouse, France, 255–258. doi:10.1145/1148550.1148590

    arXiv 2005

  16. [16]

    2007.Fitts’ Law in the Wild: A Field Study of Aimed Movements

    Olivier Chapuis, Renaud Blanch, and Michel Beaudouin-Lafon. 2007.Fitts’ Law in the Wild: A Field Study of Aimed Movements. Technical Report. https: //hal.science/hal-00612026 LRITechnicalRepportNumber1480,Univ.Paris-Sud, 11 pages

    2007

  17. [17]

    Olivier Chapuis, Jean-Baptiste Labrune, and Emmanuel Pietriga. 2009. DynaSpot: speed-dependentareacursor.InProceedingsoftheSIGCHIConferenceonHuman Factors in Computing Systems(Boston, MA, USA)(CHI ’09). Association for Computing Machinery, New York, NY, USA, 1391–1400. doi:10.1145/1518701. 1518911

    doi:10.1145/1518701 2009

  18. [18]

    Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object detection for graphical user interface: old fashioned or deep learning or a combination?. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Virtual Event...

    arXiv 2020

  19. [19]

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. InProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology(Québec City, QC, Canada)(UIST ’17). Association for Computing Machin...

    arXiv 2017

  20. [20]

    Prefab:implementingadvancedbehaviors using pixel-based reverse engineering of interface structure

    MorganDixonandJamesFogarty.2010. Prefab:implementingadvancedbehaviors using pixel-based reverse engineering of interface structure. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1525–1534

    2010

  21. [21]

    Morgan Dixon, James Fogarty, and Jacob Wobbrock. 2012. A general-purpose target-awarepointingenhancementusingpixel-levelanalysisofgraphicalinterfaces. InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems. 3167–3176

    2012

  22. [22]

    Abigail Evans and Jacob Wobbrock. 2012. Taming wild behavior: the input observer for obtaining text entry and mouse pointing measures from everyday computer use. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Austin, Texas, USA)(CHI ’12). Association for Computing Machinery, New York, NY, USA, 1947–1956. doi:10.1145/2207676.2208338

    arXiv 2012

  23. [23]

    Wobbrock

    Leah Findlater, Alex Jansen, Kristen Shinohara, Morgan Dixon, Peter Kamb, Joshua Rakita, and Jacob O. Wobbrock. 2010. Enhanced area cursors: reducing fine pointing demands for people with motor impairments. InProceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology(New York, New York, USA)(UIST ’10). Association for Computing...

    arXiv 2010

  24. [24]

    Krzysztof Gajos, Katharina Reinecke, and Charles Herrmann. 2012. Accurate measurements of pointing performance from in situ observations. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Austin, Texas, USA)(CHI ’12). Association for Computing Machinery, New York, NY, USA, 3157–3166. doi:10.1145/2207676.2208733

    arXiv 2012

  25. [25]

    Julien Gori and Olivier Rioul. 2019. Regression to a linear lower bound with outliers: An exponentially modified Gaussian noise model. In2019 27th European Signal Processing Conference (EUSIPCO). IEEE, 1–5

    2019

  26. [26]

    Thebubblecursor:enhancingtarget acquisition by dynamic resizing of the cursor’s activation area

    ToviGrossmanandRavinBalakrishnan.2005. Thebubblecursor:enhancingtarget acquisition by dynamic resizing of the cursor’s activation area. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Portland, Oregon, USA)(CHI ’05). Association for Computing Machinery, New York, NY, USA, 281–290. doi:10.1145/1054972.1055012

    arXiv 2005

  27. [27]

    Yves Guiard, Renaud Blanch, and Michel Beaudouin-Lafon. 2004. Object point- ing: a complement to bitmap pointing in GUIs. InProceedings of Graphics Interface 2004(London, Ontario, Canada)(GI ’04). Canadian Human-Computer Communications Society, Waterloo, CAN, 9–16

    2004

  28. [28]

    Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. The Unreasonable Effectiveness of Data.IEEE Intelligent Systems24, 2 (2009), 8–12. doi:10.1109/ MIS.2009.36

    2009

  29. [29]

    Morten Hertzum and Kasper Hornbæk. 2007. Input techniques that dynamically changetheircursoractivationarea:Acomparisonofbubbleandcellcursors.Int.J. Hum.-Comput.Stud.65,10(Oct.2007),833–851. doi:10.1016/j.ijhcs.2007.05.001

    work page doi:10.1016/j.ijhcs.2007.05.001 2007

  30. [30]

    Errol R Hoffmann. 1992. Fitts’ law with transmission delay.Ergonomics35, 1 (1992), 37–48

    1992

  31. [31]

    Zheng Hui, Yinheng Li, Dan zhao, Tianyi Chen, Colby Banbury, and Kazuhito Koishida. 2025. WinClick: GUI Grounding with Multimodal Large Language Models. arXiv:2503.04730 [cs.CL] https://arxiv.org/abs/2503.04730

    Pith/arXiv arXiv 2025

  32. [32]

    Hudson, and Jennifer Mankoff

    Amy Hurst, Scott E. Hudson, and Jennifer Mankoff. 2007. Dynamic detection of novice vs. skilled use without a task model. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(San Jose, California, USA) (CHI ’07). Association for Computing Machinery, New York, NY, USA, 271–280. doi:10.1145/1240624.1240669 11 Preprint, HAL, 2026 Ahme...

    arXiv 2007

  33. [33]

    InProceedings of the 15th International Conference on Intelligent User Interfaces(Hong Kong, China) (IUI ’10)

    AmyHurst,ScottE.Hudson,andJenniferMankoff.2010.Automaticallyidentifying targets users interact with during real world tasks. InProceedings of the 15th International Conference on Intelligent User Interfaces(Hong Kong, China) (IUI ’10). Association for Computing Machinery, New York, NY, USA, 11–20. doi:10.1145/1719970.1719973

    arXiv 2010

  34. [34]

    Wobbrock

    Alex Jansen, Leah Findlater, and Jacob O. Wobbrock. 2011. From the lab to the world: lessons from extending a pointing technique for real-world use. InCHI ’11 Extended Abstracts on Human Factors in Computing Systems(Vancouver, BC, Canada)(CHI EA ’11). Association for Computing Machinery, New York, NY, USA, 1867–1872. doi:10.1145/1979742.1979888

    arXiv 2011

  35. [35]

    Paul Kabbash and William A. S. Buxton. 1995. The “prince” technique: Fitts’ law and selection using area cursors. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Denver, Colorado, USA)(CHI ’95). ACM Press/Addison-Wesley Publishing Co., USA, 273–279. doi:10.1145/223904. 223939

    doi:10.1145/223904 1995

  36. [36]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2025. OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web. InComputer Vision – ECCV 2024, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds...

    2025

  37. [37]

    Byungjoo Lee, Mathieu Nancel, Sunjun Kim, and Antti Oulasvirta. 2020. Au- toGain: Gain Function Adaptation with Submovement Efficiency Optimization. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3313831.3376244

    arXiv 2020

  38. [38]

    InProceedings of the AAAI conference on artificial intelligence, Vol

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,ZhoujunLi,andFuruWei.2023.Trocr:Transformer-basedopticalcharacter recognition with pre-trained models. InProceedings of the AAAI conference on artificial intelligence, Vol. 37. 13094–13102

    2023

  39. [39]

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117–2125

    2017

  40. [40]

    InComputer Vision – ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.)

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan,PiotrDollár,andC.LawrenceZitnick.2014.MicrosoftCOCO:Common Objects in Context. InComputer Vision – ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 740–755

    2014

  41. [41]

    Wanyu Liu, Rafael Lucas D’Oliveira, Michel Beaudouin-Lafon, and Olivier Rioul

  42. [42]

    OmniParser for Pure Vision Based GUI Agent

    YadongLu,JianweiYang,YelongShen,andAhmedAwadallah.2024. OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203 [cs.CV] https://arxiv.org/ abs/2408.00203

    Pith/arXiv arXiv 2024

  43. [43]

    Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman

  44. [45]

    Martínez-Rojas, A

    A. Martínez-Rojas, A. Rodríguez-Ruíz, J. G. Enríquez, and A. Jiménez-Ramírez

  45. [46]

    Fabio Matti, Pierre Dillenbourg, and Ludovico Novelli. 2023. A Click Ahead: Real-TimeForecastingofKeyboardandMouseActionsusingRNNsandComputer Vision. arXiv:2309.12170 [cs.HC] https://arxiv.org/abs/2309.12170

    Pith/arXiv arXiv 2023

  46. [47]

    Acquisitionofexpandingtargets

    MichaelMcGuffinandRavinBalakrishnan.2002. Acquisitionofexpandingtargets. InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems (Minneapolis,Minnesota,USA)(CHI’02).AssociationforComputingMachinery, New York, NY, USA, 57–64. doi:10.1145/503376.503388

    arXiv 2002

  47. [48]

    Rodriguez, Montek Kalsi, Nicolas Chapados, M

    Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. 2025. UI- Vision:ADesktop-centricGUIBenchmarkforVisualPerceptionandInteraction.In Forty-second International Conference on Machine Learning...

    2025

  48. [49]

    Springer- Verlag, Berlin, Heidelberg, 256–272

    What’s Behind the Screen? Unveiling UI Hierarchies in Process-Related UI Logs.InBusinessProcessManagement:22ndInternationalConference,BPM2024, Krakow, Poland, September 1–6, 2024, Proceedings(Krakow, Poland). Springer- Verlag, Berlin, Heidelberg, 256–272. doi:10.1007/978-3-031-70396-6_15

    work page doi:10.1007/978-3-031-70396-6_15 2024

  49. [50]

    Baker, Yasmine N

    Rohan Patel, Pedro Breton, Catherine M. Baker, Yasmine N. El-Glaly, and Kristen Shinohara. 2020. Why Software is Not Accessible: Technology Professionals’ Perspectives and Challenges. InExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI EA ’20). Association for Computing Machinery, New York, NY, USA,...

    arXiv 2020

  50. [51]

    DP-YOLO:Alightweight traffic sign detection model for small object detection.Digital Signal Processing 165 (2025), 105311

    JiQiu,WenboZhang,SiyuanXu,andHuiyuZhou.2025. DP-YOLO:Alightweight traffic sign detection model for small object detection.Digital Signal Processing 165 (2025), 105311

    2025

  51. [52]

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. 2025. SAM 2: Segment Anything in Images and Videos. InThe Thirteenth Inter...

    2025

  52. [53]

    Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In2015 30th IEEE/ACM In- ternational Conference on Automated Software Engineering (ASE). 248–259. doi:10.1109/ASE.2015.32

    work page doi:10.1109/ase.2015.32 2015

  53. [54]

    SmartUIDetector

    TezanSahu.2020. SmartUIDetector. https://github.com/tezansahu/smart_ui_tf20. Accessed: 2025-09-10

    2020

  54. [55]

    Xiaojun Su, Oscar Kin-Chung Au, and Rynson W.H. Lau. 2014. The implicit fan cursor: a velocity dependent area cursor. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 753–762. doi:10.1145/2556288.2557095

    arXiv 2014

  55. [56]

    Thebitterlesson

    RichardSutton.2019. Thebitterlesson. http://www.incompleteideas.net/IncIdeas/ BitterLesson.html. Accessed: 2025-09-10

    2019

  56. [57]

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/ Redmon_You_Only_Look_CVPR_2016_paper.pdf

    2016

  57. [58]

    Jinwang Wang, Chang Xu, Wen Yang, and Lei Yu. 2021. A normalized Gaussian Wasserstein distance for tiny object detection.arXiv preprint arXiv:2110.13389 (2021)

    Pith/arXiv arXiv 2021

  58. [59]

    Aileen Worden, Nef Walker, Krishna Bharat, and Scott Hudson. 1997. Making computers easier for older adults to use: area cursors and sticky icons. InProceed- ings of the ACM SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA)(CHI ’97). Association for Computing Machinery, New York, NY, USA, 266–271. doi:10.1145/258549.258724

    arXiv 1997

  59. [60]

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao

  60. [61]

    Amanda Swearngin and Yang Li. 2019. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. doi:10.1145/3290605.3300305

    arXiv 2019

  61. [62]

    Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen

  62. [63]

    Shumin Zhai, Stéphane Conversy, Michel Beaudouin-Lafon, and Yves Guiard

  63. [64]

    arXiv:2306.14289 [cs.CV] https://arxiv.org/abs/ 2306.14289

    ChaoningZhang,DongshenHan,YuQiao,JungUkKim,Sung-HoBae,Seungkyu Lee,andChoongSeonHong.2023.FasterSegmentAnything:TowardsLightweight SAM for Mobile Applications. arXiv:2306.14289 [cs.CV] https://arxiv.org/abs/ 2306.14289

    Pith/arXiv arXiv 2023

  64. [65]

    Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems(Yokohama,...

    arXiv 2021

  65. [67]

    Andrew Bagnell

    Brian Ziebart, Anind Dey, and J. Andrew Bagnell. 2012. Probabilistic pointing target prediction via inverse optimal control. InProceedings of the 2012 ACM International Conference on Intelligent User Interfaces(Lisbon, Portugal)(IUI ’12). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10. 1145/2166966.2166968 9 Appendices 9.1 Semi-autom...

    arXiv 2012

  66. [69]

    https://github.com/MulongXie/ UIED

    UIED: User Interface Element Detection. https://github.com/MulongXie/ UIED. Accessed: 2025-09-10

    2025

  67. [74]

    AnIntelligent Labeling andDataset GeneratingMethod for DesktopApplication Element Object Detection

    JundaZhu,JunAi,TaoShi,CongranZhang,andXinyueWu.2023. AnIntelligent Labeling andDataset GeneratingMethod for DesktopApplication Element Object Detection. In2023 10th International Conference on Dependable Systems and Their Applications (DSA). 860–869. doi:10.1109/DSA59317.2023.00121 12 TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces Prepr...

    arXiv 2023

  68. [2003]

    InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Ft

    Human on-line response to target expansion. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Ft. Lauderdale, Florida, USA)(CHI ’03). Association for Computing Machinery, New York, NY, USA, 177–184. doi:10.1145/642611.642644

    arXiv

  69. [2013]

    InProceedings of the 26th Annual ACM Symposium on User Interface Software and Technology(St

    Skillometers: reflective widgets that motivate and help users to improve performance. InProceedings of the 26th Annual ACM Symposium on User Interface Software and Technology(St. Andrews, Scotland, United Kingdom) (UIST’13).AssociationforComputingMachinery,NewYork,NY,USA,321–330. doi:10.1145/2501988.2501996

    arXiv

  70. [2017]

    In Proceedingsofthe2017CHIConferenceonHumanFactorsinComputingSystems (Denver, Colorado, USA)(CHI ’17)

    BIGnav: Bayesian Information Gain for Guiding Multiscale Navigation. In Proceedingsofthe2017CHIConferenceonHumanFactorsinComputingSystems (Denver, Colorado, USA)(CHI ’17). Association for Computing Machinery, New York, NY, USA, 5869–5880. doi:10.1145/3025453.3025524

    arXiv

  71. [2020]

    InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Virtual Event, USA) (ESEC/FSE 2020)

    UIED: a hybrid tool for GUI element detection. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1655–1659. doi:10.1145/3368089.3417940

    arXiv 2020

  72. [2024]

    https://github

    uidetection: Implementation of Screen2SOM (YOLOv8). https://github. com/RPA-US/uidetection/. Accessed: 2025-09-10

    2025

  73. [2025]

    In The Thirteenth International Conference on Learning Representations

    OS-ATLAS: Foundation Action Model for Generalist GUI Agents. In The Thirteenth International Conference on Learning Representations. https: //openreview.net/forum?id=n9PDaFNi8t