{"id":"b7b8ade2-16e5-4340-92fc-a068c6c2ae1d","arxiv_id":"2607.19907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned YOLO pipeline on a new 520-screenshot, 38,000-widget dataset detects desktop GUI widgets from pixels and drives system-wide Bubble Cursor and Semantic Pointing.","lead":"A machine-vision system that finds buttons, menus, sliders, and other interface widgets just by looking at a screen, on Windows, macOS, Linux, and the web. It aims to let pointing aids such as Bubble Cursor and Semantic Pointing work across every desktop application, without needing app developers' cooperation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Small-widget recall (0.303) and lack of interaction evaluation leave the target-aware feasibility claim unproven.","rationale":"The reader's weakest assumption already identified the completeness and timeliness of detections as the key risk for target-aware pointing. My review confirms and sharpens this: Table 7 provides concrete evidence that the smallest widgets—precisely those the bubble cursor is designed to help with—are detected with recall 0.303. The paper does not report any evaluation of selection accuracy, neither via user study nor via simulation, so the gap between detection metrics and interaction outcomes is unmeasured. This is a genuine, load-bearing concern because the central claim in the abstract and introduction is that TargetFinder makes universal target-aware techniques feasible. The detection and dataset contributions are solid and should not be rejected; they are novel and well-evaluated in the object-detection sense. However, the interaction feasibility claim is conditional on demonstrating that the detected widget set suffices for correct target selection. Since the reader already assigned CONDITIONAL, my verdict is unchanged. No new objection beyond the reader's weakest assumption arises, but I provide a concrete computational test that would quantify the impact of small-widget misses on selection accuracy, which would either substantiate or alleviate the concern.","tokens_in":24533,"tokens_out":4580,"duration_ms":51182,"concrete_test":"Run a Bubble Cursor selection simulation on the held-out Test set (52 images) using the released YOLO26n-640 detections. For each ground-truth widget, place the cursor at several random offsets within the widget's neighbourhood, compute the nearest ground-truth widget and the nearest detected widget, and record whether they match. Report selection accuracy separately for widgets below the 10th percentile of the size distribution. If accuracy for these small widgets is substantially lower than for larger widgets (e.g., below 80%), the interaction premise is undermined and the conditional verdict should be retained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TargetFinder enables system-wide target-aware pointing rests on the completeness and accuracy of the widget set supplied to techniques like Bubble Cursor and Semantic Pointing. The paper's own Table 7 (§9.6) reports that for the smallest decile of widgets (by YOLO-pixel size), YOLO26n-640 achieves recall 0.303 and F1 0.421—about 70% of these targets are missed. Bubble Cursor selects the nearest target to the cursor; if a small widget is absent from the detection set, the bubble will snap to a different, likely larger, widget, causing a selection error exactly in the regime where target-aware techniques are most beneficial. Semantic Pointing similarly relies on accurate target extents to modulate the C/D ratio; missing or mis-sized small targets distorts the motor-space mapping. The paper provides only video demonstrations of the two techniques, with no user study or even a simulation-based accuracy evaluation. The ~200 ms end-to-end latency (§7) adds a related risk immediately after interface changes, during which detections are stale. Consequently, the abstract's 'feasibility of deploying universal target-aware techniques' is supported by a running prototype but not by evidence that the detection quality is sufficient for correct interaction. The concern is not that the detection pipeline is flawed—it clearly outperforms baselines—but that the load-bearing link from detection quality to usable pointing is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TargetFinder is a computer-vision system for real-time widget detection on desktop GUIs. The authors contribute a new dataset of 520 annotated screenshots (37,919 annotations) spanning Windows, macOS, Ubuntu, and web interfaces; multiple fine-tuned YOLO models; and an open-source Python library. On a held-out test set, their best configuration (YOLO26n-640) achieves precision 0.936, recall 0.840, and F1 0.885, outperforming all tested baselines (OmniParser, REMAUI, UIED, MobileSAM, and Martínez-Rojas et al.). Additional experiments examine generalization to legacy software, mobile interfaces, and cross-OS transfer, and the paper describes system-wide implementations of Bubble Cursor and Semantic Pointing as demonstrations.","tokens_in":24850,"tokens_out":8014,"duration_ms":76982,"significance":"If the reported detection accuracy is reproducible, this is a valuable contribution: it provides a cross-platform, accessibility-API-independent source of widget geometry with the potential to enable target-aware interaction techniques and other pixel-based GUI understanding tasks. The detection evaluation is methodologically careful: a stratified held-out test set, inter-annotator agreement of F1=0.933, comparisons against several baselines, explicit generalization tests, and 5-fold cross-validation for hyperparameter robustness. The planned public release of the dataset, models, annotation tool, and library is a community asset. The main weakness is that the paper's central interaction-level claim—that these detections make system-wide target-aware pointing feasible—is supported only by video demonstrations, with no user study or simulation quantifying the impact of detection errors and latency on pointing behavior.","major_comments":[{"comment":"The claim that 'system-wide implementations of Bubble Cursor and Semantic Pointing demonstrate the feasibility of deploying universal target-aware techniques' is not supported by the evidence presented. The implementations are described architecturally (PyQt overlay, pyautogui click redirection) and illustrated with videos; there is no user study, no simulation of selection accuracy, and no analysis of how detection errors affect the techniques. This is load-bearing because the abstract presents these demonstrations as a central outcome. Compounding the gap, Table 7 (§9.6) reports that for the smallest decile of widgets, YOLO26n-640 has recall 0.303 and F1 0.421; a bubble cursor using this detector would miss roughly 70% of small targets and may snap to unintended neighboring widgets, and semantic pointing's motor-space mapping would be computed on incomplete geometry. To support the fea","section":"§6.3–6.4 and Abstract"},{"comment":"The manuscript does not state which YOLO model is used in the Bubble Cursor and Semantic Pointing demonstrations. §5.3 suggests using the 's×1920' model for the presented use cases, but all small-widget performance results in Table 7 are reported for YOLO26n-640. If the demos use a different model, the 0.303 small-widget recall does not directly apply to the feasibility claim. The authors should specify the deployed model and report its performance on the small-widget subsets, so that the link between detector accuracy and pointing behavior can be assessed.","section":"§5.3 and §6.3–6.4"}],"minor_comments":[{"comment":"The abstract claims 'millisecond responsiveness suitable for interactive use,' but §7 reports an end-to-end latency of about 200 ms. This wording is inconsistent and overstates responsiveness; please reconcile the abstract and introduction with the actual measured latency.","section":"Abstract and §7"},{"comment":"The caption of Figure 1 contains a long string of invalid Unicode escape sequences ('\\uni00000025...'), which appears to be a text corruption. This should be fixed before publication.","section":"Figure 1 caption"},{"comment":"The abstract states 'We release the dataset, models, annotation tool, and an open-source library,' but §3.3 says the dataset 'will be made open-access and publicly hosted after the review period.' This discrepancy should be clarified.","section":"§3.3 and Abstract"},{"comment":"The inter-annotator agreement is reported as a mean F1 of 0.933, but per-image F1 values range as low as 0.772 (e.g., img_003, Ann3). A brief discussion of this range would strengthen confidence in the annotation protocol.","section":"Table 4"},{"comment":"The OmniParser comparison removes the OCR stage, reporting the YOLO11m-1280 detector alone as the 'best-performing configuration for object detection.' A one-sentence justification in the main text (beyond the supplementary) would help readers understand why this is a fair comparison.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The detection evaluation is solid and the dataset release is a valuable community contribution. The paper's main issue is that it overclaims interaction-level feasibility without a user study or even a simulation; this is addressable either by adding such an analysis or by softening the claim to a prototype demonstration. I am confident in the detection results, so I do not recommend rejection, but the claims need to be brought in line with the evidence before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid systems-and-data paper, and the detection evaluation is better than most work in this area. The dataset of 520 multi-OS desktop screenshots with ~38k human annotations is a real gap-filler. The inter-annotator agreement (F1 0.933) and the held-out test protocol give me reasonable confidence in the numbers. TargetFinder's YOLO26n mono-class F1 of 0.885 clearly beats OmniParser's detector (0.698), REMAUI, UIED, MobileSAM, and Martínez-Rojas. That comparison is fair: using OmniParser's YOLO component alone is the right way to isolate detection quality, since the OCR/fusion stages serve a different end-to-end task. The generalization tests to legacy, mobile, and cross-OS data are a plus; the legacy result (F1 0.695) is respectable.\n\nThe soft spot is the target-aware pointing part. The paper implements Bubble Cursor and Semantic Pointing system-wide and shows videos, but there is no user study and no simulation. The stress-test note makes the right point: Table 7 shows recall of 0.303 for the smallest decile of widgets. Bubble Cursor picks the nearest target; if the small widget near the cursor is missing from the detected set, the bubble will select something else. That is exactly the condition where target-aware techniques are supposed to help. So the phrase 'demonstrate the feasibility of deploying universal target-aware techniques' in the abstract is too strong. The demos show the plumbing works; they do not show the detection quality is sufficient for correct interaction. The latency point is related: ~200 ms end-to-end is fine for static interfaces but stale right after a UI change, and the abstract's 'millisecond responsiveness' overstates it. To be fair, the paper's Limitations section is unusually candid about both latency and small-widget difficulties. The mismatch is between that candor and the abstract.\n\nMinor: §3.3 says the dataset will be public after review, while the front matter claims it is public. That should be cleaned up.\n\nVerdict: the core detection contribution is real, and the code/dataset/models are explicitly released. This deserves a serious referee. The authors should be pushed to either add a user study (even a small one) or re-frame the pointing techniques as engineering demonstrations rather than validated feasibility. Anyone working on GUI grounding, accessibility, or pointing facilitation will want to cite the dataset and the library.","headline":"Worth reviewing for the dataset and detection evaluation; the target-aware pointing feasibility claim is a prototype demonstration, not yet evidence it works.","tokens_in":25354,"tokens_out":3204,"would_cite":true,"duration_ms":31764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TargetFinder shows that a computer-vision pipeline can detect desktop widgets in real time purely from pixels, enabling target-aware pointing techniques such as Bubble Cursor and Semantic Pointing to run system-wide without application coop","keywords":["GUI widget detection","computer vision","YOLO","target-aware pointing","Bubble Cursor","Semantic Pointing","desktop dataset","real-time object detection"],"falsifier":"Run a controlled user study comparing TargetFinder-driven Bubble Cursor with the standard cursor on a dense small-widget interface (e.g., a toolbar). If selection time or error rate fails to improve, or a log shows the hovered small widget frequently absent from the detection stream, the practical claim collapses. A cheaper check: instrument a session and count the fraction of frames where the widget under the cursor is not in the detector output.","tokens_in":24424,"feed_emoji":"🖱️","tokens_out":5970,"duration_ms":63449,"temperature":0.7,"pith_summary":"This paper sets out to show that the interactive elements of a desktop interface—buttons, text fields, sliders, and the like—can be identified fast enough and accurately enough from screen pixels alone, without any help from the applications. If that holds, two classic 'target-aware' pointing enhancements, Bubble Cursor (which expands the clickable area around the nearest widget) and Semantic Pointing (which enlarges targets in motor space), can work everywhere on a desktop, not just inside a custom demo or a web browser. The authors support the claim with a new dataset of 520 fully annotated desktop screenshots across Windows, macOS, Ubuntu, and web interfaces, fine-tuned YOLO detection models, and a technical evaluation where their detector outperforms the compared baselines. They also provide working system-wide implementations of both techniques, plus open releases of the dataset, models, annotation tool, and library. The acknowledged remaining gap is detection of small widgets and the roughly 200 ms latency after interface changes.","feed_headline":"Pixel-only widget detection brings smart cursors to any desktop","feed_subtitle":"No app cooperation needed: real-time widget geometry works across Windows, macOS, Ubuntu, and the web.","key_machinery":"The central object is a fine-tuned YOLO object detection network (YOLO26n-640, with variants at other sizes and input resolutions) that takes a full-resolution screenshot and outputs bounding boxes for six widget classes: button, text input, slider, toggle, text, and hyperlink. Around it sits a lightweight pipeline: continuous low-resolution screen capture at 30 Hz, a change-detection step that triggers full inference only when the screen changes, and a separate thread that publishes detection results for downstream use. The dataset—520 manually annotated screenshots (~38,000 instances) with documented bounding-box rules and measured annotator agreement—is what makes the detector work across","core_discovery":"TargetFinder is a system that detects the interactive widgets in a desktop GUI by analyzing screenshots, without touching application internals. The authors built a dataset of 520 fully annotated screenshots (37,919 widget instances) spanning Windows, macOS, Ubuntu, and web interfaces, and fine-tuned YOLO models on it. In their evaluation, the best configuration (YOLO26n-640) reaches a mono-class F1 of 0.885 on the held-out test set, compared with 0.698 for OmniParser and 0.455 for REMAUI, and it maintains reasonable performance on legacy desktop software while transferring poorly to mobile UIs. They also implement system-wide Bubble Cursor and Semantic Pointing over this detector, arguing t","pith_inferences":["If small-widget recall stays near the reported 0.303 for the smallest decile, target-aware techniques will likely falter on dense toolbars; a natural extension is to feed the detector's confidence or uncertainty into the pointing algorithm instead of treating detections as ground truth.","The ~200 ms end-to-end latency after interface changes is not a uniform cursor delay; it suggests system-wide techniques should either precompute geometry for likely next windows or tolerate a brief blind spot after changes.","The desktop-to-mobile transfer failure documented in the paper suggests the reverse direction (mobile-to-desktop) is also unlikely to hold; a testable route is fine-tuning on mixed data, which the paper implies but does not test.","Recursively running the detector inside detected widgets, which the paper floats as a future direction, would add hierarchy (e.g., close-tab within tab) and might also recover small widgets missed at the full-image scale."],"forward_implications":["System-wide target-aware pointing becomes feasible on unmodified third-party applications, since widget geometry no longer depends on accessibility APIs or browser layout engines.","A relatively small, manually curated desktop dataset can outperform a far larger automatically labeled web-only dataset on desktop layouts, indicating that annotation quality and platform diversity matter more than raw scale.","The same pixel-level geometry stream can support in-the-wild studies of pointing (e.g., Fitts' law), accessibility tools that work from pixels, and GUI-automation agents that need exhaustive widget lists.","The open release of dataset, models, annotation tool, and library lets independent groups reproduce the detector and build interaction techniques on top of it."],"fun_headline_variants":["Pixel-only widget spotting brings smart cursors to any desktop","Seeing GUI widgets in screenshots enables cross-app smart pointing","TargetFinder reads pixels to find widgets in real time across OSes","Universal smart cursors via real-time pixel-based widget detection","From pixels to pointers: real-time widget detection on any interface"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the assumption that the detector supplies a nearly complete and accurate set of widgets around the cursor quickly enough to steer target-aware pointing; the paper's own result of 0.303 recall on the smallest decile shows this premise is not yet guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Pixel-only widget spotting brings smart cursors to any desktop","Seeing GUI widgets in screenshots enables cross-app smart pointing","TargetFinder reads pixels to find widgets in real time across OSes","Universal smart cursors via real-time pixel-based widget detection","From pixels to pointers: real-time widget detection on any interface"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1118,"prompt_tokens":701,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":445,"tokens_out":417,"duration_ms":4858,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:19:51.201411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled user study comparing TargetFinder-driven Bubble Cursor with the standard cursor on a dense small-widget interface (e.g., a toolbar). If selection time or error rate fails to improve, or a log shows the hovered small widget frequently absent from the detection stream, the practical claim collapses. A cheaper check: instrument a session and count the fraction of frames where the widget under the cursor is not in the detector output.","supporting_citations":[],"review_version":1}