{"id":"dffdc3b9-a45b-4ff9-a64d-25245e1f7309","arxiv_id":"2506.03143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.","lead":"GUI-Actor replaces text-based coordinate prediction in GUI agents with an attention head that grounds actions by attending to visual patches, and adds a verifier to pick the best candidate region. On ScreenSpot-Pro it reports 40.7 with Qwen2-VL-7B, above UI-TARS-72B's 38.1, using a fraction of the parameters and training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uniform KL target over the GT box is never connected to the single-click-point evaluation metric, and the missing ablation leaves the claimed benefit of multi-patch supervision unvalidated.","rationale":"The reader's weakest_assumption identified the mismatch between the uniform KL target and the single-point extraction as the key unanalyzed assumption. I agree this is the right locus, though I would phrase the mechanism differently: rather than attention mass spreading to the perimeter, the more precise risk is that the training objective optimizes a distribution over the box while the evaluation metric rewards a single point, and the paper provides no evidence that this optimization transfers. The missing control—training the same action head with a center-point target—directly tests both the compatibility and the paper's causal claim that multi-patch supervision drives the gains. The other issues the reader raised (no error bars, benchmark-specific thresholds, verifier omitted from LiteTrain parameter count, no released code) are real but secondary: they affect confidence in the numbers, not the internal logic of the method. My concern reinforces the CONDITIONAL verdict rather than changing it, so I set verdict_should_be to UNCHANGED. Agreement with the reader is partial: the core concern is shared, but I doubt the specific perimeter-drift mechanism and prefer a cleaner ablation-based test.","tokens_in":21179,"tokens_out":9414,"duration_ms":109749,"concrete_test":"Retrain GUI-Actor-7B with the identical architecture and data recipe but replace the uniform KL target in Eq. 6 with a single positive patch at the GT box center, and compare ScreenSpot-Pro Element Accuracy. If the center-target variant matches or exceeds the uniform-target variant, the uniform target is not the source of the gains and the compatibility concern is resolved against the paper's mechanism; if it degrades significantly, the uniform target is validated. Also record the distribution of extracted-point distances from the GT box center on large elements to check for boundary drift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"GUI-Actor is trained with a KL loss (Eq. 6) that pushes the <ACTOR> attention to a uniform distribution over all patches intersecting the ground-truth box, while the evaluation metric Element Accuracy scores a single click point extracted from that attention by the post-hoc clustering procedure in Appendix F.2. The paper never analyzes whether this distribution-to-point conversion is reliable, and it never ablates the supervision target: the same attention head trained with a single center-patch or Gaussian target could perform equally well or better, which would mean the reported gains over coordinate baselines come from the architecture rather than the claimed multi-patch supervision. The benchmarks may also not surface failures on elongated elements or boxes with large non-clickable margins, where a uniform target could pull the extracted point away from the functional area. This is a load-bearing gap because the paper explicitly attributes the method's robustness and sample efficiency to the multi-patch supervision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes GUI-Actor, a GUI visual-grounding method that replaces text-based coordinate generation with an attention-based action head. A dedicated <ACTOR> token is inserted into the VLM output sequence, and a lightweight MLP head computes an attention distribution over visual patch tokens; training uses a KL loss that encourages uniform attention over all patches intersecting the ground-truth bounding box. A separately trained grounding verifier clusters and scores top-attention candidates to select a final click point. The authors report results on ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, and a 49-task OSWorld subset, including ablations against Aguvis under matched training data, a lightweight LiteTrain variant, and sample-efficiency curves.","tokens_in":21328,"tokens_out":7582,"duration_ms":79829,"significance":"If the reported results are reproducible, GUI-Actor is a meaningful contribution: it offers a concrete alternative to coordinate generation, produces multiple candidate regions in one forward pass, and decouples grounding from text decoding. The ablations against Aguvis under the same data recipe are a strength, as is the explicit exclusion of Wave-UI overlap with test sets. The central architectural idea is plausible and the gains on ScreenSpot-Pro are sizable. However, the empirical evidence is weakened by the absence of statistical uncertainty, per-benchmark tuning of the verifier, and especially the lack of any ablation isolating the multi-patch supervision target from the attention architecture and the verifier.","major_comments":[{"comment":"The training objective L_Action_Attn pushes the <ACTOR> attention distribution toward a uniform distribution over all patches intersecting the ground-truth box, but the only reported evaluation metric is Element Accuracy of a single click point extracted by the clustering/verifier procedure in Appendix F.2. The manuscript does not analyze this distribution-to-point conversion and does not ablate the supervision target. Since the paper attributes robustness and sample efficiency specifically to multi-patch supervision (Section 5, 'Improved Sample Efficiency'), an ablation with a single-center-patch or Gaussian target under identical architecture and data is needed, together with a failure analysis on elongated elements or boxes with large inactive margins, to substantiate the claim.","section":"Section 3, Eq. (6); Appendix B; Appendix F.2"},{"comment":"All benchmark numbers are reported as point estimates from a single training run, with no standard deviations, confidence intervals, or significance tests. Several margins that support the headline superiority are small (e.g., Table 4, ScreenSpot-v2: GUI-Actor-7B 92.1 vs. Jedi-7B 91.7; Table 3, ScreenSpot-v2: GUI-Actor-2B 88.6 vs. UGround-V1-7B 87.6), and Tables 2-3 show UI-TARS-7B above GUI-Actor-7B on two of the three main benchmarks. The 'consistently outperform' claim is therefore not established to the usual standard; the authors should report multiple seeds and error bars, or at least a formal comparison for the key claims.","section":"Section 5, Tables 1-4"},{"comment":"The verifier confidence threshold gamma is set to 0.95 for ScreenSpot-Pro and 0.8 for ScreenSpot/ScreenSpot-v2, and additional verifier crop sizes are introduced for ScreenSpot-Pro in Appendix G.1. Because the verifier contributes up to 13 points for LiteTrain on ScreenSpot-Pro (Table 5), the per-benchmark choice of gamma is a free parameter that could materially affect the headline generalization result. Please provide a sensitivity analysis over gamma and l_crop, and justify the choices using a validation split rather than the test benchmarks.","section":"Section 5 Implementation Details; Appendix F.2"}],"minor_comments":[{"comment":"The statement that LiteTrain reaches performance 'comparable to previous state-of-the-art models' is broader than Table 5 supports: even with the verifier, LiteTrain-7B scores 81.3 vs. 88.4 on ScreenSpot and 83.8 vs. 90.3 on ScreenSpot-v2 relative to UI-TARS-72B, with only ScreenSpot-Pro close (35.8 vs. 38.1).","section":"Abstract and Section 5"},{"comment":"The sentence 'we introduce a simple mitigation strategy in this work' for small interface elements is never followed by a description of that strategy; please add the description or remove the sentence.","section":"Appendix A"},{"comment":"The column header says '# of Screenshots' but the listed counts sum to exactly the reported 730K examples; clarify whether the column reports screenshots, queries, or generated examples.","section":"Table 8"},{"comment":"The note that the attention grid may need W//2 and H//2 due to the Naive Dynamic Resolution operation in Qwen2-VL should be formalized in the main text or in Appendix F.2, because the mapping from attention-map indices to screen coordinates is essential to the claimed coordinate-free grounding.","section":"Appendix C and Section 3"},{"comment":"No weight is specified for L_Action_Attn relative to L_NTP; please state whether the two losses were balanced and how the action-head warm-up interacts with the subsequent full fine-tuning stage.","section":"Section 3, Eq. (5)"},{"comment":"The x-axis is labeled 'Percentage of Training Data (%)' with values 0-100, while the text says GUI-Actor 'reaches its final accuracy ... using only ~60% of the training data'; clarify how the percentage is computed and how many optimization steps each curve covers.","section":"Figure 3"},{"comment":"There is a typo in 'align natually' that should read 'align naturally'.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's contribution is sound in principle, but the advertised mechanism (multi-patch supervision) is not isolated from the attention architecture and the verifier, and the headline empirical claims rest on single runs with small margins plus per-benchmark verifier thresholds. The missing center/Gaussian ablation and the absence of error bars are the key obstacles. I do not see a circularity problem beyond the acknowledged Wave-UI exclusion, but the gamma choices should be documented as validation-set choices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core take: this is a well-executed empirical paper with a genuinely new architecture—storing the action as an attention distribution over visual patches instead of generated coordinate text. The controlled ablation against Aguvis under the same data and recipe is the strongest evidence, and the ScreenSpot-Pro numbers (40.7 for a 7B model, beating UI-TARS-72B's 38.1) are impressive. It deserves a serious referee.\n\nWhat's actually new: the <ACTOR> token with an attention head trained by KL divergence toward a multi-patch uniform target. That is distinct from training-free attention grounding in Xu et al. [46] and from coordinate-generation baselines. The verifier is a known trick, but the multi-candidate-in-one-forward-pass framing is neat and makes the verifier cheap.\n\nNow the soft spots, in order of how much they matter. First, the KL target is uniform over the whole ground-truth box, but the evaluation and the extracted click point are single points. The paper never ablates the supervision target—center patch, Gaussian, or uniform—so the claimed benefit of multi-patch supervision is not isolated. The stress-test note hits this exactly. I'd call it a missing mechanism analysis rather than a fatal flaw: the full architecture still beats coordinate generation, and the controlled ablation supports that claim. But a referee should ask for that ablation. Second, no error bars or significance tests anywhere, and the verifier thresholds and crop sizes are set per benchmark without a visible validation split. Those choices could overstate the verifier gains. Third, the LiteTrain framing understates the verifier's role: it's a separate fine-tuned 2B model, not part of the ~100M updated parameters. The tables are transparent, but the headline claim is generous. Fourth, no code or weights yet. That is common but painful in this area, where baselines are expensive to rerun.\n\nOn the citation pattern: the related work is fair and positions against the right baselines. The claim that training data is disjoint from test sets (Wave-UI excluded) is plausible and good practice.\n\nWho this is for: people building visual GUI agents or working on VLM grounding generally. The multi-region prediction idea might be useful beyond GUIs.\n\nRecommendation: send it to peer review. It is a solid contribution that needs moderate revisions—chiefly the supervision-target ablation, validation-split evidence for thresholds, and error bars or significance tests—but the core method is sound and interesting.","headline":"Solid, genuinely novel attention-based grounding method; the missing supervision-target ablation and lack of error bars are the main soft spots, but the core claim holds up.","tokens_in":21945,"tokens_out":3760,"would_cite":true,"duration_ms":44327,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GUI-Actor replaces text-coordinate generation with an attention-based action head that grounds a dedicated <ACTOR> token to visual patches, and this coordinate-free design lets a 7B model outperform a 72B model on ScreenSpot-Pro.","keywords":["GUI visual grounding","coordinate-free grounding","attention-based action head","<ACTOR> token","multi-patch supervision","grounding verifier","vision-language models","GUI agents"],"falsifier":"Train GUI-Actor identically but replace the uniform-over-box KL target with a Gaussian target centered on the element's functional click point; if element accuracy on ScreenSpot-Pro does not drop, the dense uniform target is not what drives the reported gains, and the assumption of compatibility with single-point extraction is falsified.","tokens_in":20928,"feed_emoji":"🖱️","tokens_out":8666,"duration_ms":80141,"temperature":0.7,"pith_summary":"GUI-Actor claims that GUI agents should stop predicting actions as x,y text coordinates and instead localize target elements directly in the visual space of the backbone. The paper introduces an attention-based action head: a dedicated <ACTOR> token, placed in the text response, is trained to attend to the screenshot's visual patch tokens, so the attention map itself is the action proposal. Because all patches inside the ground-truth bounding box are treated as positive targets, the model tolerates the natural ambiguity of where inside an element a click is valid, and it can propose several candidate regions in one forward pass. The paper reports that this coordinate-free design beats coordinate-generation baselines on ScreenSpot-Pro, a high-resolution, out-of-distribution GUI grounding benchmark, with GUI-Actor-7B scoring 40.7 (Qwen2-VL backbone) and 44.6 (Qwen2.5-VL backbone), above UI-TARS-72B at 38.1; it also reports that freezing the backbone and training only the new action head is enough for competitive grounding.","feed_headline":"No coordinates: 7B GUI model out-grounds a 72B rival on ScreenSpot-Pro","feed_subtitle":"Acting on patch attention maps instead of x,y text is more accurate and more sample-efficient across screen sizes.","key_machinery":"The central mechanism is the attention-based action head, built around a dedicated <ACTOR> token that acts as the contextual anchor for the intended action. The head applies a self-attention layer over visual patch features, projects the <ACTOR> hidden state and the contextualized patches into a shared embedding space via two MLPs, and computes an attention distribution over all patches. This distribution is trained with a KL target that is uniform over every patch inside the ground-truth bounding box, turning a single click label into dense spatial supervision. The same attention map yields multiple candidate regions in one forward pass, and a separately trained grounding verifier—a lightweight VLM that judges whether a marked point satisfies the instruction—selects among them at inference.","core_discovery":"The paper's central discovery is that GUI action grounding can be cast as a patch-selection problem instead of a coordinate regression problem, and that the attention distribution of a special token provides a high-quality patch-selection mechanism. GUI-Actor inserts <ACTOR_START><ACTOR><ACTOR_END> into the action string, takes the last-layer hidden state of <ACTOR>, projects it and the contextualized visual patch features into a shared space, and computes attention weights $a_i$. The training loss is next-token prediction plus the KL divergence between this attention distribution and a uniform distribution over all patches covered by the ground-truth bounding box. A lightweight grounding verifier then scores candidate click points derived from the attention map and picks the most plausible one. On ScreenSpot-Pro, the 7B model reaches 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL, surpassing UI-TARS-72B at 38.1, and the paper attributes the gains to explicit spatial-semantic alignment at the backbone's native patch resolution.","pith_inferences":["Extending the paper's logic, the uniform-over-box KL target deserves a direct test against a center-weighted target such as a Gaussian over the bounding box, since the paper does not analyze whether the training distribution matches the single point extracted at inference.","If attention maps encode actionable regions this reliably, they could serve as a cheap source of pseudo-labels for grounding in websites and apps where DOM metadata is noisy or unavailable.","The same <ACTOR> attention mechanism could carry over to referring-expression segmentation or to instructions that name several elements, because the model already learns to distribute probability over all relevant patches."],"forward_implications":["A 7B-parameter GUI-Actor can surpass the much larger UI-TARS-72B on ScreenSpot-Pro, so model scale is not the dominant factor in GUI grounding; how actions are represented in the model matters.","Because grounding happens at the vision backbone's patch resolution, the method transfers to unseen screen sizes and layouts without coordinate recalibration.","Fine-tuning only the action head (about 100M parameters of the 7B model) with the backbone frozen yields competitive grounding, so grounding can be added to a general-purpose VLM without destroying its other capabilities.","The same attention map can nominate several distinct valid action regions in one forward pass, which enables cheap verification-based selection and multi-target operations such as clicking one of several visually equivalent options."],"supporting_citations":[{"why":"It supplies the coordinate-generation baseline and the pyautogui-style training format that GUI-Actor replaces with special tokens.","marker":"[7]"},{"why":"It supplies the main coordinate-based UI-TARS baseline and the lightweight backbone for the grounding verifier.","marker":"[10]"},{"why":"It provides the UGround Web-Hybrid training dataset, the largest data source, and a coordinate-generation baseline.","marker":"[8]"},{"why":"It provides the OS-Atlas bounding-box triplets used to build verifier training data and serves as a baseline.","marker":"[45]"},{"why":"It defines the ScreenSpot-Pro benchmark whose high-resolution, out-of-distribution tasks measure generalization.","marker":"[48]"},{"why":"It supplies the Qwen2-VL backbone and the fixed 28x28 patch grid that determines the action space.","marker":"[38]"},{"why":"It motivates the grounding verifier with the principle that verification is easier than generation.","marker":"[47]"},{"why":"It defines the original ScreenSpot benchmark and the SeeClick baseline for GUI grounding comparison.","marker":"[34]"}],"fun_headline_variants":["Patch attention, not coordinates, powers GUI agent grounding","GUI-Actor: 7B model beats 72B by ignoring coordinates","Coordinate-free GUI grounding: attention over patches wins","Tiny 7B agent outscores 72B rival on GUI grounding","No x,y needed: 7B GUI agent tops 72B with patch focus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training the attention map to spread probability uniformly over every patch inside the ground-truth box is compatible with extracting one precise, functional click point at inference; if attention mass drifts toward the box edges, the weighted center can land outside the actually clickable part of the element.","fun_headline_variants_meta":{"raw":{"variants":["Patch attention, not coordinates, powers GUI agent grounding","GUI-Actor: 7B model beats 72B by ignoring coordinates","Coordinate-free GUI grounding: attention over patches wins","Tiny 7B agent outscores 72B rival on GUI grounding","No x,y needed: 7B GUI agent tops 72B with patch focus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1682,"prompt_tokens":1103,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":719,"tokens_out":579,"duration_ms":6570,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:08:09.852194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train GUI-Actor identically but replace the uniform-over-box KL target with a Gaussian target centered on the element's functional click point; if element accuracy on ScreenSpot-Pro does not drop, the dense uniform target is not what drives the reported gains, and the assumption of compatibility with single-point extraction is falsified.","supporting_citations":[],"review_version":1}