{"id":"04293580-7fcc-479a-973d-855830764da9","arxiv_id":"2608.09723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A closed-loop GUI grounder that reflects on its own predicted coordinates via marker-rendered patches and a locate/confirm protocol, trained with SFT then GRPO, achieves state-of-the-art grounding and refusal accuracy.","lead":"LookAgain turns GUI grounding into a multi-turn loop: the model predicts a location, sees a marker on that spot, then confirms or corrects itself. On benchmarks like OSWorld-G and ScreenSpot-Pro it beats the base model by large margins, suggesting that letting a screen agent re-check its own clicks is a practical way to cut errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains attributed to multi-turn reflection are confounded by SFT on Gemini-distilled trajectories; Table 3 compares full LookAgain (SFT+GRPO) against single-turn GRPO without SFT, so the gap may be teacher distillation, not the closed-loop mechanism.","rationale":"The reader identifies the unvalidated CutPaste refusal synthesis as the weakest assumption; that is a real concern for the refusal-aware subset, but it does not threaten the general grounding results (ScreenSpot-Pro, UIVision, MMBench-L2-GUI). A more load-bearing weakness sits at the center of the paper's contribution: the claim that the closed-loop reflection mechanism, rather than RL or the teacher-distilled SFT, drives the gains. Table 3 cannot support this because the single-turn comparison lacks the SFT stage. Given that Table 4 demonstrates large SFT-only improvements, the reported advantage of LookAgain over single-turn GRPO may largely reflect distillation from Gemini3.1-Pro's successful reflective trajectories. This does not invalidate the empirical results, but it means the mechanism is not established. A single-turn SFT+GRPO control would settle whether the multi-turn protocol itself is responsible. The verdict remains conditional: the paper should either add this control or soften the mechanistic attribution. The refusal-data concern, while valid, is secondary because it only affects the refusal subset and is already partially mitigated by evaluation on OSWorld-G/VenusBench-GD refusal splits, though the synthetic-to-real gap deserves its own check.","tokens_in":17862,"tokens_out":8632,"duration_ms":60305,"concrete_test":"Train a control, 'LookAgain-SingleTurn-SFT+GRPO', using the same 70k teacher-distilled trajectories reformatted to single-turn (teacher final coordinate only, no intermediate locate patches), followed by the same GRPO single-turn stage. If this variant reaches ~60 on ScreenSpot-Pro (matching LookAgain-8B), the gain is attributable to SFT distillation rather than multi-turn reflection. As a secondary check, train LookAgain's multi-turn GRPO directly from the base model without SFT; if it fails to reproduce the reported gains, the SFT cold-start is necessary, but the single-turn SFT control is the decisive test.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central mechanistic claim, that the improvement comes from multi-turn reflection rather than RL alone, rests on Table 3, which compares LookAgain-8B (60.2 on ScreenSpot-Pro) with '+GRPO-Single Turn' variants (47.6/55.7). However, LookAgain receives a supervised cold-start on ~70k Gemini3.1-Pro-distilled reflective trajectories, while the single-turn GRPO variants are trained from the base model with RL only. The comparison therefore conflates the multi-turn protocol with the SFT stage. Table 4 shows SFT alone already yields large gains (OSWorld-G-Refine 67.8 to 73.5; UIVision 23.3 to 37.5), so the residual gap in Table 3 could be explained by distillation from a much stronger teacher rather than by the predict-look-again-refine loop. No control with a single-turn SFT cold-start (or a multi-turn GRPO without SFT) is reported, leaving the central attribution underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LookAgain, a closed-loop GUI grounding method that reformulates grounding as a multi-turn predict–look-again–refine process. A 'locate' tool posts a coordinate hypothesis, renders a red marker on the screenshot, and appends a local patch around the predicted region; a 'confirm' tool accepts or rejects the hypothesis and terminates the episode. Training is two-stage: SFT on roughly 70k Gemini3.1-Pro-distilled reflective trajectories, followed by GRPO with a terminal grounding correctness reward, with additional refusal cases synthesized via a CutPaste-style heuristic. Experiments on OSWorld-G, VenusBench-GD, ScreenSpot-Pro, UIVision, and MMBench-L2-GUI report large gains over the base Qwen3-VL model and state-of-the-art results. Ablations in Tables 3–5 attempt to attribute the gains to the multi-turn reflection mechanism and show generalization to a second backbone, but the central attribution is complicated by the SFT/RL comparison.","tokens_in":18049,"tokens_out":10240,"duration_ms":68649,"significance":"The contribution is conceptually clean and potentially impactful: it introduces a simple, general protocol for post-prediction visual verification, and the empirical gains are substantial and consistent across several benchmarks, including a new refusal-aware capability. The paper ships code, uses a rule-based reward, and provides multiple ablations, backbone generalization, and an analysis of refinement dynamics. If the mechanistic attribution is confirmed, this would be a useful recipe for the community. However, the current comparison does not cleanly separate the multi-turn protocol from the teacher-distilled SFT cold-start, and the refusal behavior is trained on synthetic composites whose validity is unverified; both issues need to be resolved before the central claims can be accepted.","major_comments":[{"comment":"The comparison in Table 3 between LookAgain-8B and the '+GRPO-Single Turn' variants does not isolate the multi-turn reflection mechanism because LookAgain-8B includes an SFT cold-start on roughly 70k Gemini3.1-Pro-distilled trajectories, while the single-turn GRPO variants are trained from the base model with reinforcement learning only. Table 4 shows that SFT alone raises UIVision from 23.3 to 37.5 (close to LookAgain's 38.8) and OSWorld-G-Refine from 67.8 to 73.5, so the residual gap in Table 3 (e.g., 60.2 versus 47.6/55.7 on ScreenSpot-Pro) could be substantially explained by teacher distillation rather than by the closed-loop protocol. The statement 'showing that the gain stems from multi-turn reflection itself rather than from simply avoiding the refusal burden' is therefore not supported by the reported experiments. Please add an ablation with a single-turn SFT+GRPO variant (or a multi-turn GRPO variant without SFT) to separate these factors.","section":"Ablation Study, Table 3"},{"comment":"The refusal training data is synthesized by pasting a patch from an adjacent side over the ground-truth bounding box and feathering the boundary (CutPaste-style). With only about 600 refusal trajectories for SFT and 300 for RL, the model may learn to detect composite-specific artifacts (seams, texture discontinuities) rather than to reason that the instruction's referent is absent. This is a particular concern because the paper reports large refusal gains over baselines that score 0.0 (e.g., 57.4 on OSWorld-G Refusal), and the benchmarks' refusal cases are presumably natural inapplicable instructions. The paper provides no validation that the synthetic refusal distribution matches real refusal scenarios. Please include a human evaluation of the synthetic composites (e.g., how often they look like genuine screenshots) and/or an evaluation on a held-out set of naturally occurring inapplicable instructions.","section":"Appendix, Refusal Data Construction"}],"minor_comments":[{"comment":"There is a typo in the first paragraph: 'degradation sharply small targets' is missing 'on' (should be 'degradation sharply on small targets').","section":"Introduction"},{"comment":"The locate repetition limit K used in the Violation condition is not specified anywhere; please report its value alongside the maximum interaction rounds.","section":"Implementation Details, Eq. (3)"},{"comment":"Figure 3 is referenced in the Analysis of Direct Closed-Loop Grounding section, but the quantitative scores for the direct closed-loop application to Gemini3.1-Pro, Qwen3-VL-8B, and the GUI-Cursor variant are not given in the text; please report the exact numbers.","section":"Method, Figure 3"},{"comment":"The benchmark name is inconsistent: 'VenusBench-GD' in Table 1 caption and 'Venus-GD' in Table 3; please standardize.","section":"Tables 1 and 3"},{"comment":"The statement 'All evaluations are repeated three times to ensure negligible fluctuation in results' should be accompanied by the actual variance (standard deviation or range) in the main tables; otherwise the 'negligible' claim is unverifiable.","section":"Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper has strong empirical results and a clean protocol. The main issue is the confound in the central ablation, which is fixable with additional training runs, and the refusal synthesis needs validation. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LookAgain is a well-engineered paper with a genuinely new angle: instead of adding more reasoning or zoom before the coordinate, it treats the emitted coordinate as a hypothesis and lets the model inspect a marker-rendered crop of its own prediction before confirming or refining. The locate/confirm protocol, the selective-mask SFT on teacher trajectories, and the GRPO stage with terminal-reward gating are all clearly described, and the results are strong across OSWorld-G, ScreenSpot-Pro, UIVision, and VenusBench-GD. They also show the recipe transfers to a second backbone and that smaller targets trigger more refinement rounds, which is consistent with the intended mechanism. I believe the main results are real and the contribution is worth a serious look.\n\nThe soft spot is the central mechanistic claim. The paper says the gain comes from multi-turn reflection rather than RL alone, but Table 3 compares full LookAgain (SFT+GRPO, multi-turn) against single-turn GRPO started from the base model, with no SFT. Table 4 shows SFT alone already accounts for most of the UIVision gain (23.3 to 37.5, versus 38.8 after GRPO), so the residual gap on ScreenSpot-Pro in Table 3 could be explained by the Gemini-distilled cold start rather than the closed-loop itself. The proper controls are missing: single-turn SFT on the same teacher data, or multi-turn GRPO without SFT (they mention a GUI-Cursor variant in the analysis, but don't give it a numbered ablation). This doesn't invalidate the approach, but it leaves the stated attribution underdetermined.\n\nTwo smaller issues. The refusal training set is synthesized by CutPaste-style patch replacement, with only 600 SFT and 300 RL samples, and no validation that the model is refusing because the referent is genuinely absent rather than because of compositing artifacts. The refusal results are dramatic, so this deserves a check. Also, they say evaluations were repeated three times but report no variance anywhere; that's a minor reporting gap.\n\nOverall, this is a solid, imaginative paper with a real protocol contribution. It deserves peer review, but the reviewers should ask for the clean ablations that separate teacher-distillation from the multi-turn mechanism, and some evidence on refusal data realism.","headline":"A genuinely new closed-loop GUI-grounding protocol with strong results, but the paper's key ablation confounds the multi-turn mechanism with SFT on Gemini-distilled trajectories.","tokens_in":18628,"tokens_out":2915,"would_cite":true,"duration_ms":19832,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LookAgain reformulates GUI grounding as a closed predict-look-again-refine loop, treating each emitted coordinate as a hypothesis to be verified and revised, and reports large accuracy gains over one-shot grounders.","keywords":["GUI grounding","closed-loop refinement","visually grounded reflection","multi-turn grounding","refusal-aware grounding","GRPO","screen agents"],"falsifier":"Evaluate LookAgain on refusal cases where the absent element is not covered by a same-image patch — for example, instructions referring to a button that never existed in a crowded toolbar. If refusal accuracy on this set is close to the synthetic refusal accuracy, the model understands absence; if it collapses while accuracy on the feathered-adjacent-patch cases stays high, the model has learned compositing artifacts rather than absence. A cheaper check is to remove the feathering from the synthetic refusal images: if accuracy drops sharply, the seam is the cue.","tokens_in":17644,"feed_emoji":"👀","tokens_out":9629,"duration_ms":66517,"temperature":0.7,"pith_summary":"This paper claims that the main bottleneck in GUI grounding is not what happens before a click but what happens after it: existing grounders freeze a predicted coordinate at the moment it is emitted, so no mechanism can challenge it under new visual evidence. LookAgain reformulates grounding as a closed predict-look-again-refine loop, where a 'locate' call posts a coordinate, renders a red marker at that point, and appends a high-resolution patch of the region, and a 'confirm' call either commits the coordinate or refuses the instruction. On top of a base 8B vision-language model, the closed loop raises OSWorld-G overall accuracy from 51.3 to 73.0, with the largest gains on small targets, dense layouts, and refusal cases where the instruction does not apply. The paper's ablation isolates the source of the gain: a single-turn reinforcement-learning variant reaches only 47.6 on ScreenSpot-Pro, while the full multi-turn reflection model reaches 60.2. The claim matters because grounding errors are a dominant cause of cascading failures in screen-operating agents, and this result suggests grounding should be treated as an evidence-accumulating decision rather than a one-shot regression.","feed_headline":"Re-checking its own clicks lifts GUI accuracy to 73.0","feed_subtitle":"The LookAgain loop corrects wrong clicks, raising OSWorld-G overall from 51.3 to 73.0.","key_machinery":"The load-bearing mechanism is the closed predict-look-again-refine loop built from two tool-use primitives. 'locate' renders a red marker at the predicted center and appends a 512×512 patch of that exact region, making the model's own hypothesis the addressable subject of the next reasoning step; 'confirm' commits or refuses and terminates. Around these primitives, the training recipe is the second half of the machinery: a turn-selective SFT mask that supervises genuine post-hoc reflections but never imitates a wrong intermediate coordinate, and a GRPO stage whose only reward is terminal grounding correctness, gated on a successful confirm and zeroed out on malformed calls, repeated locate boxes, or truncated trajectories.","core_discovery":"LookAgain's central claim is that a produced coordinate should be treated as a hypothesis to be reflected upon and revised under new visual evidence. The method implements this as a multi-turn tool-use protocol with two primitives: 'locate' posts a bounding-box hypothesis, overlays a red-dot-and-ring marker at its center, and appends a 512×512 crop of that exact region to the dialogue history; 'confirm' accepts or rejects the hypothesis and ends the episode, with 'failed' status used to abstain on inapplicable instructions. Every subsequent thought is therefore conditioned on the model's own previous prediction and the visual patch it summoned, turning iterative zoom-in over views into iterative correction over coordinates. Training is a two-stage pipeline: supervised fine-tuning on teacher-generated reflective trajectories (with a selective token mask that avoids imitating wrong intermediate coordinates), followed by GRPO whose only reward is whether the final confirm-committed prediction hits the ground-truth box (or, on refusal samples, whether the model correctly declares the instruction inapplicable). The paper reports that the 8B model reaches 73.0 overall on OSWorld-G versus 51.3 for the base model, 84.5 on MMBench-L2-GUI, 60.2 on ScreenSpot-Pro, and 38.8 on UIVision, and that the multi-turn reflection mechanism — not reinforcement learning alone — is what produces the gain.","pith_inferences":["A natural testable extension is to run the same locate-confirm loop on other fine-grained localization domains — document extraction, medical imaging, remote sensing — where a coordinate hypothesis can be rendered and inspected; the paper does not claim this, but the mechanism is domain-agnostic.","The refusal numbers depend on a CutPaste-style synthetic distribution where the absent element is always erased by pasting an adjacent patch over it; real inapplicable instructions often lack such a same-image patch to paste, so transfer of the reported 57.4 refusal accuracy to organic refusal cases remains an open question.","If verification is separated from commitment, evaluation should start reporting budgeted multi-turn grounding accuracy — for example, accuracy at a fixed number of locate calls — rather than single-shot accuracy alone, since the paper's bbox-count analysis suggests such budgets would trade accuracy against latency."],"forward_implications":["Grounding errors on small targets and dense layouts, a dominant source of cascading failures in screen agents, can be reduced by letting the model inspect its own predicted coordinate.","Refusal awareness — correctly deciding an instruction is inapplicable — can be trained, and on OSWorld-G the refusal subset jumps from 0.0 for several strong baselines to 57.4 for the 8B model without sacrificing standard accuracy.","The multi-turn reflection gain is not an artifact of reinforcement learning alone: a single-turn GRPO variant reaches only 47.6 on ScreenSpot-Pro versus 60.2 for LookAgain.","The same recipe transfers across backbones; on a 9B model it raises OSWorld-G-Refine to 84.9, UIVision to 49.9, and VenusBench-GD to 72.1."],"supporting_citations":[{"why":"Supplies the base 8B and 32B vision-language models that LookAgain is built on and compared against.","marker":"Bai et al. 2025"},{"why":"Provides the OSWorld-G and OSWorld-G-Refine benchmarks where the largest gains and refusal scores are measured.","marker":"Xie et al. 2026"},{"why":"CutPaste synthesis is used to construct the synthetic 'element does not exist' refusal training data.","marker":"Li et al. 2021"},{"why":"GRPO, the reinforcement-learning algorithm used in stage two with terminal grounding correctness as the sole reward.","marker":"Guo et al. 2025"},{"why":"Reflexion, the self-refinement precedent that motivates post-hoc reflection over longer upfront thinking.","marker":"Shinn et al. 2023"},{"why":"GUI-Cursor, the direct reinforcement-learning training recipe that the paper shows is insufficient to elicit reflection.","marker":"Zhao et al. 2025"},{"why":"MVP, the multi-view prediction baseline that aggregates independent hypotheses instead of refining one.","marker":"Zhang et al. 2026"},{"why":"Propose-then-Critic, the candidate-selection baseline that LookAgain is compared against as an alternative form of reflection.","marker":"Wang et al. 2026a"},{"why":"ScreenSpot-Pro, one of the three general GUI grounding benchmarks on which LookAgain reports large gains.","marker":"Li et al. 2025a"},{"why":"UIVision, the desktop-centric GUI benchmark where the 8B model gains 15.5 points over the base model.","marker":"Nayak et al. 2025"}],"fun_headline_variants":["GUI grounder that double-checks its picks hits 73.0","LookAgain: AI that rethinks its clicks, boosting GUI accuracy to 73%","Closed-loop self-reflection lifts GUI grounding to 73.0","LookAgain: visual reflection corrects clicked coordinates, reaches 73.0","Grounding GUI by re-examining its own predictions: 73.0"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The refusal behavior is trained almost entirely on synthetic 'the element is missing' images made by pasting a feathered patch from an adjacent side over the target, and the reported refusal accuracy transfers to real inapplicable instructions only if those composite images are a faithful stand-in for genuine absences.","fun_headline_variants_meta":{"raw":{"variants":["GUI grounder that double-checks its picks hits 73.0","LookAgain: AI that rethinks its clicks, boosting GUI accuracy to 73%","Closed-loop self-reflection lifts GUI grounding to 73.0","LookAgain: visual reflection corrects clicked coordinates, reaches 73.0","Grounding GUI by re-examining its own predictions: 73.0"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2463,"prompt_tokens":1160,"completion_tokens":1303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":776,"completion_tokens_details":{"reasoning_tokens":1202}},"tokens_in":776,"tokens_out":1303,"duration_ms":8809,"temperature":1.0,"reasoning_tokens":1202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:55:07.686548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate LookAgain on refusal cases where the absent element is not covered by a same-image patch — for example, instructions referring to a button that never existed in a crowded toolbar. If refusal accuracy on this set is close to the synthetic refusal accuracy, the model understands absence; if it collapses while accuracy on the feathered-adjacent-patch cases stays high, the model has learned compositing artifacts rather than absence. A cheaper check is to remove the feathering from the synthetic refusal images: if accuracy drops sharply, the seam is the cue.","supporting_citations":[{"cited_title":"2026 , url=","cited_arxiv_id":null,"evidence_quote":"MVP, the multi-view prediction baseline that aggregates independent hypotheses instead of refining one."}],"review_version":1}