{"id":"d0c9de9b-3e57-403a-bad1-96d01c4170bf","arxiv_id":"2502.01977","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AutoGUI automatically generates contextual functionality annotations for 704k UI elements by having an LLM compare screen state before and after simulated interactions, and training on this data improves VLM UI grounding.","lead":"This paper builds an automatic pipeline that uses a large language model to write descriptions of what buttons and menu items do, by watching how the screen changes after simulated clicks. It creates a 704k-sample dataset and shows that fine-tuning vision-language models on it improves their ability to locate UI elements, with gains growing as the dataset grows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains may come from annotation style rather than label correctness; the two-LLM verification shares the same interaction evidence, and the 145-sample human check cannot rule out systematic false labels.","rationale":"The reader's weakest-assumption analysis is on target: Section 3.2 formulates annotation as f = LLM(p_anno, s_t, s_t+1), and the human evaluation in Sec. 3.6 is the only external check on that mapping, with N around 145. I agree that this is the load-bearing point, and I sharpen it into a testable confound. The paper's strongest empirical evidence, Table 4 and Fig. 5, shows consistent gains on external benchmarks, but those gains would also occur if the model learned the register, length, and vocabulary of AutoGUI's annotations rather than their true referents. The Table 5 comparison against condensed functionality and HTML annotations supports the 'richness matters' reading, not the 'correctness matters' reading. The two-LLM verification is partially circular for the same reason: both verifiers receive the same interaction outcome from which the annotation was derived, so genuine errors consistent with that outcome can pass; Mistral's independence is only a difference of model, not of evidence. The 96.7% correctness estimate is computed on a small, author-rubric sample and cannot bound per-class systematic errors, such as navigation-triggering elements, icon-only elements, or elements whose interaction yields only attribute updates. A shuffled-label control is the decisive check because it holds annotation style constant and breaks only the semantic link between description and element. If that control shows correctness is causally important, the dataset claim stands; if not, the pipeline's quality apparatus may be optimizing for plausible-sounding language rather than functionality grounding. I therefore keep the CONDITIONAL verdict: the contribution is plausible and externally benchmarked, but it should not be taken as benchmark-grade until the label-correctness confound is resolved and ideally the dataset and artifacts are released.","tokens_in":30580,"tokens_out":4859,"duration_ms":53022,"concrete_test":"Train two control variants with the same hyperparameters as the 702k runs in Table 4: (1) per-sample shuffle: randomly permute the functionality descriptions among elements within the same screenshot or trajectory so every description is paired with a wrong element; (2) per-category shuffle: permute descriptions only within the same coarse functionality category, preserving style and topic statistics while breaking the element-description link. Evaluate both controls on FuncPred, ScreenSpot(-v2), MoTIF, and VWB. If the shuffled-label models retain most of the true-label 702k gain over the base model (e.g., more than 80% of the ScreenSpot gain), the fine-tuning results cannot be attributed to annotation correctness; if they collapse to near-base performance, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing link is that correct contextual functionality annotations, not merely their linguistic form or distribution, drive the grounding gains in Table 4 and Fig. 5. The only external quality check is the human evaluation in Sec. 3.6 (N=145-146) using author-defined criteria (Fig. I), which is far too small to characterize systematic label errors across 704k samples. Internally, the verification stage (Sec. 3.4) feeds both Llama-3-70B and Mistral-7B the same UI-state differences that produced the annotation; agreement between the two verifiers therefore reflects shared input evidence, not independent ground truth. Concretely, if an element's interaction outcome admits a plausible but wrong functionality (e.g., a button whose only change is a focus attribute, Sec. A.5 and Tab. D), the verifiers have no independent signal to contradict it. The comparison in Tab. 5 shows full descriptions beat condensed and HTML forms, but this only demonstrates that richer text helps; it does not demonstrate that the text is semantically correct. Without a label-perturbation control, the observed benchmark improvements are consistent with VLMs learning annotation style and UI-text statistics rather than true element functionality. This is the weakest load-bearing assumption because the entire quality-control apparatus (rejection, verification, 96.7% claim) is meant to guarantee something that the fine-tuning experiments never isolate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoGUI, a pipeline that automatically generates contextual functionality annotations for GUI elements by prompting an LLM (Llama-3-70B) to infer element functionality from UI state changes before and after simulated interactions. A hand-rule plus LLM rejector removes invalid samples, and two LLM verifiers (Llama-3-70B and Mistral-7B) filter annotations. The authors curate AutoGUI-704k, a web and mobile dataset of 704k element-functionality pairs, and report a human-evaluation correctness of 96.7% versus 95.5% for a trained human annotator. Fine-tuning open VLMs (Qwen-VL, SliME, Qwen2-VL, SeeClick) on AutoGUI data improves grounding on FuncPred, ScreenSpot, MoTIF, and VisualWebBench, with accuracy increasing as the data scale grows from 25k to 702k. The paper also shows a potential use of the dataset in a two-stage GUI agent planning pipeline evaluated on AITW.","tokens_in":30888,"tokens_out":7013,"duration_ms":63723,"significance":"If the claims hold, AutoGUI is a valuable contribution: it directly addresses the scarcity of large-scale contextual functionality annotations for GUI grounding, provides a reproducible automatic pipeline and a public dataset, and shows consistent gains on multiple external grounding benchmarks. Strengths include the dataset scale (704k samples), the multi-benchmark evaluation, the data-scaling analysis, and the use of open models and open-source codebases. The concerns lie in the strength of the evidence for annotation correctness: the human evaluation is small, the LLM verification shares evidence with the annotation step, and the fine-tuning results do not yet isolate label correctness from annotation style. These gaps are addressable and do not undermine the overall value of the dataset, but they should be resolved before the central claims are taken as established.","major_comments":[{"comment":"The human evaluation is too small to support the precision implied by the 96.7% versus 95.5% comparison. Section 3.6 states N=145 samples (99 valid and 46 invalid), while Section B.1 says the human annotator evaluated 146 samples; this inconsistency should be fixed. With N≈145 and a correctness proportion near 0.95, the standard error is roughly 1.6 percentage points, so the reported difference between AutoGUI and the human annotator is within noise. Additionally, the correctness metric C/(N-R) subtracts rejected samples from the denominator, so high correctness can be achieved by aggressive rejection; the paper should report the raw accuracy, the rejection rate, and a precision/recall breakdown so that the 96.7% claim can be interpreted properly.","section":"Section 3.6, Table 3"},{"comment":"The two-LLM verification does not provide an independent check on annotation correctness. Both verifiers receive the same UI state differences (AXTree diffs or UI descriptions) that produced the annotation, so agreement between Llama-3-70B and Mistral-7B demonstrates consistency of inference from the same evidence, not correctness against ground truth. The human evaluation is the only external check and is too small to detect class-level systematic errors, such as elements whose only UI change is a focus or attribute update (as in Section A.5 and Figure D). This matters because the 96.7% correctness claim is used to justify the downstream fine-tuning results; the paper should either enlarge the human evaluation with a more diverse sample or provide a targeted analysis of error rates by interaction-outcome type.","section":"Section 3.4, 3.6"},{"comment":"The comparison of annotation types in Table 5 (Elem-HTML, Condensed Func., Func.) does not isolate label correctness from annotation style. The full functionality descriptions are longer and more structured than the condensed and HTML variants, so the gains could reflect the linguistic form and statistics of the annotations rather than their semantic correctness. A label-perturbation control is needed: for example, fine-tuning on functionality descriptions that are fluent and contextually styled but describe the wrong element (or shuffled element-description pairs) would show whether the grounding improvements in Table 4 and Figure 5 are driven by true functionality semantics. Without such a control, the observed gains are consistent with VLMs learning the annotation distribution rather than genuine element functionality.","section":"Section 4.2, Table 5"},{"comment":"The scaling curves and the fine-tuned model accuracies are reported from single runs with no error bars or significance tests. This is a load-bearing issue because the central empirical claim includes a monotonic scaling effect from 25k to 125k to 702k samples, and some individual benchmark gains are small (e.g., Qwen2-VL on MoTIF, +1.2 in Table 4). Run-to-run variance in LoRA fine-tuning could be comparable to these differences. I request multiple seeds (at least three) for the key scaling points, or confidence intervals, for the FuncPred, ScreenSpot, and MoTIF results in Figures 5 and Table 4.","section":"Figure 5, Table 4"},{"comment":"FuncPred is constructed by the same annotation pipeline as the training data, so improvements on FuncPred may partly reflect in-distribution alignment with the AutoGUI annotation style rather than generalizable functionality understanding. The external benchmarks (ScreenSpot, MoTIF, VisualWebBench) are independent but use brief instructions or action intents rather than full functionality descriptions, so they do not directly validate the semantic correctness of the functionality annotations. The paper should explicitly acknowledge this asymmetry in the evaluation and discuss how much of the evidence for 'functionality grounding' rests on the in-distribution FuncPred test set. Reporting results on a held-out set of manually authored functionality descriptions would strengthen the claim considerably.","section":"Section 4.1, FuncPred"}],"minor_comments":[{"comment":"There is a typo: 'avoidding' should be 'avoiding'.","section":"Appendix A.1"},{"comment":"The limitation heading 'AutoGUI is Not Indented to Record Task-Oriented Interaction Trajectories' contains a typo: 'Indented' should be 'Intended'.","section":"Limitations"},{"comment":"In Table 6, the Install and Single columns for the Qwen2-VL-7B SFT w/ AutoGUI rows show identical values (25.59/22.49 under GPT-4o-mini and 50.95/48.95 under Gemini), which looks like a copy-paste error; please verify these numbers.","section":"Table 6"},{"comment":"The sentence 'After experimenting with three runs, Tab. 3 shows...' is unclear: it is ambiguous whether the three runs refer to the human annotation, the AutoGUI pipeline, or the three LLM runs. Please specify what the three runs are.","section":"Section 3.6"},{"comment":"The citation for Qwen2-VL in Table 4 and Section 4.1 is given as (Bai et al., 2023), but the reference list correctly attributes Qwen2-VL to Wang et al., 2024a; the in-text citation should be corrected.","section":"Table 4, Section 4.1"},{"comment":"The caption of Figure L says 'ours-625k' while the text in Section D.2 consistently refers to 'ours-702k'; please make the model scale consistent.","section":"Figure L, Section D.2"},{"comment":"The 'Avg. Words' column in Table 2 would be clearer as 'Avg. words per annotation', since the 'Anno. Tokens' column already gives the total token count.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid dataset-and-pipeline paper with a strong empirical backbone. The main technical concern is the lack of a control that isolates annotation correctness from annotation style, and the small human evaluation. I would ask for a label-perturbation experiment and multi-seed results for the scaling claims. These are feasible additions and would materially strengthen the paper. I also noticed the duplicate numbers in Table 6; those should be checked for a possible transcription error."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: AutoGUI is a real step forward for GUI grounding data. The idea of using LLMs to infer element functionality by diffing accessibility trees before and after simulated clicks is genuinely new, and the scale they reach (704k annotations) is much larger than prior contextual functionality datasets. The paper is worth a serious referee and, with revision, likely a place in the field's toolkit.\n\nWhat's good: The pipeline is well thought out. Rejecting invalid samples with rules plus an LLM scorer, then verifying annotations with two LLM verifiers, is a sensible quality-control chain. The human evaluation, though small, shows the pipeline roughly matches a trained annotator (96.7% vs 95.5%), and the ablations (r2-r6) demonstrate that each stage contributes. The fine-tuning experiments are broad: multiple base models, external benchmarks (ScreenSpot, MoTIF, VWB), and a scaling curve from 25k to 702k. Consistent gains across those benchmarks are hard to fake. The appendix is honest about limitations: mobile app diversity, lack of task-orientation, and read-only bias.\n\nSoft spots: The human eval is 145-146 samples with author-defined criteria. That's enough for a sanity check but not enough to characterize error rates across 704k diverse elements. More importantly, the verification stage feeds both Llama-3-70B (also the annotator) and Mistral-7B the same interaction evidence. Their agreement reflects shared input, not independent ground truth, so a systematically plausible-but-wrong label could slip through. The stress-test note is right that no label-perturbation control isolates whether gains come from true functional semantics or from the linguistic style/distribution of the annotations. The Table 5 comparison against condensed/HTML forms shows richer text helps, but not that the text is semantically correct. That said, the external benchmark gains are real and consistent, so this is a soft spot, not a fatal flaw. It should be addressed with a larger human eval and a perturbation control (e.g., corrupting labels and showing performance drop).\n\nThe FuncPred benchmark is generated by the same pipeline, so it's in-distribution; the claims don't hinge on it alone, since the external benchmarks are independent.\n\nRecommendation: This paper deserves peer review. It's novel, useful, and mostly solid. Send it to reviewers with a request to probe the annotation quality more deeply, but do not desk-reject.","headline":"A genuinely new annotation mechanism for GUI grounding at scale, with broad empirical support; the quality-control evidence needs deepening but the contribution deserves peer review.","tokens_in":31397,"tokens_out":2194,"would_cite":true,"duration_ms":20531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoGUI shows that an LLM reading a UI's before-and-after state can generate, at scale, functionality labels whose quality matches a trained human annotator, and that these labels teach vision-language models to locate GUI elements much…","keywords":["GUI grounding","functionality annotation","vision-language model","UI understanding","automatic annotation","accessibility tree","LLM verification","data scaling"],"falsifier":"Run the paper's own human-evaluation protocol on an order-of-magnitude larger sample, roughly 1,500 randomly drawn annotations spanning all 916 training domains plus the 2k-sample test split, with two independent trained annotators applying the paper's three criteria (context-specificity, appropriate detail, no hallucination). If measured correctness falls materially below the 95.5% human baseline, or if errors concentrate in one functional class such as filters, form fields, or nested menus, the claim that state-difference inference plus two-LLM verification reproduces human annotation quality would be falsified. A complementary check is to feed the two verifier LLMs a set of human-graded wrong annotations and measure how many still receive full scores from both, since the verification stage is only load-bearing if it catches what humans call wrong.","tokens_in":2136,"feed_emoji":"🖱️","tokens_out":2370,"duration_ms":124372,"temperature":0.7,"pith_summary":"AutoGUI claims that the reason vision-language models fail at GUI grounding is a data bottleneck, not a model bottleneck: there has been no way to obtain contextual functionality descriptions for interface elements at scale. The paper's pipeline removes the human from that loop by simulating a click or hover, computing the difference between the accessibility trees before and after, and asking an LLM to read that difference as evidence of the element's function, with an LLM rejector and two independent LLM verifiers filtering out unreadable and wrong samples. The resulting AutoGUI-704k dataset reaches 96.7% annotation correctness on a small human evaluation, slightly above a trained human annotator, and offers a diversity and detail of descriptions that prior datasets lack. Fine-tuning open VLMs on this data lifts their grounding accuracy on five benchmarks, with accuracy rising monotonically as the training set scales from 25k to 702k samples. If the claim holds, GUI grounding stops being a labor-bound annotation problem and becomes an automatic, scalable data-generation problem that can directly feed GUI-agent development.","feed_headline":"LLM-written UI function labels teach VLMs where to click","feed_subtitle":"AutoGUI infers what a button does from the screen changes it causes, verified by two LLMs, no human annotation.","key_machinery":"The load-bearing mechanism is the state-difference inference $f = \\mathrm{LLM}(p_{\\mathrm{anno}}, s_t, s_{t+1})$: an LLM reads the diff between accessibility-tree snapshots taken before and after a simulated interaction and produces a contextual functionality summary. Because naive text diffing makes a stable element look both deleted and added, the pipeline adds custom 'attribute update' and 'repositioning' markers so the LLM sees structural changes accurately. Two filters keep the labels trustworthy: an LLM rejection stage scores whether the observed change is explicit, relevant, and predictable and discards the bottom 30% of samples, and a verification stage keeps a sample only when two different LLMs both score the annotation as fully supported by the evidence. The verified triplets are then templated into paired grounding and captioning tasks with coordinates normalized to $[0, 999]$.","core_discovery":"The central discovery is that an element's contextual functionality is legible in the change it produces, so functionality labels can be manufactured at scale without human annotators and without reading HTML or other surface metadata. The pipeline, formalized as $f = \\mathrm{LLM}(p_{\\mathrm{anno}}, s_t, s_{t+1})$, gives an LLM the accessibility tree before ($s_t$) and after ($s_{t+1}$) a simulated click or hover and asks it to reason through the added, deleted, unchanged, attribute-updated, and repositioned lines and summarize what the element does. Quality control is a rejection stage that scores each sample on explicitness, relevance, and predictability of the observed change, discarding the worst 30%, followed by a verification stage that keeps a sample only if two different LLMs both award full marks to the claimed functionality. The authors report 96.7% annotation correctness against 95.5% for a trained human annotator, and show that fine-tuning Qwen-VL, SliME, Qwen2-VL, and SeeClick on the resulting 702k-sample training set improves grounding on FuncPred, ScreenSpot, ScreenSpot-v2, MoTIF, and VisualWebBench, with gains growing as data scales and with full contextual descriptions outperforming condensed descriptions and raw HTML-element pairs.","pith_inferences":["Because the paper's scaling curve shows no saturation through 702k samples, a direct extrapolation is that feeding the same pipeline multi-million-scale trajectories and stronger open LLMs would push open VLMs clearly past current UI experts; the released 2k-sample test set makes this measurable.","The paper's own limitations imply a coverage gap: elements that modify internet content are excluded, so AutoGUI-trained models likely ground informational affordances (navigation, filtering, menus) better than transactional ones (submit, purchase, post); the dataset's verb-noun statistics make this per-category accuracy testable.","Since Llama-3-70B both annotates and serves as one of the two verifiers, the verification stage could partly reflect self-preference; an independent check would be to verify with two models from non-overlapping model families and re-measure the 96.7% correctness number.","The state-difference mechanism is not GUI-specific: any interactive surface with a serializable state, such as spreadsheet cells, IDE panels, or multi-step web forms, could receive the same automatic functionality annotation, which would let instruction-following data be generated for software beyond web and mobile interfaces."],"forward_implications":["General-purpose VLMs fine-tuned on AutoGUI-702k gain large grounding accuracy on all five benchmarks, with Qwen2-VL-7B reaching 65.0% on FuncPred and 80.0% on ScreenSpot, approaching or exceeding specialized UI experts on the functionality-focused benchmark.","Grounding accuracy rises monotonically as the AutoGUI training data scales from 25k to 125k to 702k samples, indicating that the pipeline converts compute and crawling effort directly into model capability.","Full contextual functionality descriptions outperform both raw HTML-element pairs and condensed one-line descriptions as training annotations, and the gap is widest on the FuncPred benchmark that emphasizes contextual functionality grounding.","In a two-stage GUI-agent setup, replacing a proprietary planner's own coordinate prediction with an AutoGUI-trained grounding model raises average step accuracy on AITW from 14.01% to 18.37% with one planner and from 29.50% to 39.23% with another.","Each quality-control component adds measurable correctness: rules alone reach 83.1%, adding the LLM rejector raises this to 94.4%, one verifier to 96.0%, and two verifiers to 96.7%, against the 95.5% human baseline."],"supporting_citations":[{"why":"Supplies Llama-3-70B, the LLM that performs functionality annotation, rejection scoring, and one of the two verification votes.","marker":"AI@Meta (2024)"},{"why":"Chain-of-thought prompting is the technique that makes the annotation LLM reason through the diff before summarizing the element's function.","marker":"Wei et al. (2022)"},{"why":"Step-by-step LLM verification is the inspiration for the verification stage that filters out incorrect functionality annotations.","marker":"Lightman et al. (2023)"},{"why":"SeeClick is the UI-expert baseline model that AutoGUI data must improve, the source of the ScreenSpot benchmark, and the fine-tuning setup used for Qwen-VL.","marker":"Cheng et al. (2024)"},{"why":"CogAgent provides the Elem-HTML annotation type from UI metadata that AutoGUI functionality annotations are compared against in Table 5.","marker":"Hong et al. (2024)"},{"why":"UGround is the strongest grounding expert baseline, and its two-stage planner-grounding setup is adopted for the AITW agent experiment.","marker":"Gou et al. (2025)"},{"why":"MoTIF serves as the mobile navigation grounding benchmark used to evaluate the fine-tuned models.","marker":"Burns et al. (2022)"},{"why":"VisualWebBench provides the web element and action grounding tasks used as evaluation benchmarks.","marker":"Liu et al. (2024a)"},{"why":"AITW is the downstream GUI-agent benchmark on which the two-stage planning experiment measures step accuracy.","marker":"Rawles et al. (2023)"},{"why":"AndroidControl trajectories are one of the mobile interaction sources that extend the pipeline's coverage to Android UIs.","marker":"Li et al. (2024a)"}],"fun_headline_variants":["LLMs auto-label UI functions by watching state changes","AutoGUI: LLM-written GUI labels from before/after clicks","Scaling UI grounding with LLM-verified state-diff annotations","AutoGUI: no humans needed, LLMs describe UI elements from diffs","Click, diff, describe: AutoGUI's LLM pipeline for UI labels"],"cache_read_input_tokens":33536,"weakest_assumption_plain":"Everything downstream rests on the assumption that an LLM reading a before-and-after snapshot of a screen's accessibility tree can correctly name what the clicked element does; if that inference is systematically wrong for some class of elements, the rejection and verification filters could keep confidently false labels, and the small (145-146 sample) human evaluation is the only external check.","fun_headline_variants_meta":{"raw":{"variants":["LLMs auto-label UI functions by watching state changes","AutoGUI: LLM-written GUI labels from before/after clicks","Scaling UI grounding with LLM-verified state-diff annotations","AutoGUI: no humans needed, LLMs describe UI elements from diffs","Click, diff, describe: AutoGUI's LLM pipeline for UI labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2062,"prompt_tokens":1022,"completion_tokens":1040,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":948}},"tokens_in":638,"tokens_out":1040,"duration_ms":10438,"temperature":1.0,"reasoning_tokens":948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:48:53.715169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own human-evaluation protocol on an order-of-magnitude larger sample, roughly 1,500 randomly drawn annotations spanning all 916 training domains plus the 2k-sample test split, with two independent trained annotators applying the paper's three criteria (context-specificity, appropriate detail, no hallucination). If measured correctness falls materially below the 95.5% human baseline, or if errors concentrate in one functional class such as filters, form fields, or nested menus, the claim that state-difference inference plus two-LLM verification reproduces human annotation quality would be falsified. A complementary check is to feed the two verifier LLMs a set of human-graded wrong annotations and measure how many still receive full scores from both, since the verification stage is only load-bearing if it catches what humans call wrong.","supporting_citations":[],"review_version":1}