{"id":"c68557d2-2f3c-4bb2-b20d-73f4ee375bfc","arxiv_id":"2605.14747","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Video2GUI converts 500 million video metadata entries into a 12-million-trajectory dataset spanning 1500+ applications, and pretraining on it improves GUI grounding and action benchmarks by 5-20%.","lead":"Video2GUI is an automated pipeline that mines unlabeled internet videos to generate millions of structured GUI interaction trajectories for training agents. A smart generalist might read it because scalable data collection is currently the main barrier to building GUI agents that work across many different apps and websites.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader correctly flags the filtering-to-trajectory step as the weakest link given only the abstract. Full text would allow direct evaluation of that step; until then the claim cannot be accepted or rejected.","tokens_in":1687,"tokens_out":251,"duration_ms":10952,"concrete_test":"Sample 200 trajectories from the released WildGUI; have two independent annotators score action type, target element, and sequence correctness; compute inter-annotator agreement and mean accuracy. If mean accuracy < 85 % or agreement < 0.7, re-train the reported models on the filtered subset and re-measure benchmark deltas.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim rests on WildGUI trajectories being sufficiently accurate for pre-training gains. The abstract describes a coarse-to-fine pipeline applied at scale, but without the full text the precise error rates, validation protocol, or ablation on noisy trajectories cannot be assessed. If the full methods section supplies quantitative checks (e.g., human agreement on extracted actions or downstream ablation removing low-confidence samples), the assumption holds; otherwise the 5-20 % gains remain unanchored.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Video2GUI, an automated coarse-to-fine pipeline that mines unlabeled internet videos to extract structured GUI interaction trajectories, yielding the WildGUI dataset of 12 million trajectories across >1,500 applications. It reports that pre-training Qwen2.5-VL and Mimo-VL on WildGUI produces 5-20% gains on GUI grounding and action benchmarks, reaching or exceeding prior state-of-the-art.","tokens_in":1769,"tokens_out":402,"duration_ms":13707,"significance":"A scalable, annotation-free source of diverse real-world GUI trajectories would address a central bottleneck for GUI agents. If the extracted trajectories prove sufficiently accurate, the reported gains would indicate a practical route to improved generalization; the release of both dataset and pipeline would further amplify impact.","major_comments":[{"comment":"Abstract and §4 (dataset construction): the claim that the coarse-to-fine filtering produces high-quality trajectories rests on an unquantified assumption; no human agreement rates, extraction error rates, or ablation removing low-confidence samples are supplied to anchor the 5-20% downstream gains.","section":"Abstract, §4"},{"comment":"§5 (experiments): baseline comparisons and statistical tests for the reported improvements are not described; it is therefore unclear whether the gains are robust to different random seeds, trajectory noise levels, or alternative filtering thresholds.","section":"§5"}],"minor_comments":[{"comment":"Notation for trajectory elements (e.g., action types, grounding coordinates) should be defined once in a table or figure caption rather than repeated inline.","section":null},{"comment":"The 500-million metadata entry figure would benefit from a brief breakdown of how many videos survive each filtering stage.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and commit to revisions that strengthen the validation of trajectory quality and experimental reporting.","responses":[{"response":"We agree that the manuscript would benefit from direct quantification of trajectory quality. The original submission did not include human evaluation metrics or ablations on the filtering stages. In the revision we will add a human study on a sampled subset of trajectories reporting inter-annotator agreement and extraction error rates, together with an ablation that removes low-confidence samples and measures the resulting change in downstream pre-training gains.","revision_made":"yes","referee_comment":"[Abstract, §4] Abstract and §4 (dataset construction): the claim that the coarse-to-fine filtering produces high-quality trajectories rests on an unquantified assumption; no human agreement rates, extraction error rates, or ablation removing low-confidence samples are supplied to anchor the 5-20% downstream gains."},{"response":"We acknowledge the need for more detailed experimental reporting. The revised §5 will expand the description of all baselines, report results averaged across multiple random seeds with standard deviations, and include statistical significance tests. We will also add supplementary experiments that vary trajectory noise levels and filtering thresholds to demonstrate robustness of the observed gains.","revision_made":"yes","referee_comment":"[§5] §5 (experiments): baseline comparisons and statistical tests for the reported improvements are not described; it is therefore unclear whether the gains are robust to different random seeds, trajectory noise levels, or alternative filtering thresholds."}],"tokens_in":1284,"tokens_out":345,"duration_ms":26734,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core new piece is Video2GUI, which runs a coarse-to-fine filter over 500 million video metadata entries to pull out GUI tutorial clips and convert them into structured trajectories. The result is WildGUI, 12 million trajectories covering more than 1,500 applications and sites. That scale is bigger than the manually labeled sets people have been using, and the authors release both the data and the pipeline.\n\nThey then pretrain Qwen2.5-VL and Mimo-VL on it and report 5-20% gains on several grounding and action benchmarks, sometimes matching or beating prior numbers. The automation itself is the practical advance here; it sidesteps the cost of hand annotation and opens a route to more diverse real-world coverage.\n\nThe main gap is evidence on trajectory quality. The abstract describes the filtering steps but gives no error rates, no human agreement numbers on the extracted actions, and no ablation that removes low-confidence samples to check whether the gains survive cleaner data. Without those checks it is hard to know how much noise is in the set or whether the reported lifts are robust. The citation pattern looks standard for the area, but the experimental section will need to carry more weight than the abstract suggests.\n\nThis is for groups working on GUI agents who are bottlenecked by data volume rather than by model architecture. A reader who needs large-scale interaction examples will get concrete value from the released dataset even if the validation story needs tightening. The work is coherent on its own terms and shows clear thinking about the data problem, so it deserves a serious referee.\n\nI would send it to review and ask specifically for quantitative validation of the trajectories and for ablations on data quality.","headline":"The paper gives a scalable video-mining pipeline for GUI trajectories and a 12M-example dataset that lifts two VL models on agent benchmarks, but the accuracy of the extracted data is not shown in enough detail.","tokens_in":2247,"tokens_out":431,"would_cite":false,"duration_ms":17280,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Video2GUI extracts 12 million GUI interaction trajectories from unlabeled videos to pretrain agents.","keywords":["GUI agents","video-to-trajectory synthesis","multimodal pretraining","WildGUI dataset","automated data generation","graphical user interfaces","interaction trajectories","agent generalization"],"falsifier":"Pre-training the same models on WildGUI and measuring no improvement or a performance drop on the GUI benchmarks, or manual inspection revealing high rates of incorrect action sequences in the extracted trajectories.","tokens_in":2600,"feed_emoji":"🖥️","tokens_out":635,"duration_ms":29884,"temperature":0.7,"pith_summary":"The paper presents Video2GUI, an automated framework that identifies GUI tutorial videos from internet sources and converts them into structured interaction trajectories. Processing 500 million video metadata entries produces the WildGUI dataset spanning over 1,500 applications and websites. Pre-training multimodal models on this data delivers 5-20% gains on GUI grounding and action benchmarks. The approach targets the scarcity of large, diverse training data that currently limits how well GUI agents generalize beyond narrow domains.","feed_headline":"12M GUI trajectories extracted from unlabeled videos","feed_subtitle":"Pretraining on the resulting dataset lifts multimodal models 5-20% on grounding and action benchmarks.","key_machinery":"The coarse-to-fine filtering strategy within Video2GUI that selects GUI tutorial videos and structures their content into accurate interaction trajectories for large-scale pretraining.","core_discovery":"Video2GUI is a fully automated framework that employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos from unlabeled internet videos and converts their visual content into structured agent trajectories. Applying the pipeline to 500 million video metadata entries yields the WildGUI dataset containing 12 million interaction trajectories across more than 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI produces consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance.","pith_inferences":["The same video-to-trajectory approach could supply training data for agents in related interactive domains such as web navigation or mobile robotics.","Mixing the automatically generated trajectories with smaller manually annotated sets might produce further gains.","Future refinements could add verification steps to reduce any residual noise in the extracted sequences."],"forward_implications":["GUI agents can be pretrained on trajectories from thousands of diverse real-world applications without manual annotation.","Existing multimodal models gain measurable improvements on grounding and action tasks after exposure to the dataset.","The released dataset and pipeline support continued scaling of training data for generalized agents.","Performance gains appear consistent across the two model families tested in the experiments."],"fun_headline_variants":["Video2GUI generates 12M GUI trajectories from web videos","WildGUI contains 12M trajectories from unlabeled videos","12M interaction trajectories extracted via automated pipeline","GUI agent data scaled using Video2GUI from internet videos"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The filtering strategy can reliably select high-quality videos and convert their content into accurate, structured trajectories without introducing substantial noise or systematic errors.","fun_headline_variants_meta":{"raw":{"variants":["Video2GUI generates 12M GUI trajectories from web videos","WildGUI contains 12M trajectories from unlabeled videos","12M interaction trajectories extracted via automated pipeline","GUI agent data scaled using Video2GUI from internet videos"]},"model":"grok-4.3","cost_usd":0.006716,"raw_usage":{"total_tokens":3128,"prompt_tokens":668,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":67162000,"prompt_tokens_details":{"text_tokens":668,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2398,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":668,"tokens_out":62,"duration_ms":28805,"temperature":1.0,"reasoning_tokens":2398,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T20:58:42.285536+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Pre-training the same models on WildGUI and measuring no improvement or a performance drop on the GUI benchmarks, or manual inspection revealing high rates of incorrect action sequences in the extracted trajectories.","supporting_citations":[],"review_version":1}