{"id":"9a0aa244-a972-4e04-b5b0-c7c969b32f23","arxiv_id":"2505.12632","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MONDAY automatically converts 20,000 YouTube mobile instructional videos into 313K annotated navigation frames and reports that models trained on it generalize better on an unseen mobile OS.","lead":"MONDAY is a dataset of 313,000 frames from 20,000 YouTube instructional videos that teaches AI agents how to navigate mobile apps on iOS and Android. The authors also build an automated pipeline that turns such videos into annotated task steps, and they claim that agents trained on it generalize better to a mobile platform they never saw.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 'unseen OS' generalization rests on a Windows Mobile test set segmented by the same pipeline that generated MONDAY training frames, so the 18.11pp gain may reflect shared segmentation bias rather than cross-platform transfer.","rationale":"The central claim is that automatically mined YouTube video data transfers to a never-seen OS, and the only direct evidence for the headline 18.11pp number is the Windows Mobile column of Table 5. The reader's concern is that this test set is not a clean OOD benchmark: Section 4.2.1 says it was annotated 'following the same annotation protocol' as the MONDAY evaluation set, and Supplementary D says that protocol uses the authors' scene-transition detection output to define the scenes between which annotators label actions. Thus the Windows Mobile episodes share the same automatic frame-generation distribution as MONDAY training data, giving MONDAY-induced models a distributional advantage over baselines trained only on AitW/AMEX. The independent test sets provide only weak support: on AitW and AMEX the gains are mostly 0.5-4pp and one AMEX comparison regresses, whereas the Windows Mobile gains range from about 12 to 23pp. If the models had truly learned platform-agnostic navigation patterns, one would expect more substantial and consistent improvements on independent benchmarks. The proposed re-annotation with independent scene boundaries directly tests whether the gain is tied to the pipeline's segmentation. This does not undermine the dataset or framework contributions, but it means the generalization claim requires this independent validation before being treated as established. The reader's CONDITIONAL verdict is appropriate and unchanged by this stress-test pass.","tokens_in":20343,"tokens_out":5753,"duration_ms":66398,"concrete_test":"Take the same 50 Windows Mobile videos and have independent annotators define scene transitions and action labels from scratch, without using the MONDAY pipeline's scene-transition output. Then rerun the four Table 5 finetuned-vs-baseline comparisons on this independently segmented test set. If the average MONDAY-induced gain on Windows Mobile drops to roughly the 1-4pp level seen on AitW/AMEX, the 18.11pp 'unseen OS' claim is an artifact of shared pipeline segmentation rather than cross-platform generalization. If the gain persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the headline claim is that the Windows Mobile evaluation is a clean out-of-distribution test. Section 4.2.1 says the Windows Mobile test was annotated 'following the same annotation protocol' as the MONDAY evaluation set, and Supplementary Section D reveals that this protocol is: 'Using our scene transition detection output, annotators labeled actions between consecutive scenes.' Therefore the Windows Mobile test episodes are defined by the same OCR-based scene-transition detector and midpoint frame-sampling rule used to generate MONDAY training frames. MONDAY-trained models are evaluated on frames drawn from the same automatic segmentation distribution they were trained on, while the AitW/AMEX-only baselines are not. The reported +13 to +23pp gains on Windows Mobile could thus reflect adaptation to pipeline-specific segmentation regularities rather than to Windows Mobile as an operating system. Independent evidence on AitW and AMEX is weak and mixed: gains are mostly 0.5-4pp, with one regression on AMEX (66.13 vs 68.19). The large effect is not corroborated by independent benchmarks, so the central generalization claim depends on a test set that is not fully independent of the framework.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MONDAY, a dataset of 313K annotated frames from 20K YouTube instructional videos covering iOS and Android navigation, together with an automated pipeline that uses OCR-based scene transition detection, GroundingDINO-based UI element detection, and a three-step GPT-4o action identification process to generate task episodes without manual annotation. Component-level evaluations report 95.04% F1 for scene detection, 99.87% hit ratio for UI element detection, and 80.90% action identification accuracy. The authors then fine-tune SeeClick and Llama-3.2 with LoRA on MONDAY and evaluate on AitW, AMEX, MONDAY, and a Windows Mobile test set, claiming an average 18.11 percentage-point gain on the unseen Windows Mobile platform and robust cross-platform generalization.","tokens_in":20586,"tokens_out":3521,"duration_ms":36974,"significance":"If the central claims hold, the work would be a valuable contribution: it proposes a scalable, low-cost alternative to manual dataset construction for mobile GUI agents, releases a large dataset and code, and reports strong component-level results with a clear automated pipeline. The claim that YouTube-sourced data transfers to a never-seen OS is ambitious and practically important. However, the headline cross-platform result currently rests on a Windows Mobile test set whose frames are selected by the same automatic scene-transition detector used to build MONDAY training episodes, and the supporting independent evidence on AitW and AMEX is mixed and without uncertainty quantification. The dataset release and component evaluations are credible strengths, but the generalization claim needs stronger, independent validation.","major_comments":[{"comment":"The Windows Mobile evaluation is not an independent out-of-distribution test. Section 4.2.1 states the Windows Mobile test set was annotated 'following the same annotation protocol' as the main evaluation set, and Supplementary Section D reveals that this protocol uses 'our scene transition detection output' to define the frames and episodes that annotators label. Because MONDAY training frames are also extracted using this same OCR-based transition detector with the midpoint sampling rule, the MONDAY-finetuned models are evaluated on frames drawn from the same automatic segmentation distribution they were trained on, while the AitW/AMEX-only baselines see that distribution only at test time. The reported +13 to +23 percentage-point gains on Windows Mobile could therefore reflect adaptation to pipeline-specific segmentation regularities rather than transfer to Windows Mobile as an operating system. To support the headline 'unseen OS' claim, the authors should either re-annotate the Windows Mobile test set using human-defined scene boundaries and frame selection independent of the MONDAY pipeline, or provide evidence (e.g., evaluating on all frames rather than pipeline-selected midpoints) that the frame-sampling choice does not affect the comparison.","section":"Section 4.2.1 and Supplementary D"},{"comment":"The abstract's claim that MONDAY-based models 'consistently outperform' models trained on existing single-OS datasets is directly contradicted by Table 5: in the AMEX-finetuned SeeClick row, SeeClick-MONDAY scores 66.13 on AMEX versus 68.19 for the baseline, a regression, and several other comparisons show gains of less than one point. The paper also reports no error bars, no multiple-seed variance, and no significance testing, so even the positive gaps on AitW and AMEX (mostly 0.5-4 percentage points) cannot be distinguished from noise. At minimum, the authors should rephrase the 'consistently' claim, report per-seed results or confidence intervals, and state which differences are statistically reliable.","section":"Abstract vs. Table 5 and Section 4.2.2"},{"comment":"The paper's own human evaluation of the MONDAY training set finds that only 80.40% of 250 sampled actions are accurate, with 8.60% rated 'not enough information' and the remainder inaccurate. This label noise rate is comparable to the action identification accuracy reported in Table 4 (80.90% All), so the abstract's characterization of the dataset as providing 'reliable action sequences' and being created 'without manual annotation' overstates label quality. The authors should report the incorrect/inconclusive breakdown explicitly in the main text, discuss the impact of roughly 20% label noise on downstream agent training, and temper the claim that the pipeline produces high-quality annotations without human intervention.","section":"Supplementary Section F"}],"minor_comments":[{"comment":"The caption says 'Please visit Section 4.1.2 for the details' but the UI element detection evaluation is described in Section 4.1.3; the cross-reference should be corrected.","section":"Table 3 caption"},{"comment":"Reference [20] gives the year of Levenshtein's paper as 1996; the correct year is 1966. Reference [11] is listed twice with different years (2023 and 2024); the duplicate entry should be removed or merged.","section":"References"},{"comment":"The third step of the action identification process is called 'refined action identification' and 'action refinement' in the text, but Figure 2 labels it 'Localized action identification'; the terminology should be made consistent.","section":"Figure 2 and Section 3.3.2"},{"comment":"The sentence 'Inter-annotator disagreement occurred in only 3.93% of actions' should clarify whether this is the percentage of action instances where the two annotators disagreed before the third-annotator adjudication, and should report agreement per annotation type if available.","section":"Section 4.1.1"},{"comment":"The unification of the action space restricts evaluation to touch, long press, and typing, covering 78.51% of AitW, 82.60% of MONDAY, and 94.39% of the Windows Mobile test set; this restriction should be stated in the main text alongside the Windows Mobile results, since the comparison is not over the full action space.","section":"Supplementary Section G.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset and pipeline are potentially valuable and the component evaluations are well executed, but the central 'unseen OS' generalization claim currently depends on a test set whose frame selection is generated by the same automatic pipeline as the training data. The authors can address this by re-annotating the Windows Mobile test with human-defined scene boundaries or otherwise demonstrating that the pipeline's frame sampling does not drive the gains; until then the headline number should not be presented as clean cross-platform generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things you should know about this paper. First, the MONDAY dataset is a real contribution: 313K frames from 20K YouTube tutorials, spanning iOS and Android, produced by an automated pipeline that costs about $0.34 per video. That is a clear step up from Video2Action's 6K videos and Chen et al.'s 128. Second, the headline 18.11pp gain on an \"unseen\" OS is not as clean as the abstract implies, because the Windows Mobile test frames and scene boundaries come from the same OCR-based scene detector used to build MONDAY. That shared segmentation is the real soft spot.\n\nWhat is genuinely good: the pipeline is sensible, and the component evaluations are solid. OCR-based scene detection beats YUV-diff and SceneDetect by a wide margin on a manually annotated 100-video set. UI element detection (99.87% hit ratio) and the 3-step action identification with ablations are convincing. They also ran a human check on training samples (80.4% accurate), which is more honesty than many dataset papers show.\n\nNow the soft spots. Supplementary Section D reveals that action annotation in the evaluation sets \"uses our scene transition detection output.\" That means the Windows Mobile test episodes are defined by the same automatic segmentation the model saw in MONDAY training. A model that has learned pipeline-specific scene boundaries will look better on that test than on a truly independent Windows Mobile benchmark. Independent evidence on AitW and AMEX is much weaker: most gains are 0.5–4pp, and there's a regression on AMEX for one configuration. No error bars are reported, and the Windows Mobile set is only 605 frames, so the 18.11pp average could easily be noisy.\n\nNone of this kills the paper. The dataset is valuable, and the framework is a meaningful advance. But the generalization claim should be re-framed or re-validated with an independently segmented test set, or at least reported with the caveat and per-video variance.\n\nWho is this for? Researchers working on mobile GUI agents, automated data collection, and cross-platform generalization. It deserves a serious referee; the dataset and pipeline merit review even if the evaluation section needs heavy revision. My recommendation: send it to review, but flag the Windows Mobile test-set dependence as a major revision.","headline":"MONDAY is a genuinely useful large-scale dataset and pipeline, but the headline 18.11pp 'unseen OS' gain is weakened by a Windows Mobile test set annotated using the same scene detector as the training data.","tokens_in":21145,"tokens_out":2745,"would_cite":true,"duration_ms":28849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that 20,000 YouTube instructional videos can be turned into a 313K-frame mobile navigation dataset that improves cross-platform agent performance by 18.11 percentage points on an unseen OS.","keywords":["mobile OS navigation","GUI agents","instructional videos","automated dataset generation","OCR scene detection","UI element detection","action identification","cross-platform generalization"],"falsifier":"Have independent annotators re-label the Windows Mobile videos' scene boundaries and action boxes from scratch, without seeing MONDAY's OCR transition or numbered-element outputs, then rerun the finetuned-model comparisons; if the MONDAY-induced advantage over base models shrinks to near zero or reverses, the cross-platform generalization claim does not survive.","tokens_in":20169,"feed_emoji":"📱","tokens_out":8691,"duration_ms":82247,"temperature":0.7,"pith_summary":"This paper introduces MONDAY, a 313K-frame dataset of mobile navigation actions automatically extracted from 20,000 instructional videos, together with a pipeline that detects scene changes by OCR, finds UI elements, and labels actions through a three-stage vision-language reasoning process. The paper's central claim is that adding MONDAY to a model's pre-training improves cross-platform generalization: models pre-trained with it outperform models trained on existing single-OS datasets and gain an average of 18.11 percentage points on an unseen mobile OS. A sympathetic reader should care because if this holds, large training corpora for GUI agents can be built from public videos without manual annotation or platform-specific simulators, and the same approach can keep datasets current as operating systems change.","feed_headline":"YouTube video mining beats single-OS datasets on unseen platforms","feed_subtitle":"A 313K-frame automatically annotated dataset lifts navigation accuracy by 18.11 percentage points on Windows Mobile.","key_machinery":"The load-bearing mechanism is the automated video-to-dataset pipeline. It uses OCR text tracking with a string-edit distance to cut videos at meaningful screen changes, a grounded open-set object detector combined with mobile-specific heuristics to mark interactive UI elements, and a three-step action identification stage in which a large multimodal model summarizes each frame, proposes actions using numbered overlays on UI elements together with narration and neighboring frames, then refines the chosen element's location on a zoomed view. This pipeline converts raw footage into task-screenshot-action examples, and the accuracy of each stage is what makes the downstream transfer claim credible.","core_discovery":"The paper claims that instructional videos of mobile screens contain enough structured navigation signal that an automated pipeline can convert them into a large, diverse action dataset, and that exposing a vision-language model to that dataset during pre-training teaches platform-agnostic navigation that transfers to an operating system never seen in training. Specifically, pre-training two existing vision-language agents, SeeClick and Llama-3.2, on MONDAY before finetuning on AitW or AMEX gives an average gain of 18.11 percentage points over the base agents on a Windows Mobile test set, while mostly improving or matching accuracy on the existing Android benchmarks. The claim stands or falls on whether the automatic extraction is faithful, since MONDAY is generated entirely by OCR scene segmentation, detector-based UI element identification, and a three-step action labeling process.","pith_inferences":["An implication the paper leaves implicit: if the reported 18.11 percentage-point gain is real, it should scale with corpus diversity, so adding more languages, device types, and app categories to MONDAY would directly test whether the transfer mechanism is genuine platform-agnostic navigation rather than memorized layouts.","The paper's own human evaluation of 100 random MONDAY sequences found 80.4% of sampled actions accurate, meaning roughly one in five automatically labeled actions is wrong; a direct test of whether this noise matters would be to train on a cleaned subset and compare downstream accuracy.","Because the Windows Mobile test set was annotated with the same protocol and the same automatic scene-transition output as the MONDAY training data, an independent human re-annotation of those test videos would be needed to confirm that the headline gain reflects true generalization rather than shared segmentation biases."],"forward_implications":["Pre-training with MONDAY and then finetuning on AitW or AMEX mostly outperforms the same models finetuned without MONDAY on those test sets, so the benefit is not confined to MONDAY's own test distribution.","At roughly $0.34 per video versus $5.76 for expert annotation, the pipeline makes continuous dataset refresh feasible as mobile OS versions and UI patterns change.","If the unseen-OS result holds, generalist mobile agents could be built from one diverse corpus rather than per-OS demonstration collections.","The modular design means future improvements to scene detection, UI detection, or the language model can be substituted into the same pipeline to produce better datasets."],"supporting_citations":[{"why":"Defines the main Android benchmark and the exact action-matching evaluation protocol used for all test sets.","marker":"[41]"},{"why":"Supplies the second Android dataset used as a finetuning target and evaluation test set.","marker":"[4]"},{"why":"The prior semi-automatic video-to-action method whose YUV-diff scene detection serves as a baseline for scene transitions.","marker":"[13]"},{"why":"Base GUI-grounding model that is fine-tuned on MONDAY to create SeeClick-MONDAY.","marker":"[8]"},{"why":"Base vision-language model that is fine-tuned on MONDAY to create Llama-3.2-MONDAY.","marker":"[31]"},{"why":"Open-set object detector used for both phone-screen isolation and UI element detection in the pipeline.","marker":"[26]"},{"why":"Large multimodal model that performs scene summaries and the three-step action identification.","marker":"[37]"},{"why":"Recent GUI parser used as the comparison baseline in UI element detection hit-ratio evaluation.","marker":"[28]"},{"why":"Parameter-efficient finetuning method used to create the MONDAY pre-trained variants and downstream finetuned models.","marker":"[15]"}],"fun_headline_variants":["YouTube videos teach mobile agents unseen OS tasks","Auto-scraped video dataset boosts agent accuracy by 18%","From YouTube to new OS: 313K frames for agent training","Video mining yields cross-platform agents, 18% gain on new OS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gain on the unseen Windows Mobile platform is interpreted as cross-platform generalization, and that interpretation assumes the Windows Mobile test set was annotated independently of MONDAY's automatic pipeline, yet the paper describes it as created with the same annotation protocol and the same scene-transition output used for the training data.","fun_headline_variants_meta":{"raw":{"variants":["YouTube videos teach mobile agents unseen OS tasks","Auto-scraped video dataset boosts agent accuracy by 18%","From YouTube to new OS: 313K frames for agent training","Video mining yields cross-platform agents, 18% gain on new OS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1576,"prompt_tokens":891,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":507,"tokens_out":685,"duration_ms":6470,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:29:48.744067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators re-label the Windows Mobile videos' scene boundaries and action boxes from scratch, without seeing MONDAY's OCR transition or numbered-element outputs, then rerun the finetuned-model comparisons; if the MONDAY-induced advantage over base models shrinks to near zero or reverses, the cross-platform generalization claim does not survive.","supporting_citations":[{"cited_title":"AndroidInTheWild: A Large- Scale Dataset For Android Device Control","cited_arxiv_id":null,"evidence_quote":"Defines the main Android benchmark and the exact action-matching evaluation protocol used for all test sets."},{"cited_title":"Video2Action: Reducing Human Interactions in Action An- notation of App Tutorial Videos","cited_arxiv_id":null,"evidence_quote":"The prior semi-automatic video-to-action method whose YUV-diff scene detection serves as a baseline for scene transitions."},{"cited_title":"SeeClick: Har- nessing GUI Grounding for Advanced Visual GUI Agents","cited_arxiv_id":null,"evidence_quote":"Base GUI-grounding model that is fine-tuned on MONDAY to create SeeClick-MONDAY."},{"cited_title":"Llama-3.2","cited_arxiv_id":null,"evidence_quote":"Base vision-language model that is fine-tuned on MONDAY to create Llama-3.2-MONDAY."},{"cited_title":"Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection","cited_arxiv_id":null,"evidence_quote":"Open-set object detector used for both phone-screen isolation and UI element detection in the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large multimodal model that performs scene summaries and the three-step action identification."},{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"Parameter-efficient finetuning method used to create the MONDAY pre-trained variants and downstream finetuned models."}],"review_version":1}