Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that 20,000 YouTube instructional videos can be turned into a 313K-frame mobile navigation dataset that improves cross-platform agent performance by 18.11 percentage points on an unseen OS.

desk verdict MONDAY is a genuinely useful large-scale dataset and pipeline, but the headline 18.11pp 'unseen OS' gain is weakened by a Windows Mobile test set annotated using the same scene detector as the training data. read the letter →

arxiv 2505.12632 v1 pith:GJ3MKMRU submitted 2025-05-19 cs.CV cs.AIcs.CLcs.LG

Yunseok Jang , Yeda Song , Sungryull Sohn , Lajanugen Logeswaran , Tiange Luo , Dong-Ki Kim , Kyunghoon Bae , Honglak Lee This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords mobileOSnavigationGUIagentsinstructionalvideosautomateddatasetgenerationOCRscenedetectionUIelementactionidentificationcross-platformgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MONDAY, a 313K-frame dataset of mobile navigation actions automatically extracted from 20,000 instructional videos, together with a pipeline that detects scene changes by OCR, finds UI elements, and labels actions through a three-stage vision-language reasoning process. The paper's central claim is that adding MONDAY to a model's pre-training improves cross-platform generalization: models pre-trained with it outperform models trained on existing single-OS datasets and gain an average of 18.11 percentage points on an unseen mobile OS. A sympathetic reader should care because if this holds, large training corpora for GUI agents can be built from public videos without manual annotation or platform-specific simulators, and the same approach can keep datasets current as operating systems change.

What carries the argument

The load-bearing mechanism is the automated video-to-dataset pipeline. It uses OCR text tracking with a string-edit distance to cut videos at meaningful screen changes, a grounded open-set object detector combined with mobile-specific heuristics to mark interactive UI elements, and a three-step action identification stage in which a large multimodal model summarizes each frame, proposes actions using numbered overlays on UI elements together with narration and neighboring frames, then refines the chosen element's location on a zoomed view. This pipeline converts raw footage into task-screenshot-action examples, and the accuracy of each stage is what makes the downstream transfer claim credible.

What would settle it

Have independent annotators re-label the Windows Mobile videos' scene boundaries and action boxes from scratch, without seeing MONDAY's OCR transition or numbered-element outputs, then rerun the finetuned-model comparisons; if the MONDAY-induced advantage over base models shrinks to near zero or reverses, the cross-platform generalization claim does not survive.

Watch

Extended reading notes

Core claim

The paper claims that instructional videos of mobile screens contain enough structured navigation signal that an automated pipeline can convert them into a large, diverse action dataset, and that exposing a vision-language model to that dataset during pre-training teaches platform-agnostic navigation that transfers to an operating system never seen in training. Specifically, pre-training two existing vision-language agents, SeeClick and Llama-3.2, on MONDAY before finetuning on AitW or AMEX gives an average gain of 18.11 percentage points over the base agents on a Windows Mobile test set, while mostly improving or matching accuracy on the existing Android benchmarks. The claim stands or falls on whether the automatic extraction is faithful, since MONDAY is generated entirely by OCR scene segmentation, detector-based UI element identification, and a three-step action labeling process.

Load-bearing premise

The headline gain on the unseen Windows Mobile platform is interpreted as cross-platform generalization, and that interpretation assumes the Windows Mobile test set was annotated independently of MONDAY's automatic pipeline, yet the paper describes it as created with the same annotation protocol and the same scene-transition output used for the training data.

Editorial extensions

If this is right

  • Pre-training with MONDAY and then finetuning on AitW or AMEX mostly outperforms the same models finetuned without MONDAY on those test sets, so the benefit is not confined to MONDAY's own test distribution.
  • At roughly $0.34 per video versus $5.76 for expert annotation, the pipeline makes continuous dataset refresh feasible as mobile OS versions and UI patterns change.
  • If the unseen-OS result holds, generalist mobile agents could be built from one diverse corpus rather than per-OS demonstration collections.
  • The modular design means future improvements to scene detection, UI detection, or the language model can be substituted into the same pipeline to produce better datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: if the reported 18.11 percentage-point gain is real, it should scale with corpus diversity, so adding more languages, device types, and app categories to MONDAY would directly test whether the transfer mechanism is genuine platform-agnostic navigation rather than memorized layouts.
  • The paper's own human evaluation of 100 random MONDAY sequences found 80.4% of sampled actions accurate, meaning roughly one in five automatically labeled actions is wrong; a direct test of whether this noise matters would be to train on a cleaned subset and compare downstream accuracy.
  • Because the Windows Mobile test set was annotated with the same protocol and the same automatic scene-transition output as the MONDAY training data, an independent human re-annotation of those test videos would be needed to confirm that the headline gain reflects true generalization rather than shared segmentation biases.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MONDAY, a dataset of 313K annotated frames from 20K YouTube instructional videos covering iOS and Android navigation, together with an automated pipeline that uses OCR-based scene transition detection, GroundingDINO-based UI element detection, and a three-step GPT-4o action identification process to generate task episodes without manual annotation. Component-level evaluations report 95.04% F1 for scene detection, 99.87% hit ratio for UI element detection, and 80.90% action identification accuracy. The authors then fine-tune SeeClick and Llama-3.2 with LoRA on MONDAY and evaluate on AitW, AMEX, MONDAY, and a Windows Mobile test set, claiming an average 18.11 percentage-point gain on the unseen Windows Mobile platform and robust cross-platform generalization.

Significance. If the central claims hold, the work would be a valuable contribution: it proposes a scalable, low-cost alternative to manual dataset construction for mobile GUI agents, releases a large dataset and code, and reports strong component-level results with a clear automated pipeline. The claim that YouTube-sourced data transfers to a never-seen OS is ambitious and practically important. However, the headline cross-platform result currently rests on a Windows Mobile test set whose frames are selected by the same automatic scene-transition detector used to build MONDAY training episodes, and the supporting independent evidence on AitW and AMEX is mixed and without uncertainty quantification. The dataset release and component evaluations are credible strengths, but the generalization claim needs stronger, independent validation.

major comments (3)
  1. [Section 4.2.1 and Supplementary D] The Windows Mobile evaluation is not an independent out-of-distribution test. Section 4.2.1 states the Windows Mobile test set was annotated 'following the same annotation protocol' as the main evaluation set, and Supplementary Section D reveals that this protocol uses 'our scene transition detection output' to define the frames and episodes that annotators label. Because MONDAY training frames are also extracted using this same OCR-based transition detector with the midpoint sampling rule, the MONDAY-finetuned models are evaluated on frames drawn from the same automatic segmentation distribution they were trained on, while the AitW/AMEX-only baselines see that distribution only at test time. The reported +13 to +23 percentage-point gains on Windows Mobile could therefore reflect adaptation to pipeline-specific segmentation regularities rather than transfer to Windows Mobile as an operating system. To support the headline 'unseen OS' claim, the authors should either re-annotate the Windows Mobile test set using human-defined scene boundaries and frame selection independent of the MONDAY pipeline, or provide evidence (e.g., evaluating on all frames rather than pipeline-selected midpoints) that the frame-sampling choice does not affect the comparison.
  2. [Abstract vs. Table 5 and Section 4.2.2] The abstract's claim that MONDAY-based models 'consistently outperform' models trained on existing single-OS datasets is directly contradicted by Table 5: in the AMEX-finetuned SeeClick row, SeeClick-MONDAY scores 66.13 on AMEX versus 68.19 for the baseline, a regression, and several other comparisons show gains of less than one point. The paper also reports no error bars, no multiple-seed variance, and no significance testing, so even the positive gaps on AitW and AMEX (mostly 0.5-4 percentage points) cannot be distinguished from noise. At minimum, the authors should rephrase the 'consistently' claim, report per-seed results or confidence intervals, and state which differences are statistically reliable.
  3. [Supplementary Section F] The paper's own human evaluation of the MONDAY training set finds that only 80.40% of 250 sampled actions are accurate, with 8.60% rated 'not enough information' and the remainder inaccurate. This label noise rate is comparable to the action identification accuracy reported in Table 4 (80.90% All), so the abstract's characterization of the dataset as providing 'reliable action sequences' and being created 'without manual annotation' overstates label quality. The authors should report the incorrect/inconclusive breakdown explicitly in the main text, discuss the impact of roughly 20% label noise on downstream agent training, and temper the claim that the pipeline produces high-quality annotations without human intervention.
minor comments (5)
  1. [Table 3 caption] The caption says 'Please visit Section 4.1.2 for the details' but the UI element detection evaluation is described in Section 4.1.3; the cross-reference should be corrected.
  2. [References] Reference [20] gives the year of Levenshtein's paper as 1996; the correct year is 1966. Reference [11] is listed twice with different years (2023 and 2024); the duplicate entry should be removed or merged.
  3. [Figure 2 and Section 3.3.2] The third step of the action identification process is called 'refined action identification' and 'action refinement' in the text, but Figure 2 labels it 'Localized action identification'; the terminology should be made consistent.
  4. [Section 4.1.1] The sentence 'Inter-annotator disagreement occurred in only 3.93% of actions' should clarify whether this is the percentage of action instances where the two annotators disagreed before the third-annotator adjudication, and should report agreement per annotation type if available.
  5. [Supplementary Section G.2] The unification of the action space restricts evaluation to touch, long press, and typing, covering 78.51% of AitW, 82.60% of MONDAY, and 94.39% of the Windows Mobile test set; this restriction should be stated in the main text alongside the Windows Mobile results, since the comparison is not over the full action space.

Circularity Check

1 steps flagged · score 5.0 of 10

Headline cross-platform generalization is evaluated on a Windows Mobile test set generated with the same OCR scene-transition pipeline that builds MONDAY, so the 18.11pp gain is partly a pipeline-overlap artifact rather than clean out-of-distribution transfer.

  1. other [Supplementary Section D (Annotation of the Evaluation Dataset); Section 4.2.1]
    "Following the same annotation protocol used in the previous section, we manually annotated 50 videos containing 605 valid frames and 554 actions. / Using our scene transition detection output, annotators labeled actions between consecutive scenes using Label Studio with a custom interface."

    MONDAY training episodes are produced by the automated framework whose scene boundaries come from the OCR-based transition detector, with action labels inserted between consecutive detected scenes. The Windows Mobile test set is created following the same annotation protocol, and Supplementary Section D confirms that protocol uses the paper's own scene transition detection output to define the scenes that annotators label. Thus the training episodes and the supposedly 'unseen platform' test episodes share the same segmentation distribution: both are frame sequences selected at the midpoint of the same OCR detector's transitions.

full rationale

The dataset-construction pipeline itself is not circular: action labels are human-verified, UI detection is checked against a manual evaluation set, and scene detection is benchmarked on manually marked transitions. Nor is there load-bearing self-citation: the cited prior works by the authors are only related-work context. The circularity lies in the evaluation design for the headline claim. Section 4.2.1 says the Windows Mobile test was annotated following the same annotation protocol, and Supplementary Section D reveals that protocol includes using the paper's scene transition detection output to define the frames to be labeled. Since MONDAY's training episodes are also defined by that same OCR scene-transition output, the 'unseen OS' test is not independent of the framework's generative process. The large average gain (+13 to +23pp on Windows Mobile) is therefore attributable, at least in part, to the model having learned pipeline-specific segmentation behavior rather than solely to transferable mobile-OS knowledge. Independent evidence on AitW and AMEX is much smaller and mixed, including one regression on AMEX (66.13 vs 68.19), so the central generalization claim rests on this partially circular benchmark.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of off-the-shelf components (Paddle OCR, GroundingDINO, MediaPipe, GPT-4o) and several hand-set thresholds. No first-principles derivation is offered. The evaluation assumes the small human test sets are representative and that the Windows Mobile test set is independent, which is questionable given it is annotated with the same scene detection pipeline.

free parameters (5)
  • Scene transition text-change threshold = 20%
    Empirically set through preliminary experiments (Section 3.2); determines which text changes count as scene transitions.
  • GroundingDINO phone screen confidence thresholds = box 0.25, text 0.25
    Set for phone screen detection (Supplementary C.1); affects which frames keep the phone bounding box.
  • GroundingDINO icon detection thresholds = box 0.04, text 0.25
    Deliberately relaxed to maximize coverage (Supplementary C.2), then filtered by heuristics.
  • OCR confidence filter = >0.9
    Filters unreliable OCR text (Supplementary C.1).
  • Zone boundaries for refined action localization = five vertical zones from 0-45%, 12.5-57.5%, 25-70%, 37.5-82.5%, 55-100% of screen height
    Hand-defined zones used in the final action localization step (Supplementary C.3).
assumptions (4)
  • domain assumption OCR text changes reliably indicate meaningful scene transitions on mobile OS screens
    The scene detection method is built on this; no theoretical proof, only empirical F1 on 100 videos.
  • domain assumption GroundingDINO-based phone screen detection and MediaPipe hand filtering produce clean, unobstructed phone screens
    Pipeline relies on these detectors being accurate across illumination and device variations; failure would introduce noise.
  • domain assumption GPT-4o provides sufficiently accurate action judgments for the 3-step annotation
    Automatic annotation is driven by GPT-4o; its reliability is only sampled on 100 videos, with 80.9% action accuracy.
  • domain assumption The 100-video human-annotated evaluation set is representative of the 20K-video dataset
    All F1, hit ratio, and action accuracy numbers come from this small test set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents." pith.science (2026). https://pith.science/paper/GJ3MKMRU

@misc{pith2026250512632,
  author       = {Pith},
  title        = {Pith review of: Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJ3MKMRU}},
  note         = {Machine review of arXiv:2505.12632}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have sparked significant interest in developing GUI visual agents. We introduce MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube), a large-scale dataset of 313K annotated frames from 20K instructional videos capturing diverse real-world mobile OS navigation across multiple platforms. Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities, consistently outperforming models trained on existing single OS datasets while achieving an average performance gain of 18.11%p on an unseen mobile OS platform. To enable continuous dataset expansion as mobile platforms evolve, we present an automated framework that leverages publicly available video content to create comprehensive task datasets without manual annotation. Our framework comprises robust OCR-based scene detection (95.04% F1score), near-perfect UI element detection (99.87% hit ratio), and novel multi-step action identification to extract reliable action sequences across diverse interface configurations. We contribute both the MONDAY dataset and our automated collection framework to facilitate future research in mobile OS navigation.

Figures

Figures reproduced from arXiv: 2505.12632 by the authors.

Figure 1
Figure 1. Example screens from our MONDAY dataset: (a) mobile [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Extracted position of the frame from each scene transi [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Comparison between our UI element detection module [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Identified actions between different ablation methods. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 44 canonical work pages

  1. [1]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966, 2023. 8

    arXiv 2023

  2. [2]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    work page 2020

  3. [3]

    PySceneDetect

    Brandon Castellano. PySceneDetect. https://www.scenedetect.com/, 2024. (accessed Sep., 2024). 5, 6

    work page 2024

  4. [4]

    AMEX: Android Multi-annotation Expo Dataset for Mo- bile GUI Agents

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. AMEX: Android Multi-annotation Expo Dataset for Mo- bile GUI Agents. arXiv preprint arXiv:2407.17490, 2024. 2, 7

    arXiv 2024

  5. [5]

    GUI-WORLD: A Dataset for GUI- oriented Multimodal LLM-based Agents

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Li- uyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. GUI-WORLD: A Dataset for GUI- oriented Multimodal LLM-based Agents. arXiv preprint arXiv:2406.10819, 2024. 7

    arXiv 2024

  6. [6]

    Extracting Replayable Interactions from Videos of Mobile App Usage

    Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. Extracting Re- playable Interactions from Videos of Mobile App Usage. arXiv:2207.04165, 2022. 2, 3, 7

    work page Pith review arXiv 2022

  7. [7]

    Towards Complete Icon Labeling in Mobile Applications

    Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. Towards Complete Icon Labeling in Mobile Applications. In CHI, 2022. 5

    work page 2022

  8. [8]

    SeeClick: Har- nessing GUI Grounding for Advanced Visual GUI Agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yan- tao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Har- nessing GUI Grounding for Advanced Visual GUI Agents. In ACL, 2024. 1, 2, 8, 5

    work page 2024

Show all 63 references
  1. [9]

    RICO: A Mobile App Dataset for Building Data-Driven Design Applications

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hib- schman, Daniel Afergan, Yang Li, Ranjitha Kumar, and Jef- frey Nichols. RICO: A Mobile App Dataset for Building Data-Driven Design Applications. In UIST, 2017. 1, 2, 3

    2017

  2. [10]

    Mind2Web: Towards a Generalist Agent for the Web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In NeurIPS, 2023. 7

    2023

  3. [11]

    Mind2Web: Towards a Generalist Agent for the Web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In NeurIPS, 2024. 2

    2024

  4. [12]

    BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding. In NAACL, 2019. 1, 2

    2019

  5. [13]

    Video2Action: Reducing Human Interactions in Action An- notation of App Tutorial Videos

    Sidong Feng, Chunyang Chen, and Zhenchang Xing. Video2Action: Reducing Human Interactions in Action An- notation of App Tutorial Videos. In UIST, 2023. 2, 3, 4, 5, 6, 7

    2023

  6. [14]

    Fouhey, Weicheng Kuo, Alexei A

    David F. Fouhey, Weicheng Kuo, Alexei A. Efros, and Jiten- dra Malik. From Lifestyle VLOGs to Everyday Interactions. In CVPR, 2018. 2

    2018

  7. [15]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022. 7, 5

    2022

  8. [16]

    Multimodal Subtask Graph Generation from Instructional Videos

    Yunseok Jang, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Moontae Lee, and Honglak Lee. Multimodal Subtask Graph Generation from Instructional Videos. In ICLRW-MRL, 2023. 2

    2023

  9. [17]

    VisualWe- bArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWe- bArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In ACL, 2024. 2

    2024

  10. [18]

    Tree Search for Language Model Agents

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree Search for Language Model Agents. arXiv:2407.01476, 2024. 2

    2024

  11. [19]

    Benchmarking Mobile Device Control Agents across Diverse Configurations

    Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking Mobile Device Control Agents across Diverse Configurations. In ICLRW, 2024. 1, 2

    2024

  12. [20]

    Binary Codes Capable of Correcting Deletions, Insertions, and Reversals

    Vladimir Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions, and Reversals. In Soviet Physics Dok- lady, 1996. 5, 2

    1996

  13. [21]

    PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System

    Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System. arXiv:2206.03001, 2022. 5, 6, 2

    2022 arXiv

  14. [22]

    On the Effects of Data Scale on UI Control Agents

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on UI Control Agents. In NeurIPS, 2024. 2

    2024

  15. [23]

    Mapping Natural Language Instructions to Mo- bile UI Action Sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping Natural Language Instructions to Mo- bile UI Action Sequences. In ACL, 2020. 4, 2

    2020

  16. [24]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. arXiv:2310.03744, 2023. 1

    2023 arXiv

  17. [25]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023. 1

    2023

  18. [26]

    Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In ECCV, 2024. 4, 5, 6, 2 9

    2024

  19. [27]

    Unsupervised Task Graph Generation from Instructional Video Transcripts

    Lajanugen Logeswaran, Sungryull Sohn, Yunseok Jang, Moontae Lee, and Honglak Lee. Unsupervised Task Graph Generation from Instructional Video Transcripts. InFindings of ACL, 2023. 2

    2023

  20. [28]

    OmniParser for Pure Vision Based GUI Agent

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203, 2024. 5, 6

    2024 arXiv

  21. [29]

    Me- diaPipe: A Framework for Building Perception Pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Me- diaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172, 2019. 4

    1906 arXiv

  22. [30]

    PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft , 2022. 5

    2022

  23. [31]

    Llama-3.2

    Meta AI. Llama-3.2. https://www.llama.com/ ,

  24. [32]

    HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV,

  25. [33]

    ScreenA- gent: A Vision Language Model-driven Computer Control Agent

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. ScreenA- gent: A Vision Language Model-driven Computer Control Agent. arXiv:2402.07945, 2024. 2

    2024 arXiv

  26. [34]

    Gpt-4 Technical Report

    OpenAI. Gpt-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. 2

    2023 arXiv

  27. [35]

    GPT-4V Limitations

    OpenAI. GPT-4V Limitations. https : / / platform.openai.com/docs/guides/vision/ limitations, 2023. Large Language/Vision Model. 5

    2023

  28. [36]

    GPT-3.5 Instruct

    OpenAI. GPT-3.5 Instruct. https : / / platform . openai.com/docs/models/gpt-3-5 , 2023. Large Language Model. 4, 2

    2023

  29. [37]

    OpenAI. GPT-4o. https://platform.openai.com/ docs/models/gpt-4o , 2024. Large Language Model. 4, 5, 8

    2024

  30. [38]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike,...

    2022

  31. [39]

    Language Models are Unsuper- vised Multitask Learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsuper- vised Multitask Learners. OpenAI Blog, 2019. 1, 2

    2019

  32. [40]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 2020. 3, 2

    2020

  33. [41]

    AndroidInTheWild: A Large- Scale Dataset For Android Device Control

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. AndroidInTheWild: A Large- Scale Dataset For Android Device Control. In NeurIPS Dataset, 2023. 1, 2, 3, 7, 8

    2023

  34. [42]

    As- sembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. As- sembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities. In CVPR, 2022. 2

    2022

  35. [43]

    LUSE: Using LLMs for Unsupervised Step Extraction in Instructional Videos

    Chuyi Shang, Emi Tran, Medhini Narasimhan, Sanjay Sub- ramanian, Dan Klein, and Trevor Darrell. LUSE: Using LLMs for Unsupervised Step Extraction in Instructional Videos. In ICCVW, 2023. 2

    2023

  36. [44]

    World of Bits: An Open-Domain Plat- form for Web-Based Agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernan- dez, and Percy Liang. World of Bits: An Open-Domain Plat- form for Web-Based Agents. In Proceedings of the 34th In- ternational Conference on Machine Learning , pages 3135–

  37. [45]

    AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Rein- forcement Learning

    Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mo- homed, Allan D Jepson, and Sheila A McIlraith. AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Rein- forcement Learning. In Canadian AI, 2021. 1, 2

    2021

  38. [46]

    Peters, Abhilasha Ravichander, Kyle Richard- son, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Mor- rison, Niklas ...

    2024 arXiv

  39. [47]

    Towards Better Semantic Understanding of Mobile Interfaces

    Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baech- ler, Yu-Chung Hsiao, Jindong, Chen, Abhanshu Sharma, and James Stout. Towards Better Semantic Understanding of Mobile Interfaces. In COLING, 2022. 5

    2022

  40. [48]

    COIN: A Large-scale Dataset for Comprehensive Instruc- tional Video Analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A Large-scale Dataset for Comprehensive Instruc- tional Video Analysis. In CVPR, 2019. 2

    2019

  41. [49]

    AndroidEnv: A Reinforce- ment Learning Platform for Android

    Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A Reinforce- ment Learning Platform for Android. arXiv:2105.13231,

    arXiv

  42. [50]

    Agent- Trek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

    Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agent- Trek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. arXiv preprint arXiv:2412.09605, 2024. 7

    2024 arXiv

  43. [51]

    GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Lin- jie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation. arXiv:2311.07562, 2023. 3

    2023 arXiv

  44. [52]

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441, 2023. 5

    2023 arXiv

  45. [53]

    Mobile-Env: Building Qual- 10 ified Evaluation Benchmarks for LLM-GUI Interaction

    Danyang Zhang, Hongshen Xu, Zihan Zhao, Lu Chen, Ruisheng Cao, and Kai Yu. Mobile-Env: Building Qual- 10 ified Evaluation Benchmarks for LLM-GUI Interaction. arXiv:2305.08144, 2023. 2

    2023 arXiv

  46. [54]

    You Only Look at Screens: Multimodal Chain-of-Action Agents

    Zhuosheng Zhang and Aston Zhang. You Only Look at Screens: Multimodal Chain-of-Action Agents. arXiv:2309.11436, 2023. 1

    2023 arXiv

  47. [55]

    GPT-4V(ision) is a Generalist Web Agent, if Grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a Generalist Web Agent, if Grounded. In ICML, 2024. 2

    2024

  48. [56]

    WebArena: A Realistic Web Envi- ronment for Building Autonomous Agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. WebArena: A Realistic Web Envi- ronment for Building Autonomous Agents. In ICLR, 2023. 2, 7

    2023

  49. [57]

    Scalable Video-to-Dataset Generation for Cross- Platform Mobile Agents

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross- Task Weakly Supervised Learning from Instructional Videos. In CVPR, 2019. 2 11 Supplementary Material of “Scalable Video-to-Dataset Generation for Cross- Platform ...

    2019

  50. [59]

    phone screen

    and Dolma [46] datasets. To ensure the mobile OS nav- igation topic, we first filter these posts using an expanded version of AndroidHowTo’s domain whitelist [23], which we augmented to include iOS-related websites alongside the original Android domains. We then employ GPT-3.5...

  51. [60]

    Scene Summary: First, we analyze each frame inde- pendently to understand the overall UI layout and compo- nent relationships, creating a comprehensive scene descrip- tion without any preconceptions about actions

  52. [61]

    Initial Action Identification: Using the scene sum- maries and temporal context from adjacent frames, we iden- tify potential actions that could lead to the observed state changes, considering both visible UI elements and narrative guidance

  53. [62]

    File: $image

    Refined Action Identification: Finally, we employ a zone-based system for precise spatial localization of the predicted action, dividing the screen into five vertical zones based on UI element positions. Zones are calculated as fol- lows: • Zone 1: 0.0 - 45.0% of screen height...

  54. [63]

    How to Delete A Direct Message on Twitter

    and Llama3.2 [31], named SeeClick-MONDAY and Llama3.2-MONDAY, by fine-tuning them on MONDAY . For SeeClick-MONDAY, we fine-tune SeeClick for 10 epochs using the AdamW optimizer (learning rate: 1e-5, co- sine decay, batch size: 16). The checkpoint from epoch 7 is selected. For ...

  55. [2024]

    Large Language/Vision Model. 8, 5

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.