{"id":"3ffad22b-62cb-4fdc-86dd-bdf9e00bbb42","arxiv_id":"2502.02982","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A hierarchical auto-annotation pipeline plus episode-aware federated aggregation lets mobile GUI agents be trained on automatically labeled user trajectories at about 1% of human annotation cost.","lead":"This paper presents a framework that automatically turns ordinary smartphone usage into training data for mobile GUI agents, then trains the agents collaboratively across users without sharing private screenshots. If it works in practice, it could cut data-collection costs to roughly one percent of human annotation while preserving user privacy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy half of the central claim is unsupported: FedVLM-A transmits LoRA updates with no DP or leakage analysis, and §E.2 concedes VLM memorization risk; Table 1's 'rigorous' privacy claim needs an attack-based test.","rationale":"The reader's weakest assumption was the absence of real-user phone evaluation, which the paper concedes in B.2. My concern is narrower and cuts deeper: even within the simulated federated setting, the privacy component of the central claim is not demonstrated. The performance/cost results are the paper's strongest contribution: Auto-Annotation's hierarchical descriptor-plus-summarizer outperforms simpler annotation baselines on data-quality and downstream metrics, the equal-data-size comparison in Table 4 shows parity with human annotation, and FedVLM-A beats seven FL baselines on non-IID splits. These are credible, reproducible-looking results. However, §E.2 explicitly acknowledges VLM memorization leakage and says DP is future work, which directly contradicts the 'rigorous user privacy protection' and 'strongest protection' language in §1, §3.3, and Table 1. Federated learning preserves data locality, not privacy: LoRA adapters and the aggregated global model can encode sensitive content. A single attack experiment on the actual pipeline would settle this. If the attack succeeds, the advertised benefit must be narrowed from 'privacy-preserving' to 'locally processed, with residual leakage risk'; the method may still be conditionally acceptable, but the privacy claim cannot stand as-is. This reinforces the reader's CONDITIONAL verdict rather than changing it, so I recommend keeping the verdict unchanged while adding the privacy attack as a condition.","tokens_in":31969,"tokens_out":13398,"duration_ms":229407,"concrete_test":"Use the released training code to run FedVLM-A for one or more rounds on a client whose AndroidControl episodes contain visible personal data, such as email addresses, contact names, or search terms. After training, apply the déjà vu memorization measure from ref. [59] to the aggregated global model, comparing how accurately the model can predict held-out training screenshots' sensitive text versus control data the model never saw. If the leakage fraction exceeds a pre-specified false-positive threshold, Table 1's privacy-protection claim is not met for the proposed protocol; if no leakage is found, the claim still requires a formal DP guarantee with a reported epsilon budget before it can be called 'rigorous.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"MobileA3gent's headline claim has two parts: beat centralized human-annotated training at roughly 1% cost and do so while preserving privacy. The first part is supported, within the paper's simulated benchmark setting, by the Auto-Annotation and FedVLM-A experiments. The second part is not. The protocol uploads LoRA adapters, and the paper's own §E.2 admits that 'potential privacy issues remain' because VLMs memorize training data and can expose sensitive information, with differential privacy 'underexplored' in this setting. Despite that, §1 promises 'rigorous user privacy protection' and Table 1 claims FedVLM-A offers the 'strongest protection' against eavesdropping, abuse, and peer exposure. Those three risk categories do not include model-inversion, membership inference, or memorization leakage through the transmitted adapters or the aggregated global model. No privacy experiment, no formal DP guarantee, and no leakage bound is provided anywhere in the main text or appendices. Keeping data on-device is necessary but not sufficient for privacy, so the central claim's privacy component is currently an assertion rather than an established property.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MobileA3gent, a two-component framework for training mobile GUI agents from decentralized user trajectories. Auto-Annotation converts raw screenshots and actions into low-level step descriptions and a high-level task instruction using a locally deployed VLM, while FedVLM-A trains VLM-based agents via federated learning with an adapted aggregation weight that combines episode counts and step counts. Experiments on AndroidControl, AitW, GUI Odyssey, and the online AndroidWorld benchmark compare against human annotation, synthetic data, and standard FL baselines, reporting that auto-annotated data are comparable to human-annotated data at much lower cost and that the adapted aggregation improves over FedAvg-family methods under non-IID splits. The paper also discusses privacy and limitations, including the absence of real-device experiments.","tokens_in":32200,"tokens_out":6554,"duration_ms":55291,"significance":"If the claims hold, MobileA3gent would be a practically valuable framework: it automates instruction annotation from interaction trajectories, makes a simple modification to FedAvg that can be dropped into existing federated VLM training stacks, and provides an unusually broad comparison across multiple benchmarks, annotation models, training models, and an online environment. Strengths include the equal-size comparison in Table 4, the model ablation in Section 4.5, the out-of-domain generalization splits in Appendix D.4, and the public code release. However, the headline claims about cost superiority and privacy go beyond what is demonstrated; the current evidence supports a more modest claim about simulated federated training with auto-annotated crowdsourced data.","major_comments":[{"comment":"The privacy component of the central claim is not substantiated. The protocol transmits LoRA adapters rather than raw data, but no differential-privacy guarantee, leakage bound, or attack-based evaluation (membership inference, model inversion, or memorization extraction) is provided. Section 3.3's Privacy Analysis addresses only eavesdropping, abuse, and peer exposure, while §E.2 concedes that VLMs memorize training data and that DP remains underexplored in this setting. Consequently, the abstract's and Section 1's 'rigorous user privacy protection' and Table 1's 'strongest protection' overstate what is demonstrated; either add a concrete privacy analysis with a threat model and experiments, or revise these claims to state that local data retention mitigates the three specified sharing risks.","section":"§3.3, Table 1, §E.2"},{"comment":"The claim that 'with 1% overall cost, MobileA3gent even surpasses the centralized human-annotated data' is not supported by the equal-size experiment. Table 3 compares Human-Annotation on 1,000 samples with Auto-Annotation on 5,000 samples for AndroidControl and 3,000 for GUI Odyssey; the equal-size comparison in Table 4 shows Auto-Annotation is comparable but often numerically lower than Human-Annotation on AndroidControl-High at 5,000 samples (Type 77.49 vs 79.14, Ground 62.67 vs 66.56, SR 58.12 vs 61.70). The superiority claim should be qualified as holding at larger data volumes, or reframed as comparable performance at roughly 1% annotation cost with further improvements as data scale increases.","section":"Table 2 caption, Table 3, Table 4"},{"comment":"The FedVLM-A improvement claim is only weakly tied to the proposed mechanism because the aggregation weight n*_k = λ·n_epi_k + n_k is defined in terms of the same episode and step counts used to create the skew, and λ is an empirically tuned hyperparameter taking values {3,5,7,9} in Table 12. The paper reports neither a λ-sensitivity analysis nor per-split, per-λ results, and the statement that the adapted aggregation 'inherits its convergence property' is asserted without proof or a supporting reference. Please report the λ used for each experimental setting, include a sensitivity sweep, and either provide a convergence argument for the adaptive weights or cite a result that covers them.","section":"§4.4, Eq. (4), Table 12"},{"comment":"The real-world framing is not validated by the experiments. All 'users' are simulated by partitioning crowdsourced benchmarks (AndroidControl, AitW, GUI Odyssey), and annotation is run on RTX 4090 GPUs rather than on mobile devices; §B.2 explicitly states that experiments on actual user phones were not conducted. The abstract's claims about users' routine phone usage, negligible per-client cost, and practical deployment therefore rest on the unvalidated assumption that on-device trajectory capture, accessibility-tree-based action conversion, and local annotation can be performed with comparable quality. A feasibility study, or a clearly stated scoping of the contributions to the simulated setting, is needed before the deployment-oriented claims can be accepted.","section":"§B.2, §4.1"}],"minor_comments":[{"comment":"The phrase 'without comprising privacy' should read 'without compromising privacy'.","section":"§3.3"},{"comment":"The label 'Executation LaborCost' appears to be a typo for 'Execution Labor Cost'.","section":"Figure 1"},{"comment":"The cost accounting is confusing: Table 8 reports per-1,000-sample costs while Table 2 reports an aggregate 'Anno. Cost' of 152.92 ¢; please state explicitly which dataset size and backend each aggregate cost corresponds to.","section":"Table 2, Table 8"},{"comment":"The Hugging Face URLs for Qwen2-VL-2B-Instruct and Qwen2-VL-7B-Instruct are identical; the 7B link should be corrected.","section":"Appendix F.4"},{"comment":"The annotation-cost formula is referenced as 'Equation 4.1' in Section D.5 but is not numbered in Section 4.1; please number the equation or fix the cross-reference.","section":"§4.1, §D.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's engineering contributions are substantial and the experimental breadth is commendable, but the abstract and introduction overclaim privacy protection and real-world deployment relative to the evidence. The authors should be asked to either add a privacy/attack evaluation and a feasibility analysis for on-device deployment, or to systematically temper the corresponding claims. The λ tuning issue for FedVLM-A also needs a sensitivity analysis and per-setting reporting. This is not a reject: the core components are reproducible, the equal-size comparison in Table 4 is a good-faith check, and the remaining issues appear addressable within a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper with a real contribution in the Auto-Annotation pipeline, but the abstract oversells the privacy half of the claim and the real-world deployment story is not tested. The controlled experiments are the strongest part: they hold trajectories fixed, vary only the annotation method, and show equivalent or better agent performance at roughly 1% of human annotation cost. The equal-size comparison in Table 4 is the right check, and it mostly holds up.\n\nWhat's new: the step-wise descriptor plus episode-wise summarizer is a clean combination that beats the individual baselines (Visual-Sense, Self-Instruct, CoT) on data quality and downstream agent metrics. Applying federated training to VLM-based GUI agents with an episode-aware reweighting term is new in this space, and the two-level heterogeneity formulation (episode length vs. step count) is a reasonable way to think about non-IID for trajectory data. The empirical comparison against seven FL baselines is useful. The code is public.\n\nSoft spots, in order. First, the privacy claim is not supported by evidence. The method keeps raw data on device and transmits LoRA adapters, but there is no DP guarantee, no membership-inference or model-inversion attack, and no leakage analysis. The appendix (E.2) actually concedes that VLMs memorize training data and DP remains underexplored. So the abstract's 'rigorous user privacy protection' and Table 1's 'strongest protection' are assertions, not results. That needs either an attack-based evaluation or a softer claim. Second, the real-world scalability story rests on simulation. All experiments partition crowdsourced benchmarks into clients; no one runs on actual phones, and annotation is on GPUs, not on-device. The authors say this in B.2, but the intro and conclusion read as if deployment is demonstrated. Third, the FedVLM-A weighting is a small modification of FedAvg (Eq. 4) with λ tuned per scenario (3,5,7,9). The improvement over FedAvg is real in their setup, but I would not call it 'novel aggregation' without more sensitivity analysis. Minor: no error bars across runs, and the headline Table 3 mixes human 1k vs model 5k, though Table 4 fixes that.\n\nBottom line: the paper is worth a serious referee. The annotation result is the most valuable piece; the federated part is a modest but useful increment. The framing overreaches, but the work itself is honest about its limitations in the appendices. I'd push for a careful revision that either provides privacy experiments or rewrites the privacy claims, and I'd ask for on-device pilot data or at least a feasibility analysis. For a reader in the mobile-agent or federated-VLM area, this is a useful read and cite.","headline":"Solid annotation pipeline and careful benchmark experiments, but privacy and real-world deployment claims outrun the evidence; worth a serious referee.","tokens_in":32727,"tokens_out":2273,"would_cite":true,"duration_ms":185948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that mobile GUI agents can be trained from decentralized, auto-annotated user trajectories at about 1% of the cost of human-annotated centralized data, without sacrificing performance or privacy.","keywords":["mobile GUI agents","federated learning","vision-language models","automatic data annotation","user trajectories","non-IID data","instruction generation","mobile benchmarks"],"falsifier":"Run the MobileA3gent pipeline on real user devices or a high-fidelity on-device logging study, then compare the trained agent's step success rate against the paper's simulated results on AndroidControl and AndroidWorld; the central claim would be falsified if real-world trajectories yield materially worse instruction quality or agent performance than the crowdsourced partitions, or if on-device annotation with small local VLMs cannot reproduce the reported ~99% cost reduction without a large drop in quality.","tokens_in":31771,"feed_emoji":"📱","tokens_out":7081,"duration_ms":66454,"temperature":0.7,"pith_summary":"The paper is trying to establish that the expensive, centralized, human-annotated datasets used to train mobile GUI agents can be replaced by a decentralized pipeline fed by ordinary users' daily phone interactions. It proposes a framework that automatically reconstructs what a user was trying to do from screenshots and touch actions, then trains the agent across many users without moving raw data off their devices. The central claim is that this pipeline produces instruction data of human-comparable quality, and that federated training with a custom aggregation rule can turn that data into agents matching or beating central human-annotation baselines. If true, it would make mobile agent training scale with the billions of active phone users at a fraction of current cost, and it would keep sensitive interaction data local. The experiments reported support this claim on three offline benchmarks and one online benchmark, at roughly 1% of the overall cost of the human-annotated baseline.","feed_headline":"Auto-annotated phone data trains GUI agents at 1% of the cost","feed_subtitle":"Everyday taps become training data without uploading sensitive screenshots, matching human-annotated quality at ~1% cost.","key_machinery":"The load-bearing mechanism is a two-stage annotation-and-training loop. Auto-Annotation first converts coordinate actions into readable action descriptions using the accessibility tree, then a local Descriptor VLM writes an atomic low-level instruction for each step, and a local Summarizer VLM, fed the concatenated screenshot strip plus all low-level instructions, writes the episode's high-level instruction. FedVLM-A then trains a LoRA-adapted VLM on each client and aggregates only the model updates; its adapted aggregation redefines each client's sample count as $n_k^* = \\lambda n^{\\mathrm{epi}}_k + n_k$, with $\\lambda$ chosen near the average episode length, and weights the global model update by $\\omega_k = n_k^* / \\sum_j n_j^*$. This balances episode-level and step-level heterogeneity, degenerating to ordinary sample-count aggregation when $\\lambda = 0$.","core_discovery":"On the paper's own terms, the central discovery is that users' routine phone usage can be harvested as a training signal: converting raw actions into textual descriptions, generating per-step low-level instructions, and then summarizing those into a high-level instruction yields annotations nearly 80% similar to human-written ground truth. When a VLM-based mobile agent is trained on these auto-annotated episodes, it matches or exceeds the performance of the same model trained on human-annotated episodes, and with 1% of the overall cost the full MobileA3gent pipeline surpasses the centralized human-annotated result on AndroidControl. The second half of the discovery is that federated training of such agents benefits from an aggregation weight that counts both episodes and steps per client, rather than only raw samples, because mobile interaction data has a two-level structure with homogeneity inside each task and heterogeneity across tasks.","pith_inferences":["If real-device capture matches the simulations, the same pipeline could support continual personalization, with agents improving on the habits and task distributions of their own users while still learning from the global population.","The episode-plus-step aggregation idea likely transfers to other hierarchically structured federated datasets, such as multi-turn conversations or logged sessions, where raw sample counts hide task-level skew.","The two-stage Descriptor/Summarizer design may be reusable outside mobile GUIs as a general method for turning interaction traces into natural-language task descriptions, for example in web automation or desktop assistants.","A deployment would still face a privacy gap the paper acknowledges only in passing: locally generated instructions and uploaded LoRA updates can encode sensitive user intent, so differential privacy or other protections would be needed before real adoption."],"forward_implications":["A mobile GUI agent can be trained from automatically annotated user interaction logs, removing the human-annotator bottleneck that currently limits dataset scale and diversity.","At 1% of the overall cost, the federated auto-annotated pipeline matches or surpasses the centralized human-annotated baseline, so the cost-quality trade-off of GUI agent training shifts dramatically.","Because performance improves as more clients contribute, aggregating daily usage from many users offers a scalable path toward ever-larger training corpora without collecting raw screenshots centrally.","Adapted aggregation that weights episodes and steps outperforms standard federated aggregation by at least 5% relative under non-IID splits, which changes how federated VLM training should be weighted for structured trajectory data.","The online AndroidWorld results indicate that agents trained only on auto-annotated AndroidControl episodes can complete previously unseen tasks, suggesting the approach yields generalizable behaviors, not just benchmark overfitting."],"supporting_citations":[{"why":"Supplies the AndroidControl dataset and the Central-Human baseline that grounds the 1% overall-cost claim.","marker":"[9]"},{"why":"Provides the Android in the Wild dataset used to show the auto-annotation method transfers across datasets.","marker":"[27]"},{"why":"Provides the GUI Odyssey cross-app navigation episodes used to validate the framework on long trajectories.","marker":"[7]"},{"why":"Supplies the AndroidWorld online benchmark used to test generalization to unseen interactive tasks.","marker":"[28]"},{"why":"Defines evaluation protocols and strong GUI-agent baselines that MobileA3gent is compared against.","marker":"[29]"},{"why":"Represents the closest synthetic-data cost-reduction baseline, used for cost and quality comparison.","marker":"[30]"},{"why":"Supplies the federated LLM training implementation that this work extends to federated VLM training.","marker":"[21]"},{"why":"Provides Low-Rank Adaptation, which makes local client-side training feasible in FedVLM-A.","marker":"[22]"},{"why":"Defines FedAvg, the base aggregation rule that the adapted aggregation generalizes and compares against.","marker":"[12]"},{"why":"Provides FedYogi, a representative adaptive federated aggregation baseline in the non-IID study.","marker":"[14]"}],"fun_headline_variants":["Auto-annotated phone taps train GUI agents at 1% cost","Federated auto-annotation from diverse users matches human quality","Routine phone use auto-generates training data for mobile agents","Privacy-preserving auto-annotation yields agents at 1% cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-world promise depends on users' natural phone usage being capturable and locally annotatable at the same quality as the crowdsourced benchmark episodes used in the experiments, since the paper states it cannot yet run experiments on actual user phones.","fun_headline_variants_meta":{"raw":{"variants":["Auto-annotated phone taps train GUI agents at 1% cost","Federated auto-annotation from diverse users matches human quality","Routine phone use auto-generates training data for mobile agents","Privacy-preserving auto-annotation yields agents at 1% cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3246,"prompt_tokens":936,"completion_tokens":2310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2236}},"tokens_in":552,"tokens_out":2310,"duration_ms":20238,"temperature":1.0,"reasoning_tokens":2236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:18:30.828960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MobileA3gent pipeline on real user devices or a high-fidelity on-device logging study, then compare the trained agent's step success rate against the paper's simulated results on AndroidControl and AndroidWorld; the central claim would be falsified if real-world trajectories yield materially worse instruction quality or agent performance than the crowdsourced partitions, or if on-device annotation with small local VLMs cannot reproduce the reported ~99% cost reduction without a large drop in quality.","supporting_citations":[{"cited_title":"On the Effects of Data Scale on Computer Control Agents, August 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the AndroidControl dataset and the Central-Human baseline that grounds the 1% overall-cost claim."},{"cited_title":"Android in the Wild: A Large-Scale Dataset for Android Device Control, October 2023","cited_arxiv_id":null,"evidence_quote":"Provides the Android in the Wild dataset used to show the auto-annotation method transfers across datasets."},{"cited_title":"AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, June 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the AndroidWorld online benchmark used to test generalization to unseen interactive tasks."},{"cited_title":"OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis, December 2024","cited_arxiv_id":null,"evidence_quote":"Represents the closest synthetic-data cost-reduction baseline, used for cost and quality comparison."},{"cited_title":"Openfedllm: Training large language models on decentralized private data via federated learning","cited_arxiv_id":null,"evidence_quote":"Supplies the federated LLM training implementation that this work extends to federated VLM training."},{"cited_title":"Adaptive federated optimization","cited_arxiv_id":null,"evidence_quote":"Provides FedYogi, a representative adaptive federated aggregation baseline in the non-IID study."}],"review_version":1}