{"id":"d8694310-9d48-4d97-a72d-4bf784184345","arxiv_id":"2509.00531","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.","lead":"This paper presents MobiAgent, a complete mobile-agent system with purpose-trained models, a record-and-replay accelerator, and a new benchmark. The authors report higher task-completion scores than GPT-5, Gemini-2.5 Pro, and UI-TARS on their own MobiFlow benchmark, plus 2 to 3 times faster execution on recurring tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on MobiFlow, an unreleased, author-curated benchmark with manual failure review; external validation is required before the comparison is established.","rationale":"The reader's CONDITIONAL verdict correctly identifies the weakest assumption: MobiFlow's validity as a neutral evaluation yardstick. The strongest claim is exclusively about task completion on MobiFlow, so the benchmark must be an unbiased measure. The manuscript itself undermines this by stating tasks were selected for executability in the current environment and that all failures were manually verified by the authors. No artifacts, raw scores, or confidence intervals are provided, and the paper cautions that absolute scores are only for reference. This does not mean the results are false, but the SOTA conclusion is not independently established. A blinded re-scoring or external benchmark could settle this, so I leave the verdict unchanged.","tokens_in":12806,"tokens_out":4486,"duration_ms":55621,"concrete_test":"Release the complete MobiFlow task definitions, DAG condition files, and raw logged trajectories, and have two independent annotators who are blind to model identity re-score every trajectory (or a random subset of at least 50) against the published DAGs; compute inter-annotator agreement and check whether the MobiAgent-vs-Gemini/GPT-5 rank order survives blinded rescoring. If the rank order changes, the SOTA claim is an artifact of the authors' manual verification protocol rather than agent capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — MobiAgent achieves state-of-the-art task completion over GPT-5, Gemini-2.5 Pro, and UI-TARS — is measured only on MobiFlow, the authors' own, unreleased benchmark. For that claim to hold, MobiFlow must be a neutral yardstick, and the paper itself introduces doubt. §5.3 states: 'we meticulously select tasks that can be correctly executed in the current environment,' and §6.1 says 'we manually verified all failed tasks.' The DAG milestones and checkers were designed by the same group that designed the decider/grounder architecture, so the benchmark may implicitly reward that architecture; no task list, DAG configuration, raw per-agent scores, sample sizes, or confidence intervals are given. The paper even cautions that absolute MobiFlow scores are 'primarily for reference' (§5.3). Without an established public benchmark or a blinded, independent re-scoring of the recorded traces, the SOTA conclusion is not independently established — though the reported numbers may be accurate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MobiAgent, a full-stack mobile agent system with three components: MobiMind agent models (Planner 4B, Decider 7B, Grounder 3B), the AgentRR record-replay acceleration framework (ActTree plus latent memory models), and the MobiFlow DAG-based benchmark. The decider and grounder are post-trained from Qwen2.5-VL using SFT followed by two-stage curriculum GRPO, with a grounder-as-reward-model scheme and a self-evolution loop. The paper claims state-of-the-art task completion versus GPT-5, Gemini-2.5 Pro, and UI-TARS on the MobiFlow benchmark, as well as 2-3x latency improvement and over 99% replay correctness under simulated user task distributions. The principal evidence is Figure 5 and Figure 6 plus a few textual counts; no per-model score table, confidence intervals, total task counts, or external validation are provided.","tokens_in":13115,"tokens_out":4820,"duration_ms":59539,"significance":"If the results hold, the paper would be a useful systems contribution: the planner/decider/grounder decomposition, the record-replay cache with invalidation and eviction, the DAG-based benchmark supporting multiple valid trajectories and partial credit, and the low-cost trajectory collection pipeline are all potentially valuable. Strengths include the explicit training equations, the detailed ActTree description, and the multi-level verification design of MobiFlow. However, the headline SOTA claim is only as strong as MobiFlow, and MobiFlow is author-curated, unreleased, and not cross-checked against any public benchmark. The paper itself cautions that absolute MobiFlow scores are 'primarily for reference' (Section 5.3). The evidence is therefore not yet sufficient to establish superiority over GPT-5, Gemini-2.5 Pro, or UI-TARS, despite the reported numbers being plausible.","major_comments":[{"comment":"The central SOTA claim is measured solely on MobiFlow, an author-designed and unreleased benchmark. Section 5.3 states that the authors 'meticulously select tasks that can be correctly executed in the current environment' and that absolute scores are 'primarily for reference'; Section 6.1 reports manual verification of failures. These choices introduce selection bias that directly affects the comparison against GPT-5, Gemini-2.5 Pro, and UI-TARS. To make the claim load-bearing, the authors need to release the MobiFlow task set, DAG configurations, raw per-agent traces, per-task scores, and a precise task count, and ideally cross-evaluate on an established public benchmark. Without this, the reported superiority over the named baselines is not independently established.","section":"Section 5.3 / Section 6.1 / Figure 5"},{"comment":"There is a potential train/evaluation overlap that is not addressed. The self-evolution procedure in Section 3.2 collects new tasks at test time, corrects failed traces, and merges them into the training data. Simultaneously, the MobiFlow task templates are authored by the same group that designs the decider/grounder and its output format. The paper does not state whether any MobiFlow tasks or their templates were seen during training, self-evolution, or data collection. If the same or paraphrased tasks appear in both training and evaluation, the SOTA comparison is circular. The revision should state the disjointness criterion explicitly and ideally pre-register the MobiFlow DAGs before model development.","section":"Section 3.2 / Section 5.3"},{"comment":"The quantitative evidence is under-specified. The paper says 'we generated three real-world test cases for each type of template' (Section 6.1) but does not report the total number of templates, applications, or tasks, nor per-model scores, confidence intervals, or error bars in Figure 5. The statement that 'GPT failed to properly terminate tasks in 11 application categories' lacks a definition of category, termination penalty, and denominator. Similarly, Section 6.2 reports action replay rates and 'correctness of action replay exceeds 99%' under uniform and power-law distributions, but the distribution construction, task corpus, and replay-error definitions are not given. Providing a full results table and the underlying task lists is necessary for reproducibility.","section":"Section 6.1 / Section 6.2"},{"comment":"The incremental contribution over prior work [6] needs clarification. AgentRR is presented as a core component of MobiAgent, but reference [6] appears to be the same authors' earlier 'Get Experience from Practice' paper. The reader cannot tell which parts of the ActTree, latent memory, and replay framework are new in this manuscript and which are inherited. This matters for the claim that the 2-3x speedup is a contribution of the present system. The revision should state the delta relative to [6] and report which experimental results are newly produced here.","section":"Section 4 / Section 6.2"}],"minor_comments":[{"comment":"Several hyperparameters are left unspecified: the IoU threshold β and reward weight α in Eq. (1), the embedding similarity threshold τ1, the reranking threshold τ2, and the InfoNCE temperature τ in Section 4.4. Include the chosen values and, ideally, a sensitivity analysis.","section":"Section 3.2 / Section 4.4"},{"comment":"The Wait action is defined as Wait(sec: int) in Section 2.1, but the decider prompt in Section 3.1 says the wait action has no parameters and waits for 1 second. This discrepancy should be resolved.","section":"Section 2.1 / Section 3.1"},{"comment":"There are several typos and inconsistent terms: 'Secion' (Section 2.3), 'desciptions' (Section 2.3), 'reusablity' (Section 4.2), 'single step decison' (Section 2.3). The paper also alternates between 'Grounder' and 'Actor' (the latter appears in Section 4.5.1).","section":"Throughout"},{"comment":"The bar charts do not show per-run variability, sample counts, or statistical significance. Adding error bars or a companion table with raw scores and counts would improve interpretability.","section":"Figure 5 / Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a broad systems scope and the engineering contributions are potentially interesting, but the evaluation is entirely internal and the benchmark is not released. I would advise the editor that the key risk is not the plausibility of the reported numbers but the lack of a neutral yardstick; a revision that adds external benchmark results and releases MobiFlow artifacts could make the SOTA claim convincing. The relationship to the authors' prior paper [6] should also be clarified to assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a full-stack mobile agent system: a decider/grounder model pair trained with SFT plus curriculum GRPO, a record-replay accelerator (AgentRR, from the authors' prior work), and a new DAG-based benchmark, MobiFlow. What's genuinely new is the integration and the benchmark design — the DAG structure with AND/OR paths and multi-level checkers is a thoughtful attempt at handling real-world GUI tasks, and the AI-assisted data collection pipeline is practical. The decider/grounder split with the grounder as a reward model for RL is a sensible recipe, and the self-evolution step is plausible.\n\nThe soft spot is the SOTA claim. It rests entirely on MobiFlow, which is not released, and the paper itself says \"we meticulously select tasks that can be correctly executed in the current environment\" and \"we manually verified all failed tasks.\" That is exactly the kind of curation that can tilt the yardstick. There are no error bars, no per-app task counts, and no comparison on any established public benchmark. The paper also notes absolute MobiFlow scores are \"primarily for reference,\" which undercuts the strength of the comparative claim. The 2–3x speedup and >99% replay correctness come from simulated task distributions, not deployment, so that part is also less solid than it appears.\n\nThe acceleration core is self-cited to AgentRR [6]; the delta from that prior work is not quantified. Still, this is not a fatal flaw — the engineering is specific, the training details are reproducible in principle, and the benchmark's limitations are at least partially acknowledged. The right next step is to release MobiFlow and the task definitions, or to run on an independent benchmark like AndroidWorld or A3. Without that, the SOTA conclusion should be read as a strong possibility, not an established result.\n\nThis deserves a serious referee — it's a substantive systems paper with honest reporting of its own constraints. I'd want the authors to address external validation and release artifacts, but it's not a desk reject. I'd recommend conditional acceptance pending those additions.","headline":"A credible full-stack mobile agent system with a real SOTA claim, but the claim is measured only on an unreleased, author-curated benchmark, so external validation is needed before the comparison is established.","tokens_in":13609,"tokens_out":1407,"would_cite":false,"duration_ms":19045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MobiAgent, a 7B/3B decider-grounder pair with a record-replay cache, achieves the best task-completion scores on real Chinese mobile apps, beating far larger general-purpose and specialized GUI models.","keywords":["mobile agents","GUI grounding","vision-language models","decider-grounder architecture","record-replay cache","task completion benchmark","DAG milestone evaluation","GRPO curriculum training"],"falsifier":"Have annotators who did not design the system write a fresh set of tasks in the same Chinese apps sampled from real usage logs, run the same models, and compare completion and termination; if a frontier general-purpose model or the specialized GUI agent wins, the paper's main claim is overturned.","tokens_in":12745,"feed_emoji":"📱","tokens_out":6060,"duration_ms":65473,"temperature":0.7,"pith_summary":"Mobile-agent research usually pits one large vision-language model against a benchmark; this paper instead builds and evaluates a full stack around a small model pair. Its central claim is that a 7B 'decider' that chooses the next action, a 3B 'grounder' that locates the target UI element, a record-replay cache, and a DAG-based benchmark together yield the best task-completion rates measured on real Chinese apps—higher than much larger general-purpose models and than a leading specialized GUI agent. The authors also report that the AgentRR replay mechanism reuses 60–85% of actions under realistic user workload distributions and cuts end-to-end latency by 2–3x. If true, the paper shows that specialization and caching can substitute for raw model scale in mobile device control.","feed_headline":"MobiAgent beats GPT-5 and Gemini-2.5 Pro on phone tasks","feed_subtitle":"A 7B/3B decider-grounder pair wins on a DAG-based benchmark and replays up to 85% of actions","key_machinery":"The load-bearing object is the Decider–Grounder pair, trained so that the Grounder's bounding-box prediction doubles as an outcome reward for the Decider; the second object is ActTree, a tree of UI states and actions that lets a tiny embedding/reranker decide whether to replay a cached prefix. Together they separate what to do (language-level decisions) from where to do it (pixel-level grounding), and separate fast routine execution from slow model inference.","core_discovery":"On the paper's own terms, the discovery is that a two-model architecture—a Planner/Decider/Grounder trio where the Decider outputs high-level action descriptions and the Grounder maps them to bounding boxes—can be trained by two-stage GRPO, with the trained Grounder acting as a reward model for the Decider, to outperform all tested alternatives on the MobiFlow benchmark. MobiFlow scores tasks not by whether the final state matched one script but by whether the agent hit milestone events arranged in a DAG, with multi-level checkers and manual review of failures. The same system adds ActTree, a tree of recorded UI-state transitions, so that on repeated tasks a lightweight latent-memory model r","pith_inferences":["The benchmark's neutrality is testable: an independent group could define new tasks sampled from real user queries in the same apps, and if the ranking flips, the advantage is partly benchmark-specific.","The paper does not directly compare with open-weight agents trained at larger scale; the natural next experiment is to apply the same decider-grounder plus ActTree recipe to larger base VLMs.","ActTree-style prefix reuse might transfer to web and desktop GUI agents, where task repetition and UI stability are similar; the 99% replay correctness claim provides a concrete target for such ports.","Because GPT and Gemini sometimes used AI search to bypass steps, future benchmarks should include apps without AI-assisted search to separate planning skill from search shortcutting."],"forward_implications":["If the MobiFlow numbers hold, a 7B/3B pair can beat models an order of magnitude larger on concrete device tasks, so mobile agents do not need to wait for next-generation frontier models.","Record-replay acceleration works because real user requests follow a power-law distribution; reuse of 60–85% of actions means deployed agents can approach API-agent latency while keeping GUI generality.","DAG-based milestones allow partial credit and multiple valid trajectories, making it feasible to benchmark third-party apps whose success criteria are not scriptable.","Using the Grounder as a reward model gives a dense training signal for click decisions without requiring manual action-labels at RL time.","Training-time self-evolution, where failed traces are corrected and folded back into data, implies the agent improves on its own mistakes after each test round."],"supporting_citations":[{"why":"Supplies the Qwen2.5-VL base models used for post-training the Decider and Grounder.","marker":"[1]"},{"why":"Provides the record-replay agent idea that AgentRR extends into multi-level, tree-structured experience replay.","marker":"[6]"},{"why":"Gemini 2.5 Pro serves as a frontier general-purpose baseline in the MobiFlow comparison.","marker":"[7]"},{"why":"GPT-5 serves as the other frontier general-purpose baseline in the MobiFlow comparison.","marker":"[18]"},{"why":"UI-TARS is the specialized GUI-agent baseline whose task-completion score MobiAgent claims to beat.","marker":"[20]"},{"why":"AndroidWorld is a representative existing mobile benchmark whose verification approach MobiFlow contrasts with DAG milestones.","marker":"[21]"},{"why":"AITW is a large-scale Android-control dataset used to motivate the need for richer real-world trajectory collection.","marker":"[22]"},{"why":"The ReAct reasoning-and-acting paradigm is used to reconstruct high-level reasoning from recorded action traces.","marker":"[29]"},{"why":"Supplies the embedding and reranker models underlying the latent-memory similarity and reuse-confidence decisions.","marker":"[32]"}],"fun_headline_variants":["A 7B/3B decider-grounder pair beats GPT-5 and Gemini on mobile tasks","Decider-grounder duo powers MobiAgent past GPT-5 and Gemini","MobiAgent's two-model trick beats GPT-5 and Gemini-2.5 Pro on phones","Two-stage GRPO trains a decider-grounder that tops GPT-5 on mobile"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The ranking depends on MobiFlow's hand-curated tasks, milestone definitions, and manual failure review being a fair sample of real phone use; if those tasks were chosen around what the decider/grounder pipeline handles well, the reported advantage may not generalize to everyday app use.","fun_headline_variants_meta":{"raw":{"variants":["A 7B/3B decider-grounder pair beats GPT-5 and Gemini on mobile tasks","Decider-grounder duo powers MobiAgent past GPT-5 and Gemini","MobiAgent's two-model trick beats GPT-5 and Gemini-2.5 Pro on phones","Two-stage GRPO trains a decider-grounder that tops GPT-5 on mobile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001412,"raw_usage":{"total_tokens":5506,"prompt_tokens":674,"completion_tokens":4832,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":418,"completion_tokens_details":{"reasoning_tokens":4737}},"tokens_in":418,"tokens_out":4832,"duration_ms":37183,"temperature":1.0,"reasoning_tokens":4737,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:29:35.427006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have annotators who did not design the system write a fresh set of tasks in the same Chinese apps sampled from real usage logs, run the same models, and compare completion and termination; if a frontier general-purpose model or the specialized GUI agent wins, the paper's main claim is overturned.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gemini 2.5 Pro serves as a frontier general-purpose baseline in the MobiFlow comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-5 serves as the other frontier general-purpose baseline in the MobiFlow comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AITW is a large-scale Android-control dataset used to motivate the need for richer real-world trajectory collection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ReAct reasoning-and-acting paradigm is used to reconstruct high-level reasoning from recorded action traces."}],"review_version":1}