{"id":"06048fb7-0316-448d-b448-91b455cbe0b9","arxiv_id":"2504.19298","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AndroidGen combines retrieval, reflection, operation checking, and step-by-step evaluation to build Android agents that outperform prior systems on AndroidWorld and AitW without human-labeled trajectories.","lead":"AndroidGen is a framework that helps AI agents operate Android phones by learning from their own past attempts and from a small set of computer-generated examples. The authors report that their system beats existing Android agents on several benchmarks and can train open-source models without human-labeled data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ExpSearch retrieval database may overlap AndroidWorld test tasks, inflating the headline 46.8% and the +12.6 ExpSearch gain.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I would raise: the ExpSearch retrieval database is constructed by drawing on AndroidWorld instructions, and the paper does not establish that it is disjoint from the evaluation tasks. This is the most consequential issue because the headline comparison and the central ablation both hinge on ExpSearch. Without a disjointness guarantee, the reported gains could reflect memorization or near-duplicate retrieval rather than agent capability. The paper does have independent support: code, model, and data are released; ablations are internally consistent; and the AitW results, while based on a subset and human evaluation, are reported transparently. However, the missing retrieval boundary is a concrete, testable gap that should be resolved before the central claim is taken as established. My recommendation is consistent with the reader's conditional verdict: the paper should be accepted only after the authors clarify the data boundary and run the proposed overlap check.","tokens_in":17572,"tokens_out":3540,"duration_ms":34683,"concrete_test":"Inspect the ExpSearch database and measure overlap with AndroidWorld test tasks. For each AndroidWorld test instruction, retrieve the top-1 entry from the database using the same Contriever retriever, and have a human judge label whether the retrieved task is a near-duplicate (same app, same operation sequence structure, only parameter values changed). If any overlap exists, re-run the AndroidWorld evaluation with those database entries removed and report the average success rate and the ExpSearch ablation gain. If the gain collapses or the 46.8% drops substantially, the headline claim must be revised; if performance is unchanged, the leakage concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AndroidGen with GPT-4o reaches 46.8% on AndroidWorld and that ExpSearch contributes +12.6 points (Table 6, 34.2 to 46.8) depends on ExpSearch retrieving a useful trajectory. Section 4.1 states the trajectory database is built from 'about 300 task instructions drawing on the instructions in AndroidWorld,' and Section 3.2.1 retrieves the top-1 similar trajectory for every episode. The paper never states that the AndroidWorld test tasks are excluded from this database or that near-duplicate instructions are removed. The only leakage disclaimer ('We ensure no reward signals or golden labels are employed during training') addresses training labels, not retrieval-time example contamination. If a test task such as 'Create a playlist in Retro Music titled X with songs Y' has a database entry that is a paraphrase with different names, the retrieved top-1 example is effectively the solution trajectory, not a demonstration of general capability. This would explain both the 46.8% average and the large ExpSearch ablation gain, and it would undermine the comparison to M3A and SeeAct, which do not have access to such a database.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AndroidGen, a framework for building Android language agents under data scarcity. It combines four modules: ExpSearch, which retrieves similar trajectories from a database for in-context learning; ReflectPlan, which updates a task plan based on environment observations; AutoCheck, which verifies each generated action before execution; and StepCritic, which evaluates trajectories at sub-goal granularity. The authors use AndroidGen both as a zero-shot agent powered by GPT-4o and as a data-generation pipeline to fine-tune open-source models (GLM-4-9B and Llama-3-70B) without human-labeled trajectories. They report state-of-the-art results on AndroidWorld (46.8% average success for GPT-4o vs. 27.7% for M3A), strong AitW results, and improvements from the ExpSearch module in ablations. The paper also presents StepCritic as a more accurate evaluator than existing baselines and includes efficiency/cost comparisons with human annotation.","tokens_in":17742,"tokens_out":5555,"duration_ms":54281,"significance":"If the experimental results hold, this is a significant contribution: it demonstrates a fully synthetic data-generation pipeline for mobile agents, shows that retrieval-augmented self-reflection can substantially improve task completion, and releases code, models, and data. The framework addresses a real bottleneck (high-quality labeled trajectory scarcity) and its training-data pipeline for open-source models without human annotation is a valuable step toward reproducible mobile agents. However, the headline AndroidWorld result and the quantified benefit of ExpSearch depend critically on the integrity of the retrieval database, and the current manuscript does not provide sufficient evidence that the database is disjoint from the test tasks. The self-referential evaluation of StepCritic is a secondary but real concern. These issues are load-bearing for the central claims, so the paper needs substantive revision before the results can be accepted at face value.","major_comments":[{"comment":"The retrieval database used by ExpSearch is constructed from \"about 300 task instructions drawing on the instructions in AndroidWorld\" (Section 4.1), and AndroidWorld is exactly the benchmark reported in Table 2. The paper does not state that the AndroidWorld test tasks, or near-duplicates of them, were excluded from this database. Since Section 3.2.1 selects the top-1 similar trajectory for every episode, a database entry that paraphrases the test instruction (e.g., with different app-specific values) would provide a near-complete solution rather than a general demonstration. This could directly explain the +12.6 ExpSearch ablation gain in Table 6 and the 46.8% headline number. The sentence \"We ensure no reward signals or golden labels are employed during training\" addresses training labels only and does not rule out retrieval-time contamination. Please state explicitly whether the database is disjoint from the test tasks, describe any deduplication or near-duplicate removal, and, if necessary, re-run the AndroidWorld experiments with a database that provably excludes all test tasks and their paraphrases.","section":"Section 4.1 / Section 3.2.1 / Tables 2 and 6"},{"comment":"Section 3.2.1 states: \"each time the agent completes a task, we use StepCritic to assess the trajectory and log it to the database, which enables our agent to self-improve iteratively.\" If this logging occurs during the evaluation episodes on AndroidWorld itself, the database would accumulate trajectories taken on the test tasks, and later episodes could retrieve solutions to earlier test tasks. The manuscript does not specify whether the database used in the reported experiments is a fixed pre-collected snapshot or whether it grows online during evaluation. This ambiguity is critical for interpreting both Table 2 and the ExpSearch ablation in Table 6. Please clarify the exact protocol: was the database frozen before evaluation, and are any trajectories collected during evaluation excluded from retrieval?","section":"Section 3.2.1 / Section 4.1"},{"comment":"The StepCritic evaluator accuracy comparison uses \"trajectories generated by ANDROIDGEN as the test set,\" but these trajectories were themselves produced and filtered by the same pipeline whose quality is being assessed, and StepCritic is itself GPT-4o. This self-referential setup can inflate the reported accuracy relative to independent baselines. Furthermore, the manuscript does not describe what the \"environmental oracle prediction\" is, nor how the manual sub-goal and step-label accuracy in Table 5 was obtained. An independent human-annotated evaluation set, or at least a set generated by a different pipeline, is needed to support the claim that StepCritic outperforms the Captioner-based baselines. Please provide details of the oracle construction and the manual annotation protocol, or revise the evaluation to avoid this circularity.","section":"Section 5.4 / Table 5"}],"minor_comments":[{"comment":"The text cites \"AITW (Xing et al., 2024)\" but the reference list contains Rawles et al. (2023) for Android in the Wild; the citation appears to be conflated with the AndroidArena citation (Xing et al., 2024). Please correct the citation.","section":"Section 2 (Benchmarks of Mobile Agents)"},{"comment":"The caption ends with \"subscribe to the channel..\" (double period). Minor typographical error.","section":"Figure 2 caption"},{"comment":"The description \"about 300 task instructions\" is imprecise. Please give the exact number of generated instructions, the number of successful trajectories, and the final size of the trajectory database, as these numbers are important for assessing the scope of the retrieval pool.","section":"Section 4.1"},{"comment":"The column \"Completion Step\" is not defined. Please clarify what this metric measures (e.g., whether the model's predicted step index for each sub-goal must exactly match the human-annotated index) and how it is computed.","section":"Table 5"},{"comment":"The \"Oracle-Selection\" row is not explained in the text. Please specify what oracle labels are used (presumably environment feedback) and how they differ from StepCritic labels.","section":"Table 6 / Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The main concern raised by the stress-test is legitimate and central: the constructed retrieval database appears to be drawn from the same instruction source as the AndroidWorld test tasks, and the paper does not rule out contamination. If the authors cannot show that the database is disjoint from the test tasks, the primary empirical contribution (46.8% on AndroidWorld and the ExpSearch ablation gain) is not trustworthy. I also see a secondary circularity issue in the StepCritic evaluation and in the training-data pipeline, which should be addressed with independent evaluation. The paper is otherwise well-structured and the open-source release is a strength, but these issues must be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: AndroidGen is a serious engineering effort, and the open-source training pipeline is genuinely useful. But the headline AndroidWorld claim is not yet supported, because the ExpSearch retrieval database may contain near-duplicates of the test tasks.\n\nHere is the load-bearing problem. Section 4.1 builds the trajectory database from about 300 GPT-4o-generated instructions that 'draw on' AndroidWorld, and Section 3.2.1 retrieves the top-1 similar trajectory for every episode. AndroidWorld tasks are parameterized templates — playlist names, song titles, contact names — so an instruction that draws on the same template with different parameters is effectively a stored solution. The paper never says the test tasks or their near-duplicates are excluded from retrieval. The only leakage disclaimer is about reward signals and golden labels in training, which does not address retrieval-time contamination. If the database contains a trajectory for 'Create a playlist in Retro Music titled X with songs Y', then the agent facing '... title Z ... songs W' is being handed the solution path. That would explain both the 46.8% AndroidWorld average and the +12.6 ExpSearch ablation gain, and it makes the comparison to M3A and SeeAct, which have no such database, unfair. This is not a minor caveat; it is the difference between an agent that generalizes and an agent that copies.\n\nWhat is actually new and good: AndroidGen integrates retrieval, reflection, rule-based action checking, and LLM-based step evaluation into a coherent pipeline; the authors fine-tune open models without human-labeled trajectories; they release code, model, and data; the ablations are internally consistent; and the cost/efficiency analysis is useful. The AitW results for Llama-3-70B* are promising if the DigiRL protocol was followed faithfully, though the paper should state exactly how its 96-task sample overlaps DigiRL's test set. The StepCritic accuracy numbers in Table 5 are weakened by using the same pipeline's own trajectories as test data; that is an internal consistency check, not an independent validation.\n\nVerdict: this deserves peer review, because the engineering contribution is real and the leakage question is exactly the kind of thing a referee can force the authors to answer. The paper should not be accepted until the authors either show the retrieval database is disjoint from AndroidWorld test tasks (with overlap statistics and examples) or rerun the experiments without contaminated retrieval. If the numbers survive that check, the result is interesting. If not, the central claim collapses to a smaller, still useful claim about data generation.","headline":"AndroidGen is a competent engineering pipeline for data-scarce Android agents, but its headline AndroidWorld numbers rest on a retrieval database that likely overlaps the test tasks, so the central comparison is not yet trustworthy.","tokens_in":18287,"tokens_out":2957,"would_cite":false,"duration_ms":31236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AndroidGen reaches 46.8% average success on AndroidWorld by retrieving its own past trajectories as in-context examples.","keywords":["Android agents","language agents","data scarcity","in-context learning","trajectory retrieval","mobile UI automation","synthetic data generation","StepCritic"],"falsifier":"Open the released trajectory database and embed every AndroidWorld test instruction with the same retriever; if any test task or near-duplicate appears in the top-1 results under the same app context, the 46.8% success rate and the ExpSearch ablation gain partly measure retrieval of stored answers. Rerunning AndroidWorld with the database cleared of all AndroidWorld-derived instructions would settle it.","tokens_in":17353,"feed_emoji":"📱","tokens_out":12041,"duration_ms":97506,"temperature":0.7,"pith_summary":"The paper argues that a strong Android agent can be built without large-scale human-annotated trajectory data. Its AndroidGen framework combines four modules: ExpSearch retrieves a similar completed trajectory from a self-built database and injects it as a one-shot example; ReflectPlan initializes and revises step-by-step plans; AutoCheck verifies each operation before execution; and StepCritic decomposes tasks into sub-goals and scores each step. With GPT-4o, the framework reaches 46.8% average success on AndroidWorld under a standardized text-only action space, against 27.7% for M3A and 15.9% for SeeAct. The same pipeline also generates more than 1,000 filtered trajectories, and fine-tuning Llama-3-70B on them yields 35.3% on AndroidWorld and 74.0%/79.2% on AitW general/web splits, above the cited DigiRL agent. If correct, this means data scarcity is not a hard ceiling: one capable closed model plus automatic quality control can bootstrap both a strong agent and an open-source mobile agent.","feed_headline":"Android agent solves 46.8% of AndroidWorld tasks by self-retrieval","feed_subtitle":"No human labels needed: synthesized trajectories lift Llama-3-70B to 35.3% on AndroidWorld.","key_machinery":"The load-bearing mechanism is the ExpSearch retrieval loop: a trajectory database is populated by self-sampled, StepCritic-assessed episodes, and a dense retriever (Contriever) selects the top-1 similar trajectory for each new instruction, which then serves as a one-shot example in the model prompt. This is what the ablation credits with the largest gains: adding ExpSearch to ReflectPlan and AutoCheck raises GPT-4o from 20.7% to 46.8% on AndroidWorld, with hard tasks going from 0.0% to 11.8%. StepCritic is the supporting evaluator that makes the loop data-honest: it decomposes each task into sub-goals, maps each sub-goal to a completion step or -1, and its fine-grained judgments are used both to filter training data and to update the database.","core_discovery":"The central claim, stated in the paper's own terms, is that retrieval-augmented self-data construction turns a language model into a competitive Android agent under data scarcity. AndroidGen operates by having the model sample trajectories for roughly 300 GPT-4o-generated tasks drawn from AndroidWorld instructions, using StepCritic to evaluate whether each sub-goal is completed, and storing the assessed trajectories in a database. At each episode, ExpSearch uses Contriever to encode the current instruction and retrieve the top-1 similar trajectory, which is placed in the prompt; ReflectPlan maintains and updates a plan based on the current environment; AutoCheck rejects operations whose target element or state is not present; and StepCritic provides fine-grained labels. The paper reports that this reaches 46.8% on AndroidWorld with GPT-4o, and that the same trajectories, filtered and augmented by StepCritic, let a LoRA-tuned Llama-3-70B reach 35.3% on AndroidWorld and outperform the DigiRL agent on AitW. The intended upshot is that expert-level trajectory data can be manufactured rather than annotated, and that closed and open models both benefit from it.","pith_inferences":["If the retrieval database is truly disjoint from the test tasks, the +12.6-point ExpSearch ablation gain suggests top-1 trajectory retrieval transfers to unseen tasks, which would make the method a candidate for other GUI environments such as iOS or desktop web without retraining.","StepCritic's sub-goal decomposition could be reused as a dense reward or curriculum signal for reinforcement learning, not just as a filter; the paper itself only uses it for selection, augmentation, and database updates.","The error analysis points to a concrete next fix: memorization failures (20%) and math-counting failures (23%) suggest that an explicit external memory or a calculator-like tool inside the action space could extend the framework beyond what the current prompts achieve.","Because the 300 task instructions are drawn from AndroidWorld itself, the cleanest test of the contribution is whether the same pipeline can bootstrap a database from a disjoint task source and still produce comparable gains."],"forward_implications":["On AndroidWorld, an untrained GPT-4o agent with the four AndroidGen modules more than doubles the M3A baseline (46.8% vs 27.7%) under the same text-based action space and accessibility-tree input.","The generated trajectories can substitute for human annotation: fine-tuning Llama-3-70B on StepCritic-filtered synthetic data reaches 35.3% on AndroidWorld, close to the 46.8% of the untrained GPT-4o system.","The same data pipeline transfers across benchmarks: the fine-tuned Llama-3-70B scores 74.0% on AitW General and 79.2% on AitW Web, above the DigiRL reinforcement-learning agent's 71.9% and 67.2%.","StepCritic selection outperforms no selection (25.2% vs 16.2% on AndroidWorld with Llama-3-70B) and approaches oracle-based selection (27.0%), showing fine-grained automatic evaluation is a workable quality filter.","Quality-controlled synthetic data construction is substantially cheaper than human annotation, at roughly 5% of the cost and 5.85 times the efficiency for 1,000 trajectories."],"supporting_citations":[{"why":"Provides the AndroidWorld benchmark and the M3A baseline that AndroidGen is compared against.","marker":"Rawles et al., 2024"},{"why":"Supplies the Android-in-the-Wild dataset whose general and web splits are used in the AitW evaluation.","marker":"Rawles et al., 2023"},{"why":"Defines the DigiRL baseline and the AitW experimental setup that AndroidGen compares to.","marker":"Bai et al., 2024"},{"why":"SeeAct is the web-agent baseline adapted to AndroidWorld and the popular-app evaluation.","marker":"Zheng et al., 2024"},{"why":"Contriever is the dense retriever used by ExpSearch to choose the top-1 similar trajectory.","marker":"Izacard et al., 2021"},{"why":"Supplies the autonomous-evaluator baselines that StepCritic's accuracy is measured against.","marker":"Pan et al., 2024"},{"why":"LoRA is the fine-tuning method used to train the open GLM-4-9B and Llama-3-70B agents on generated trajectories.","marker":"Hu et al., 2021"},{"why":"AppAgent is a multimodal baseline used in the popular-app evaluation and AitW comparison.","marker":"Yang et al., 2023b"}],"fun_headline_variants":["Self-retrieval lifts Android agent to 46.8% on AndroidWorld","No human labels: Android agent hits 46.8% via self-retrieval","AndroidGen: Agent gains from synthesized trajectories, no annotations","Data-scarce Android agent: 46.8% solved via self-retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline AndroidWorld numbers assume that the retrieval database consulted during evaluation is disjoint from the test tasks, since the database is built from GPT-4o-generated instructions that draw on the same AndroidWorld task set.","fun_headline_variants_meta":{"raw":{"variants":["Self-retrieval lifts Android agent to 46.8% on AndroidWorld","No human labels: Android agent hits 46.8% via self-retrieval","AndroidGen: Agent gains from synthesized trajectories, no annotations","Data-scarce Android agent: 46.8% solved via self-retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1352,"prompt_tokens":954,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":313}},"tokens_in":570,"tokens_out":398,"duration_ms":4116,"temperature":1.0,"reasoning_tokens":313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:56:08.564992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the released trajectory database and embed every AndroidWorld test instruction with the same retriever; if any test task or near-duplicate appears in the top-1 results under the same app context, the 46.8% success rate and the ExpSearch ablation gain partly measure retrieval of stored answers. Rerunning AndroidWorld with the database cleared of all AndroidWorld-derived instructions would settle it.","supporting_citations":[],"review_version":1}