{"id":"f26a5cee-8886-4396-a523-563a2ea45370","arxiv_id":"2502.06395","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A 3B VLM fine-tuned with SFT on AndroidControl and ReST-style iterations on AndroidWorld achieves competitive AndroidWorld success rates with GPT-4o while running about ten times faster.","lead":"AppVLM is a 3-billion-parameter vision-language model fine-tuned to control Android apps, first on the AndroidControl dataset and then iteratively on the AndroidWorld simulator. It reports matching GPT-4o-level task success in online app control at up to ten times lower inference time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's AndroidControl SOTA claim is contradicted by Table 1: AppVLM-base, not the final AppVLM, achieves the highest accuracy on all four splits.","rationale":"The central claim has two empirical pillars. The AndroidWorld pillar depends on the comparability of GPT-4o baselines; the reader chose this as the weakest assumption. However, the paper's asterisk convention ('methods with results taken directly from their papers are marked with an asterisk') implies T3A/M3A/SeeAct were re-run by the authors, and Section A.3 says the 82-task subset is used for all agent evaluations, so the baseline comparability concern is at least partially addressed and testable. In contrast, the AndroidControl attribution is unambiguously contradicted by the paper's own table, making it a more concrete and load-bearing flaw. The reader flagged this in their rationale but did not make it the core condition; I agree it matters but identify it as the single most important issue. The verdict should remain CONDITIONAL because the flaw is fixable by rephrasing the claim, not by redoing the experiments.","tokens_in":14063,"tokens_out":7771,"duration_ms":59634,"concrete_test":"A deterministic check: compute the per-split maxima from Table 1. If AppVLM is strictly below AppVLM-base on all four splits, the Abstract's claim is false and must be reworded to attribute the AndroidControl record to AppVLM-base or to the AppVLM pipeline collectively. No new experiments are needed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Abstract states that 'AppVLM achieves the highest action prediction accuracy in offline evaluation on the AndroidControl dataset, compared to all evaluated baselines.' This is not supported by the paper's own Table 1. The final model AppVLM scores 69.0/62.7/61.9/62.2 on the IDD/Task-Unseen/Cat-Unseen/App-Unseen splits, while the intermediate AppVLM-base scores 73.9/65.9/65.1/65.4, strictly higher on every split. On the IDD split, LT-all-r64 (70.8) also exceeds AppVLM. Thus, the named model does not achieve the highest accuracy; only AppVLM-base does. Reading 'AppVLM' as the whole pipeline (base plus final) conflicts with Algorithm 1, which defines AppVLM as the final SFT output. This is a load-bearing correctness issue because the headline result is false as stated, even though the underlying method may be sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AppVLM, a 3B-parameter vision-language model for Android app control. The method first performs supervised fine-tuning (SFT) on the AndroidControl dataset (producing AppVLM-base), then iteratively collects successful trajectories from the AndroidWorld environment and fine-tunes the policy (Reinforce Fine-Tuning, RFT), and finally performs an additional SFT step on AppVLM-base using all collected AndroidWorld data. The paper reports AndroidControl action-prediction accuracy across four splits and AndroidWorld online success rates across difficulty levels, and claims that AppVLM achieves state-of-the-art offline accuracy, matches GPT-4o baselines online, and runs up to ten times faster than GPT-4o-based agents.","tokens_in":14357,"tokens_out":4891,"duration_ms":37695,"significance":"If the empirical claims hold after the necessary corrections, the work would be a practical contribution: it demonstrates that a small open VLM can be made competitive with large proprietary GPT-4o agents on AndroidWorld at a fraction of the inference cost. The paper's strengths include the distributed client-server data-collection pipeline, the clear description of the RFT/ReST procedure, useful ablations in Tables 3 and 4, and a candid Section 4.5 that acknowledges that AppVLM-base, not the final AppVLM, achieves the best AndroidControl accuracy. However, the advertised claims in the abstract and contribution list outrun the data in Table 1, and the online comparison with GPT-4o baselines requires explicit clarification of evaluation protocols.","major_comments":[{"comment":"The statement that 'AppVLM achieves the highest action prediction accuracy in offline evaluation on the AndroidControl dataset' is contradicted by Table 1. AppVLM-base achieves 73.9/65.9/65.1/65.4 on IDD/Task-Unseen/Cat-Unseen/App-Unseen, while the final AppVLM achieves only 69.0/62.7/61.9/62.2, and the LT-all-r64 baseline reports 70.8 on IDD. Since Algorithm 1 defines AppVLM as the output of the final SFT step, the named model is not the state-of-the-art. The abstract and the first contribution bullet should be revised to attribute the AndroidControl SOTA to AppVLM-base, or the terminology in Algorithm 1 should be changed so that 'AppVLM' refers to the whole pipeline rather than the final checkpoints.","section":"Abstract; Section 1; Table 1"},{"comment":"The paper does not state whether the GPT-4o baseline results (T3A, M3A, SeeAct) in Table 2 were obtained on the same 82-task subset used for AppVLM, under the same evaluation protocol and seeds. Appendix A.3 shows that the 82-task subset has a harder difficulty distribution than the full 116-task benchmark (46.3% vs 52.6% easy). If the baseline numbers were taken from Rawles et al. (2024) on the full benchmark, then the claim that AppVLM 'matches GPT-4o' is not a like-for-like comparison. The authors should either re-run the baselines on their subset or clearly state the provenance of every number in Table 2, including the number of seeds and any filtering applied.","section":"Section 4.4; Table 2; Appendix A.3"},{"comment":"The claim that RFT provides 'adaptability to out-of-distribution tasks' and enables the agent to 'tackle tasks beyond those encountered during the initial SFT' is not established by the reported experiments. The RFT data are collected from AndroidWorld, and the online evaluation is also run on AndroidWorld with randomized parameters; the AndroidControl OOD splits are still drawn from the same dataset distribution as the SFT data. To support the OOD generalization claim, the authors should evaluate on tasks or apps that were not used during RFT data collection, or temper the claim to 'generalization to unseen parameterizations of AndroidWorld tasks.'","section":"Introduction; Section 4.5; Section 3.3"}],"minor_comments":[{"comment":"There is a duplicated article in the sentence 'This procedure continues until either the the task is solved'.","section":"Section 4.1"},{"comment":"The reference to 'Willams, R. J.' should be 'Williams, R. J.'.","section":"References"},{"comment":"The threshold tau is described as 'solved fewer than a threshold tau times' and later defined as tau = 10; please clarify whether tau is the target number of successes used for oversampling, and state how the oversampling frequency is computed from tau.","section":"Section 3.3.1"},{"comment":"The sentence 'AndroidControl also includes a click target as part of its input-text action, while we choose to keep these as separate actions as in AndroidControl' is confusing; please reword to clearly state the difference between the AndroidControl action format and the AppVLM action space.","section":"Appendix A.1"},{"comment":"The average inference times for AppVLM and AppVLM-base are both 0.91 seconds; consider adding a note that these are the same model architecture and that the value is the average per action-generation step.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv v1 and the authors appear to be from an industrial lab. The three major issues are all addressable: fixing the abstract/contribution claims, clarifying or re-running the GPT-4o baselines on the same subset, and being precise about what 'OOD' means. If the authors can provide the missing experimental provenance, the paper would be a solid empirical contribution. I would not recommend rejection based on the current state, but the headline claim must be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen —\n\nTwo things before you read: the method is a well-executed ReST-style recipe on PaliGemma-3B, and the abstract overstates what the paper actually shows. The headline claim that AppVLM achieves the highest AndroidControl accuracy is contradicted by Table 1: AppVLM-base (73.9 IDD) beats the final AppVLM (69.0) and LT-all-r64 (70.8). Section 4.5 says exactly that. So the named model does not hold the record; the intermediate checkpoint does. And \"first lightweight VLM\" to solve AndroidWorld tasks is undercut by the paper's own InfiGUIAgent baseline (2B, 9% success). These are not fatal to the work, but the abstract and contribution bullets need rewriting.\n\nWhat's genuinely good: the recipe is simple and reproducible. SFT on AndroidControl, three rounds of RFT (rejection sampling on successful AndroidWorld trajectories), then a final SFT on the base model beats both the RFT-only and the no-SFT variants in online success (37.8% vs 35.0% and 22.4%). The ablations in Tables 3 and 4 are clean, and the authors honestly flag the AndroidControl accuracy drop after RFT. A 3B model at 0.91s per action coming within 4 points of T3A on a harder 82-task subset is a real result worth having.\n\nThe soft spots, in order of severity. First, the AndroidWorld comparison: the paper uses a custom 82-task subset with a harder difficulty mix than the full 116, but never explicitly says whether the GPT-4o baselines (T3A, M3A, SeeAct) were re-run on that same subset. They have no asterisk in Table 2, which suggests the authors re-ran them, but the text doesn't confirm it. That ambiguity matters because the 4% gap to T3A could widen or shrink. Second, no error bars on AndroidControl or AndroidWorld, so we don't know the variance across seeds. Third, the \"OOD generalisation\" framing is weaker than it sounds: the model trains on AndroidWorld trajectories and then is evaluated on AndroidWorld. That is standard self-training, not true OOD.\n\nThe citation pattern is fine. The LT-all-r64 comparison is explicitly flagged as potentially inconsistent, which is the right kind of honesty. This paper belongs in the literature, but it needs a major revision to make claims match tables and to pin down the baseline protocol. I'd send it to a serious referee, and I'd expect the referee to ask exactly those questions before acceptance.\n\n— [Your name]","headline":"Solid ReST-style recipe on a 3B VLM, but the abstract's 'highest accuracy' and 'first lightweight' claims don't survive contact with its own tables.","tokens_in":14842,"tokens_out":4146,"would_cite":true,"duration_ms":32496,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3-billion-parameter vision-language model matches GPT-4o agents at smartphone app control while running up to ten times faster.","keywords":["app agents","vision-language model","AndroidWorld","AndroidControl","reinforced self-training","smartphone automation","GUI control","efficient inference"],"falsifier":"Re-run T3A and M3A on the same 82-task AndroidWorld subset with the same three-seed protocol used for AppVLM. If their success rates differ from the reported 41.9% and 36.6%, the claimed parity with GPT-4o is not established; a complementary check is reproducing AppVLM-base's 73.9% AndroidControl IDD accuracy under the paper's relaxed matching rule.","tokens_in":13826,"feed_emoji":"📱","tokens_out":5532,"duration_ms":41502,"temperature":0.7,"pith_summary":"This paper claims that a lightweight 3B vision-language model can be trained to control Android apps well enough to match large proprietary GPT-4o agents on the AndroidWorld benchmark, while running up to ten times faster. The training recipe starts with supervised fine-tuning on the AndroidControl dataset, then alternates between collecting successful trajectories in the AndroidWorld emulator and fine-tuning on them—a loop the paper calls Reinforce Fine-Tuning (RFT). In offline action-prediction tests on AndroidControl, the SFT-only model (AppVLM-base) reaches 73.9% accuracy on the in-distribution split, the highest among the evaluated baselines. Online, the final AppVLM solves 37.8% of 82 AndroidWorld tasks on average, compared with 41.9% for the best GPT-4o baseline (T3A) and 36.6% for M3A, with an average inference time of 0.91 seconds per action. The paper argues this makes a practical, low-cost alternative to querying large proprietary models.","feed_headline":"3B model matches GPT-4o at phone app control, 10x faster","feed_subtitle":"Two-stage fine-tuning on AndroidControl and AndroidWorld lifts task success to 37.8% at a fraction of GPT-4o's inference cost.","key_machinery":"The load-bearing mechanism is the Reinforce Fine-Tuning (RFT) loop, a version of reinforced self-training: the current policy is deployed in AndroidWorld emulators through a distributed client-server queue, successful trajectories are collected at high sampling temperatures, redundant steps are removed by comparing consecutive screenshots, under-solved tasks are oversampled up to a threshold of ten successes, and the policy is fine-tuned to maximize the log-likelihood of the successful actions. The observation format matters too: each screenshot is annotated with numbered bounding boxes of clickable UI elements from the accessibility tree, and the textual input contains the goal plus the last five actions, with click targets stored as condensed UI-element descriptions. A final full SFT of the original AndroidControl-trained base model on all collected data combines offline accuracy with online competence.","core_discovery":"The central claim is that a lightweight VLM, AppVLM, can close most of the performance gap with GPT-4o agents for online Android app control, and actually surpass them on offline action prediction, because its two-stage training transfers from a static human-demonstration dataset to a dynamic emulator environment. AppVLM-base—Paligemma-3B-896 fine-tuned only on AndroidControl—attains 73.9% relaxed action accuracy on the IDD AndroidControl split and 65.4% on the app-unseen split, beating the previous best fine-tuned baseline (LT-all-r64) and all GPT-4o baselines. Adding RFT (three iterations of collecting successful AndroidWorld trajectories, filtering redundant steps, oversampling under-solved tasks, and fine-tuning with maximum likelihood on successful trajectories) raises online success from 11.4% to 37.8%, then a final SFT step on the original base model recovers offline accuracy while keeping online gains. AppVLM solves 37.8% of tasks on the paper's 82-task subset, exceeding M3A (36.6%) and SeeAct (22.0%), and coming 4 percentage points behind T3A (41.9%), while taking 0.91 seconds per action versus 4.29–15.82 seconds for GPT-4o baselines.","pith_inferences":["The headline comparison to GPT-4o would be fully like-for-like only if the T3A/M3A numbers were rerun on the same 82-task subset with the same seed protocol; the paper does not explicitly say so, and the reported 4-point gap could shift if those baselines were drawn from the full 116-task benchmark.","If the efficiency holds at deployment scale, app agents could run locally on users' devices, shifting the practical bottleneck from inference cost to task coverage—the paper itself notes failures on unseen operations like clipboard use.","A dedicated, robust reward model for app control would let the RFT loop scale beyond AndroidWorld's 116 tasks, since the current environment is the only one with an internal reward signal.","A direct testable extension is to start RFT from a different base VLM (the paper suggests Llama-3 as an alternative) and check whether the same AppVLM-base-plus-RFT improvement curve appears."],"forward_implications":["An SFT-only 3B model can already set a higher offline action-prediction accuracy on AndroidControl than GPT-4o prompt-based agents and prior fine-tuned baselines.","The RFT loop produces a large online gain (11.4% to 37.8% AndroidWorld success) without sacrificing offline performance when followed by a final SFT on the base model.","AppVLM's per-action inference time (0.91 seconds) is roughly five times faster than T3A and over ten times faster than M3A and SeeAct, making the success rates achievable at a fraction of GPT-4o's cost.","Higher AndroidControl action accuracy does not automatically imply higher AndroidWorld success: Llama-3 and T3A outperform their visual counterparts offline yet underperform online.","The model can acquire rare or absent behaviors during RFT, such as long-press (under 1% of AndroidControl actions), once such actions appear in successful collected trajectories."],"supporting_citations":[{"why":"Supplies the AndroidControl dataset used for initial supervised fine-tuning and offline evaluation, and the LT-all-r64 baseline that AppVLM-base surpasses.","marker":"Li et al. (2024)"},{"why":"Provides the AndroidWorld environment for online training and evaluation, along with the T3A and M3A GPT-4o baselines that AppVLM is compared against.","marker":"Rawles et al. (2024)"},{"why":"Introduces Paligemma-3B-896, the base VLM whose 896x896 image resolution and 3B size are central to AppVLM's visual accuracy and speed.","marker":"Beyer et al. (2024)"},{"why":"Defines Reinforced Self-Training (ReST), the methodological template for the paper's RFT grow-and-improve loop.","marker":"Gulcehre et al. (2023)"},{"why":"Provides the SeeAct GPT-4o baseline used in the AndroidWorld online comparison.","marker":"Zheng et al. (2024)"},{"why":"Supplies the InfiGUIAgent fine-tuned baseline whose reported AndroidWorld success rate is compared with AppVLM's.","marker":"Liu et al. (2025)"},{"why":"Documents the out-of-distribution drop for fine-tuned app-control models, which motivates the paper's RFT refinement stage.","marker":"Chen et al. (2024)"}],"fun_headline_variants":["3B VLM beats GPT-4o on Android actions, 10x faster","Tiny AI matches GPT-4o on app control, 10x speedup","AppVLM: 3B model rivals GPT-4o, 10x faster on tasks","Lightweight VLM hits GPT-4o success, 10x faster","Small model equals GPT-4o for app control, 10x quicker"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the GPT-4o baseline success rates (T3A 41.9%, M3A 36.6%) were measured on the same 82-task AndroidWorld subset and under the same evaluation protocol as AppVLM, something the paper does not explicitly confirm.","fun_headline_variants_meta":{"raw":{"variants":["3B VLM beats GPT-4o on Android actions, 10x faster","Tiny AI matches GPT-4o on app control, 10x speedup","AppVLM: 3B model rivals GPT-4o, 10x faster on tasks","Lightweight VLM hits GPT-4o success, 10x faster","Small model equals GPT-4o for app control, 10x quicker"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1348,"prompt_tokens":1024,"completion_tokens":324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":640,"tokens_out":324,"duration_ms":3773,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:34:39.809931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run T3A and M3A on the same 82-task AndroidWorld subset with the same three-seed protocol used for AppVLM. If their success rates differ from the reported 41.9% and 36.6%, the claimed parity with GPT-4o is not established; a complementary check is reproducing AppVLM-base's 73.9% AndroidControl IDD accuracy under the paper's relaxed matching rule.","supporting_citations":[{"cited_title":"Spa-bench: A comprehensive benchmark for smartphone agent evalu- ation","cited_arxiv_id":null,"evidence_quote":"Documents the out-of-distribution drop for fine-tuned app-control models, which motivates the paper's RFT refinement stage."}],"review_version":1}