{"id":"4b48f9d8-feac-4dba-8d79-c47ca18c5501","arxiv_id":"2605.29486","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"PhoneWorld is a pipeline that converts real mobile trajectories into scalable controllable environments, yielding large gains on four benchmarks when used to supplement training data.","lead":"PhoneWorld turns real phone GUI trajectories and screenshots into many controllable simulated mobile environments, tasks, and verifiers for training AI agents. A smart generalist might read it because scalable training data could make phone-using AI assistants more practical for everyday tasks like shopping or booking.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Pipeline's automatic recovery of screens, transitions, and verifiers from trajectories lacks independent validation","rationale":"The reader's weakest_assumption directly identifies the reconstruction step as the load-bearing risk; the full text does not supply an independent check that would remove it, so the UNVERDICTED stance remains appropriate.","tokens_in":1860,"tokens_out":304,"duration_ms":10189,"concrete_test":"Sample 30 trajectories from the PhoneWorld corpus; for each, manually compare the generated mock app's state graph and verifier predicates against the original screenshots and action sequence; report the fraction of cases where (a) all state-changing actions are captured and (b) the verifier accepts exactly the valid completions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (gains on all four benchmarks after swapping 10K AndroidWorld steps for PhoneWorld supervision) rests on the claim that real trajectories can be turned into faithful runnable mock apps plus rule-based verifiers. Section 3 describes recovering \"which screens matter, how screens connect, which interactions change state, and which goals admit automatic verification,\" then building read-only content + mutable state. No quantitative audit of reconstruction fidelity (e.g., precision of extracted state machines or verifier soundness) is reported; if the recovered graphs omit key transitions or produce overly permissive verifiers, the training signal is mis-specified and the cross-benchmark gains could be artifacts of simplified environments rather than genuine scaling.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces PhoneWorld, a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments consisting of runnable mock Android apps, executable tasks, rule-based verifiers, and training rollouts. The pipeline recovers screens, transitions, state-changing interactions, and automatically verifiable goals from trajectories to build read-only content plus mutable state. In experiments, replacing 10K steps from an auxiliary AndroidWorld corpus with PhoneWorld supervision (covering 34 apps across 16 domains) improves all four benchmarks under a fixed training budget: +17.7 on HYMobileBench, +6.0 on AndroidControl, +14.7 on AndroidWorld, and +52.5 on PhoneWorld. Additional scaling studies examine increasing supervision volume and app coverage.","tokens_in":2011,"tokens_out":545,"duration_ms":19077,"significance":"If the automatic recovery process produces faithful environments, the work would meaningfully address the scalability bottleneck in mobile-agent training by shifting from hand-crafted benchmarks to automated environment generation from real trajectories. The simultaneous gains across external and internal benchmarks, plus the scaling ablations, provide concrete evidence that the approach can improve agent performance; the multi-benchmark design and fixed-budget protocol are strengths.","major_comments":[{"comment":"Section 3: The pipeline description claims accurate recovery of 'which screens matter, how screens connect, which interactions change state, and which goals admit automatic verification,' followed by construction of runnable mocks and rule-based verifiers, yet reports no quantitative fidelity audit (e.g., precision of extracted state machines against held-out trajectories or soundness of verifiers measured by human agreement). This is load-bearing for the headline result because the reported cross-benchmark gains rest on the quality of the generated training signal; without such validation the improvements could arise from simplified or mis-specified environments rather than genuine scaling.","section":"Section 3"},{"comment":"Evaluation (results paragraph): The +52.5 point gain is reported on the PhoneWorld benchmark itself, which is constructed from the same trajectory-to-environment pipeline; while the gains on the three external benchmarks (HYMobileBench, AndroidControl, AndroidWorld) provide independent grounding, the internal result requires explicit discussion of potential circularity and should be down-weighted in the overall claim.","section":"Evaluation"}],"minor_comments":[{"comment":"The abstract and results do not specify the exact model architecture, training hyperparameters, or data-split details used in the fixed-budget experiments, which would strengthen reproducibility claims.","section":"Abstract and results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. The two major comments identify substantive gaps in validation and result interpretation. We address both below and will revise the manuscript accordingly.","responses":[{"response":"We agree that a quantitative fidelity audit is necessary to substantiate the pipeline's claims and that its absence weakens the evidential basis for attributing gains to faithful environment recovery. In the revision we will add a new subsection (or appendix) reporting (1) precision/recall of recovered state machines on held-out trajectories and (2) human agreement rates on verifier soundness. These metrics will be computed on a random sample of 200 trajectories and verifiers. We view this addition as essential.","revision_made":"yes","referee_comment":"[Section 3] Section 3: The pipeline description claims accurate recovery of 'which screens matter, how screens connect, which interactions change state, and which goals admit automatic verification,' followed by construction of runnable mocks and rule-based verifiers, yet reports no quantitative fidelity audit (e.g., precision of extracted state machines against held-out trajectories or soundness of verifiers measured by human agreement). This is load-bearing for the headline result because the reported cross-benchmark gains rest on the quality of the generated training signal; without such validation the improvements could arise from simplified or mis-specified environments rather than genuine scaling."},{"response":"We concur that the PhoneWorld-internal result carries circularity risk and should not be given equal weight. In the revised evaluation section we will (a) explicitly flag the shared pipeline origin, (b) present the three external-benchmark gains as the primary evidence, and (c) relegate the +52.5 figure to a secondary, caveated observation. The abstract and conclusion will be updated to reflect this re-weighting.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation (results paragraph): The +52.5 point gain is reported on the PhoneWorld benchmark itself, which is constructed from the same trajectory-to-environment pipeline; while the gains on the three external benchmarks (HYMobileBench, AndroidControl, AndroidWorld) provide independent grounding, the internal result requires explicit discussion of potential circularity and should be down-weighted in the overall claim."}],"tokens_in":1568,"tokens_out":484,"duration_ms":15721,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is a pipeline that takes real GUI trajectories, pulls out screens, transitions, state-changing actions, and verifiable goals, then builds runnable mock apps with rule-based verifiers and training rollouts. Swapping 10K steps of AndroidWorld data for PhoneWorld supervision lifts four separate benchmarks at once.\n\nThe work is new in its focus on automated environment generation rather than one-off benchmark construction. It covers 34 apps across 16 domains and reports clear numeric improvements: 17.7 points on HYMobileBench, 6.0 on AndroidControl, 14.7 on AndroidWorld, and 52.5 on PhoneWorld itself. The scaling experiments (more data helps, wider app coverage helps more) are straightforward and useful for the mobile-agent crowd.\n\nThe soft spot is exactly the one flagged in the stress-test note. The abstract describes recovering state machines and verifiers from trajectories but gives no quantitative check on how faithful those recoveries are. Without precision numbers on omitted transitions or verifier soundness, it is hard to know whether the gains come from better training signal or from simplified environments. That gap is real and load-bearing for the central claim.\n\nThis paper is for people working on scaling mobile agents who need more controllable training data. It has enough concrete results and a reusable method to deserve referee time, even though the fidelity validation is missing.","headline":"PhoneWorld turns trajectories into scalable mock phone environments and shows cross-benchmark gains from the resulting supervision, but the reconstruction quality of those environments is untested.","tokens_in":2562,"tokens_out":354,"would_cite":false,"duration_ms":14599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"PhoneWorld turns real GUI trajectories into controllable phone-use environments, tasks, and verifiers at scale.","keywords":["phone-use agents","GUI trajectories","mobile environments","scalable benchmarks","mock Android apps","automatic verifiers","agent supervision","environment construction"],"falsifier":"Running the pipeline on a held-out set of trajectories and checking whether the generated mock apps and verifiers produce the same task outcomes as direct human evaluation on the original screens.","tokens_in":2759,"feed_emoji":"📱","tokens_out":724,"duration_ms":18731,"temperature":0.7,"pith_summary":"The paper seeks to overcome the difficulty of building many reproducible phone environments for agent training by creating an automated pipeline from existing user data. It extracts the relevant screens, their connections, state-changing interactions, and automatically verifiable goals directly from trajectories and screenshots. These signals are then used to construct runnable mock apps that support executable tasks, rule-based checks, and training rollouts across 34 apps in 16 domains. When this generated supervision replaces a fixed amount of auxiliary data in an existing baseline, performance rises on every tested benchmark at once. A reader would care because the approach moves the field from crafting isolated benchmarks toward systematically increasing the supply of phone-use environments themselves.","feed_headline":"Pipeline turns real phone trajectories into scalable agent environments","feed_subtitle":"Recovering screens and interactions from existing data supplies 34 controllable apps that improve four benchmarks when used for training sup","key_machinery":"The PhoneWorld pipeline that extracts screens, connections, state-changing interactions, and verifiable goals from trajectories to generate mock apps and automatic verifiers.","core_discovery":"PhoneWorld recovers which screens matter, how they connect, which interactions change state, and which goals admit automatic verification from real trajectories. From these signals it builds runnable mock Android apps backed by read-only content and mutable state, then derives executable tasks, rule-based verifiers, and training rollouts from the same environments. Under a fixed training budget, replacing 10K steps from an auxiliary AndroidWorld corpus with broad PhoneWorld supervision improves HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points. Increasing the amount of PhoneWorld supervision further improves PhoneWorld perf","pith_inferences":["The recovery approach could be applied to trajectories from other operating systems to generate environments beyond Android.","Diverse app coverage may reduce overfitting to narrow task patterns that arise when training inside a single environment.","If the pipeline scales to thousands of apps, the limiting factor for phone agents could shift from environment availability to model architecture or interaction modeling."],"forward_implications":["Replacing auxiliary AndroidWorld data with PhoneWorld supervision raises scores on all four evaluation benchmarks simultaneously.","Increasing the volume of PhoneWorld supervision produces strong gains on PhoneWorld itself.","Under a fixed PhoneWorld budget, covering more apps yields larger performance improvements than adding steps within fewer apps.","The same recovery process supplies both training rollouts and automatic verifiers for the generated environments."],"fun_headline_variants":["PhoneWorld converts trajectories into controllable phone environments","Real trajectories yield 34 mock Android apps","Trajectory data recovers screens for scalable environments","PhoneWorld derives tasks from recovered phone interactions"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The pipeline can accurately recover which screens matter, how they connect, which interactions change state, and which goals admit automatic verification from real trajectories.","fun_headline_variants_meta":{"raw":{"variants":["PhoneWorld converts trajectories into controllable phone environments","Real trajectories yield 34 mock Android apps","Trajectory data recovers screens for scalable environments","PhoneWorld derives tasks from recovered phone interactions"]},"model":"grok-4.3","cost_usd":0.005419,"raw_usage":{"total_tokens":2604,"prompt_tokens":819,"num_sources_used":0,"completion_tokens":52,"cost_in_usd_ticks":54190500,"prompt_tokens_details":{"text_tokens":819,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1733,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":819,"tokens_out":52,"duration_ms":7525,"temperature":1.0,"reasoning_tokens":1733,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:39:42.996602+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the pipeline on a held-out set of trajectories and checking whether the generated mock apps and verifiers produce the same task outcomes as direct human evaluation on the original screens.","supporting_citations":[],"review_version":1}