{"id":"bf5791f4-4525-48bb-9d12-6547c7913e30","arxiv_id":"2607.24112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A joint inverse-forward pretraining stage on visual screen transitions improves GUI-agent fine-tuning by 0.6–6.2 percentage points across three benchmarks.","lead":"GUI agents usually learn from expensive, fully labeled task trajectories. This paper tests a cheaper pretraining stage that learns from simple screen-to-screen transitions and reports consistent, modest gains across three desktop and mobile benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mobile step-success gains may be within run-to-run variance; no error bars, seeds, or significance tests are reported, so the claimed 0.6–0.9 pp consistency on AndroidControl/GUIOdyssey is not yet established.","rationale":"The reader's weakest assumption (extra compute / convergence) is reasonable, but it applies mainly to the desktop settings where gains are larger and the paper does show CE-plateau evidence. My concern targets the mobile results, where the effect size (0.6–0.9 pp) is close to typical seed-level noise in this class of models, and the paper provides no variance information. This is the single most load-bearing issue because the abstract and introduction claim 'consistently outperform across all settings' — one of the three benchmark families may not support that claim once variance is accounted for. The paper is otherwise well-constructed: controlled setup, ablations, scaling analysis, public code. So I do not recommend REJECT; the appropriate verdict remains CONDITIONAL, since the concern is about evidence strength rather than a demonstrated flaw. I partially agree with the reader: we both found a weakness in the comparison's evidential basis, but I weight the missing variance/error bars more heavily than the convergence argument; the convergence point is partly addressed by the paper's CE-plateau claim, whereas the variance point is entirely unaddressed. A concrete re-run with seeds would settle it.","tokens_in":18588,"tokens_out":1498,"duration_ms":12347,"concrete_test":"Re-run the AndroidControl and GUIOdyssey conditions (both FT w/ and FT w/o STP) with at least 3–5 random seeds each, using identical hyperparameters; report mean ± std of step success rate and a paired significance test (e.g., bootstrap or permutation test over the 4 decoding samples per instance). If the 95% CI for the STP-minus-baseline difference includes 0 in either mobile benchmark, the headline 'consistently improves across all settings' is not supported for mobile.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that STP consistently improves downstream performance. The desktop gains (2.3–6.2 pp) are fairly large, but the mobile gains in the headline result are small: +0.9 pp SR on AndroidControl and +0.6 pp SR on GUIOdyssey (Table 2). The paper reports a single run per condition: 'For each fine-tuning run, we report the performance of the best-performing checkpoint' (Section 3.1), with no seeds, confidence intervals, or significance tests anywhere. In LLM fine-tuning, run-to-run variance at 7B scale is typically on the order of 0.5–1.5 pp on such benchmarks, so a 0.6 pp difference is within plausible noise. This matters because the paper's own framing (Section 3.2) explains away the smaller mobile gains as 'prior data exposure and lower task complexity'; if the true effect is zero on mobile, the universality claim ('across all settings') weakens, even though the desktop result may still hold. Also, the checkpoint-selection protocol selects the best benchmark checkpoint for each run, which can inflate small differences when variance is high. Note the reader flagged convergence (extra-compute effect) as the weakest assumption; that is a real secondary concern, but the more load-bearing issue is the absence of any variance estimate for precisely the settings where the effect is smallest.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces State Transition Pretraining (STP), a pretraining stage for GUI agents in which a unified multimodal model (BAGEL) is continually trained on step-level visual transitions (s_t, a_t, s_{t+1}) using two objectives: inverse dynamics (predict the action from consecutive screenshots) and forward dynamics (generate the next screenshot from the current state and action). The STP checkpoint is then used to initialize standard trajectory fine-tuning. The authors compare STP + fine-tuning against direct trajectory fine-tuning on AgentNetBench, AndroidControl, and GUIOdyssey, reporting consistent improvements in average or step success rate across all settings. They also present ablations on pretraining objectives, sampling granularity, model components updated, and transition-data scale, and release code.","tokens_in":18895,"tokens_out":7598,"duration_ms":72474,"significance":"If the reported effects are reliable, STP is a practical and annotation-light scaling axis for GUI agents: transition tuples can be extracted from existing trajectories or collected automatically, and the controlled comparison protocol (same backbone, same fine-tuning data, same training configuration) is a methodological strength. The paper also contains useful ablations—joint inverse+forward beats single objectives, forward dynamics beats reconstruction, full-model tuning is necessary, step-level and trajectory-level sampling are comparable, and scaling transition data helps. The Group 1c control, showing gains even when STP uses no data beyond the fine-tuning pool, is a good attempt to separate data exposure from learning-signal effects. However, the absence of any variance estimates or significance tests, together with the small magnitude of the mobile gains, means the universality claim in the abstract and Section 3.2 is not yet established. The desktop results are substantially more convincing than the mobile results.","major_comments":[{"comment":"The central claim that STP consistently improves performance in both desktop and mobile scenarios rests on single-run comparisons with no error bars, significance tests, or multiple seeds. The mobile gains are +0.9 pp SR on AndroidControl and +0.6 pp SR on GUIOdyssey (Table 2); at 7B scale, run-to-run variation on such benchmarks is typically on the order of 0.5–1.5 pp, so these differences are within plausible noise. Because the abstract and Section 3.2 use these numbers to claim improvement across all settings, the mobile claim is load-bearing and currently unsupported. Please report at least 3 seeds with mean±std or confidence intervals, or explicitly restrict the universality claim to the desktop settings where the gains are larger.","section":"Section 3.2, Tables 1–2"},{"comment":"The paper states that all trajectory fine-tuning runs are trained to convergence, as reflected by the plateaued cross-entropy loss in Figure 4 (left). However, Figure 4 (left) compares datasets rather than fine-tuning with and without STP, and a CE plateau does not imply that success rate has converged. If the FT-only baseline would still improve with more compute, the STP gains could be partly an extra-compute effect. The convergence claim needs success-rate curves for both conditions over the full training horizon, and ideally compute-matched baselines (e.g., FT-only trained for the same total number of steps as STP+FT). Without this, the conclusion in Group 1c that gains come from structured learning signals rather than additional compute is not fully supported.","section":"Section 3.3, 'Practical cost'; Section 1, Figure 1 and Figure 4 (left)"},{"comment":"Reporting 'the performance of the best-performing checkpoint on the corresponding benchmark' for a single run can amplify small differences: with a single seed, selecting the best epoch can inflate the apparent effect, and there is no way to assess the variance. The same issue affects the scaling curve in Figure 6 (right) and the ablation bars in Figure 5. Please provide full training curves, fixed-epoch results, or averaged statistics over seeds; at minimum, report variance estimates for the headline numbers.","section":"Section 3.1, 'Implementation details'; Figure 1 caption"}],"minor_comments":[{"comment":"The evaluation decodes with temperature 0.6 and averages four samples per instance, but no standard deviation across samples is reported. Reporting this spread would help calibrate the small differences in Table 2.","section":"Section 3.1, 'Experimental Setup'"},{"comment":"The joint loss weight λ is set to 0.25 for AgentNet and GUIOdyssey and 0.5 for AndroidControl, but no sensitivity analysis is provided. Since λ is a free parameter, the authors should show that the main results are stable for nearby values or justify the choice.","section":"Section 2.3, Eq. (2)"},{"comment":"Figure 1 (left) mixes two metrics (cross-entropy loss and average success rate) in a single plot; the axes and curve labels need to be explicit. Also, Figure 4 (left) is cited in the convergence argument but does not compare fine-tuning with and without STP, so the citation is misleading.","section":"Figure 1 and Figure 4 captions"},{"comment":"The explanation that smaller mobile gains stem from prior data exposure and lower task complexity is plausible but untested. It would be more appropriate to present this as a hypothesis, or to support it with a benchmark less likely to be in the backbone's pretraining distribution.","section":"Section 3.2, 'Smaller gains in mobile scenarios'"}],"recommendation":"major_revision","confidential_remarks":"For the editor: This is a well-structured empirical study with a clean controlled protocol, and the desktop results are credible. My main reservation is statistical: the mobile headline gains are within plausible run-to-run noise, and the convergence argument is not supported by the cited figure. I recommend major revision rather than rejection: the authors can either add seeds/error bars and compute-matched baselines, or narrow the universality claim to desktop settings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, well-designed empirical study of a cheap pretraining stage for GUI agents. The desktop result is believable; the mobile result is not yet established, because the gains are tiny and the paper gives no variance estimate.\n\nWhat's actually new is the combination: joint inverse and forward dynamics on raw visual state transitions, trained in one unified multimodal model, with a careful downstream comparison that holds backbone, fine-tuning data, and protocol fixed. The paper also does two things right that many similar papers skip: a no-unseen-data control (Group 1c) that separates the structured learning signal from mere data exposure, and an explicit limitations section that honestly states what wasn't tried. The ablations are informative, the scaling experiment is a nice addition, and the code is public. That earns real credit.\n\nThe soft spots are statistical, not conceptual. Every main comparison is a single run per condition: no seeds, no error bars, no significance tests. And checkpoint selection is done on the benchmark itself, which, even when applied identically to both arms, inflates small differences. The mobile gains of 0.6–0.9 percentage points are well within typical run-to-run variance for a 7B model on these benchmarks, so the \"across all settings\" claim is overstated. The desktop gains (2.3–6.2 pp) are large enough that the variance concern is minor there, though still worth addressing.\n\nThe reader flagged the extra-compute objection as the weakest assumption. That's a real secondary issue: the paper argues that fine-tuning has converged based on cross-entropy plateaus, not evaluation curves, so the gains could partly reflect additional training time. Group 1c rules out unseen data but not extra compute. The paper's own explanation for the smaller mobile gains—prior data exposure and lower task complexity—is plausible but not tested. The free parameter lambda tuned per dataset is minor, though it does add a degree of freedom.\n\nWho should read this: anyone working on GUI agents, pretraining, or world models for interface control. It's also a good case study for discussing statistical rigor in LLM fine-tuning papers. It deserves a serious referee, but the referee should push for multiple seeds and error bars on the small-effect settings, and for an evaluation-curve convergence check. If the mobile gains vanish under proper variance reporting, the paper still holds as a desktop-oriented result with a useful method.\n\nMy recommendation: send it to review, with the expectation of a major revision on the statistical front. The core idea is solid and the desktop evidence is convincing enough that this is not a desk reject.","headline":"A well-controlled empirical paper on a cheap pretraining signal for GUI agents; the desktop gains look real, but the mobile gains are too small and statistically unanchored to support the 'consistently across all settings' claim.","tokens_in":19391,"tokens_out":2186,"would_cite":true,"duration_ms":25672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new pretraining stage on screen transitions gives GUI agents a consistent win over trajectory-only fine-tuning.","keywords":["GUI agents","state transition pretraining","inverse dynamics","forward dynamics","world model","trajectory fine-tuning","multimodal pretraining","scaling"],"falsifier":"Run the FT w/o STP baseline for the same total number of optimizer steps as the STP+FT pipeline, or until its success rate on the benchmark is flat rather than only its loss; if it reaches the same final success rate, the central claim fails. A sharper check: pretrain with the same transition data but shuffled or random action/screen targets, and see whether the downstream gain disappears.","tokens_in":1106,"feed_emoji":"🖥️","tokens_out":3270,"duration_ms":62982,"temperature":0.7,"pith_summary":"This paper tries to establish that GUI agents can be scaled more cheaply by pretraining on step-level visual state transitions—(current screen, action, next screen)—before the usual trajectory fine-tuning. The proposed State Transition Pretraining (STP) jointly optimizes inverse dynamics (recover the action from two consecutive screenshots) and forward dynamics (predict the next screenshot from a screen and an action). Across desktop and mobile benchmarks, the same fine-tuning setup produces higher success rates when initialized with STP, with average gains of 2.3–6.2 percentage points on AgentNetBench and smaller but consistent gains on AndroidControl and GUIOdyssey. The paper also argues that the gains are not simply extra data exposure, and that downstream performance keeps rising as transition data grows. If correct, transition data becomes a cheaper, annotation-free scaling axis for GUI agents.","feed_headline":"Pretraining on screen transitions lifts GUI agents on every test","feed_subtitle":"A cheap, annotation-free stage predicts actions and next screens, adding 0.6–6.2 points across desktop and mobile.","key_machinery":"The central object is the transition tuple (s_t, a_t, s_{t+1}), converted into two objectives within a single multimodal model: inverse dynamics (two screenshots to action text) and forward dynamics (screen plus action to next-screen image), optimized jointly before supervised trajectory fine-tuning. These objectives separate visual grounding from task planning and give the fine-tuning stage an action-grounded visual representation plus a lightweight internal model of GUI dynamics.","core_discovery":"STP claims to improve GUI agents' initialization by decomposing trajectories into atomic transition tuples and jointly optimizing two dynamics objectives in a unified multimodal model. Inverse dynamics forces the model to localize which interface element changed and infer the action that produced the change; forward dynamics forces it to internalize how GUIs respond to actions, forming an internal world model of GUI behavior. After this pretraining, trajectory fine-tuning outperforms fine-tuning alone on every tested setting. Notably, the improvement persists even when the transitions are extracted from the exact same trajectories later used for fine-tuning, and it grows with the volume of t","pith_inferences":["Because STP needs only step-level transitions, automated exploration of interactive environments could produce effectively unlimited pretraining data; the paper identifies this as future work, and its step-level sampling ablation suggests the approach would work.","The forward-dynamics head may have a practical side benefit not exercised here: at inference, a GUI agent could simulate the next screen before committing to an action, enabling lookahead or safety checks.","STP is largely orthogonal to the trajectory fine-tuning pipeline, so it could plausibly combine with reinforcement learning or other post-training methods, not just supervised fine-tuning.","If the paper's explanation for smaller mobile gains is right—backbone prior exposure and easier tasks—STP's value should grow as benchmarks become harder and less represented in base-model pretraining."],"forward_implications":["Starting trajectory fine-tuning from an STP checkpoint improves average success rate by 2.3–6.2 percentage points on AgentNetBench, 0.9 on AndroidControl, and 0.6 on GUIOdyssey under identical fine-tuning configurations.","The improvement is not simply exposure to unseen data: STP still helps when its transitions come from the same 18K trajectories later used for fine-tuning.","Downstream performance scales with transition pretraining volume even when fine-tuning data is fixed, so collecting more step-level transitions should keep helping.","Joint inverse and forward dynamics outperform either objective alone and beat reconstruction-only pretraining, indicating that action-aware next-state prediction is the informative signal.","Updating the full model matters more than improving the visual encoder alone, so the language component must be jointly optimized during STP."],"fun_headline_variants":["State transition pretraining boosts GUI agents across the board","Teach agents how screens change, and they act better","A pretraining trick: learn from screen transitions, not just tasks","Predicting screen changes makes GUI agents smarter","From transitions to actions: scaling GUI agents with dynamics"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The comparison that isolates STP assumes the fine-tune-only baseline has converged; the paper supports convergence with cross-entropy loss plateaus rather than success-rate curves, so if more fine-tuning would have closed the gap, part of the reported gain could be an extra-compute effect instead of the pretrained initialization.","fun_headline_variants_meta":{"raw":{"variants":["State transition pretraining boosts GUI agents across the board","Teach agents how screens change, and they act better","A pretraining trick: learn from screen transitions, not just tasks","Predicting screen changes makes GUI agents smarter","From transitions to actions: scaling GUI agents with dynamics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2448,"prompt_tokens":646,"completion_tokens":1802,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":390,"completion_tokens_details":{"reasoning_tokens":1725}},"tokens_in":390,"tokens_out":1802,"duration_ms":11909,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:00:49.145374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the FT w/o STP baseline for the same total number of optimizer steps as the STP+FT pipeline, or until its success rate on the benchmark is flat rather than only its loss; if it reaches the same final success rate, the central claim fails. A sharper check: pretrain with the same transition data but shuffled or random action/screen targets, and see whether the downstream gain disappears.","supporting_citations":[],"review_version":1}