{"id":"e8c65739-517e-475f-96d4-d49b8261a022","arxiv_id":"2607.09773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Online RL with STEPO, policy-aware task filtering, DTAC curriculum, and async rollout infrastructure lifts a 32B computer-use agent to 63.2% on OSWorld-Verified.","lead":"EvoCUA-1.5 trains computer-use agents with online reinforcement learning in real desktop sandboxes, not just offline demos. It reports 63.2% success on OSWorld-Verified at 32B scale by fixing multi-turn RL credit assignment, task selection, curriculum, and slow environment infrastructure.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Headline OSWorld gains may be driven more by policy-specific task filtering than by STEPO/DTAC as general methods.","rationale":"The reader correctly locates the soft spot in validator quality and policy-dependent intermediate-pass-rate filtering (Sections 4.1–4.2; Table 8). STEPO’s conservation argument (Eqs. 10–12, Appendix B) is coherent and Fig. 6 supports it locally; mini-group batching and DTAC ablations are directionally positive. Those do not fully underwrite the strongest empirical claim—that the integrated stack yields a robust 63.2% at 32B—because the main result is a single uncontrolled point estimate, filtering is shown to be model-dependent (Table 5), and component ablations are not locked to the headline 32B configuration with variance. That keeps the verdict CONDITIONAL rather than ACCEPT or REJECT: the methods contribution is plausible and useful for the subfield if independent reimplementation confirms the filtering/curriculum assumptions, but the paper does not yet close that loop. My concern is the same as the reader’s weakest assumption, sharpened to a concrete multi-seed filter-vs-full-pool test.","tokens_in":18631,"tokens_out":699,"duration_ms":5827,"concrete_test":"Re-run the EvoCUA-1.5-32B online RL recipe three times with (i) the paper’s filtered pool, (ii) the unfiltered full pool under the same budget, and (iii) a pool re-calibrated only on a frozen earlier checkpoint (no online pass-rate updates). Report mean±std OSWorld-Verified Pass@1 and the Table 6-style category scores. If (i) does not beat (ii)/(iii) by a clear margin outside run variance, the load-bearing role of policy-aware filtering is overstated relative to STEPO/DTAC.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the full online stack (STEPO + policy-aware filtering/calibration + DTAC + async mini-group training) produces the 63.2% OSWorld-Verified result and stable multi-turn gains. The least secure condition is that the filtered, intermediate-pass-rate task pool remains a high-SNR, non-hackable, transferable training signal rather than a policy-overfit curriculum. Section 4.1 and Tables 4–5 show filtering helps and that a subset good for EvoCUA-8B fails to transfer (or slightly hurts) EvoCUA-32B; Table 8 fixes Plow=0.125, Phigh=0.375, ρinf=0.025 without sensitivity. Main Table 1 is a single Pass@1 point with no error bars, no multi-seed variance, and mixed max-step/base-model controls. Ablations (Fig. 6, Tables 3, 6) are partial and mostly on 8B/OpenCUA or Qwen3-VL-32B, not a full leave-one-component-out on the 32B EvoCUA-1.5 run. If validators are brittle or filtering overfits the current policy, the headline number can shrink while STEPO’s length-bias correction remains locally valid.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"EvoCUA-1.5 proposes an online RL framework for multi-turn computer-use agents that interact with executable desktop sandboxes under context management and sparse terminal rewards. The main technical pieces are STEPO (trajectory-level GRPO advantages redistributed as Ai/|Ti| after step decomposition; Eqs. 5–13), policy-aware filtering/pass-rate calibration of verifiable synthesized tasks, Dynamic Tri-Adaptive Curriculum (VAS + AdaPR + ICS), and a staleness-controlled asynchronous rollout–buffer–training stack with mini-group batching. On OSWorld-Verified the 32B model reports 63.2% Pass@1 (100 steps), with supporting ablations for STEPO (Fig. 6), mini-group batching (Table 3), filtered vs full data (Table 4), DTAC (Table 6), and some cross-platform/cross-domain transfer (Tables 2, 7).","tokens_in":19023,"tokens_out":1291,"duration_ms":11146,"significance":"If the gains hold under stronger controls, this is a useful systems-and-algorithm contribution for multi-turn GUI agents: it correctly identifies that context-managed trajectories break naive GRPO length balance, and it packages a practical online stack (objective + curriculum + async infrastructure) rather than a single-turn RL transplant. Strengths include a clear bias analysis for STEPO (Eqs. 5–12 and Appendix B), executable binary validators as the primary reward, and component ablations that move beyond pure leaderboard reporting. The work is of interest to the computer-use and agent RL communities even if the headline 63.2% is partly data-selection driven.","major_comments":[{"comment":"Table 1 reports a single 63.2% Pass@1 for EvoCUA-1.5-32B with no multi-seed variance, confidence intervals, or repeated evaluation protocol. Given stochastic multi-turn rollouts and policy-dependent task selection, this is insufficient to support the claim of a new 32B-scale SOTA or of stable improvement over EvoCUA-32B (56.7%/57.8%). At minimum, report evaluation variance and clarify whether the same max-step budget, backbone iteration, and decoding settings are held fixed against the closest open baselines (e.g., CUA-GYM-35B, Kimi-K2.5).","section":"§6.2, Table 1"},{"comment":"The central claim attributes the OSWorld result to the full stack (STEPO + filtering + DTAC + async mini-group training), but ablations are partial and mostly not on the final 32B EvoCUA-1.5 run: STEPO is shown on OpenCUA-32B (Fig. 6), mini-group batching on Qwen3-VL-8B (Table 3), DTAC on Qwen3-VL-32B (Table 6). There is no leave-one-component-out or matched-budget ablation that isolates each piece on the model that produces 63.2%. Without that, the headline number cannot be cleanly attributed to the proposed methods versus policy-specific data curation.","section":"§6.3; Fig. 6; Tables 3, 4, 6"},{"comment":"Tables 4–5 show that filtering helps and that a subset effective for EvoCUA-8B fails to transfer (or slightly hurts) EvoCUA-32B, which the paper itself treats as policy-dependent data quality. Combined with fixed curriculum bounds in Table 8 (Plow=0.125, Phigh=0.375, ρinf=0.025) and no sensitivity analysis, this leaves open the skeptic’s concern that intermediate-pass-rate filtering may overfit the current policy rather than yield a general high-SNR curriculum. Please add sensitivity on these thresholds and, if possible, a held-out or cross-policy evaluation of the filtered pool.","section":"§4.1–4.2; Tables 4–5, 8"}],"minor_comments":[{"comment":"Figure 1 and Table 1 mix models with different max-step budgets (50 vs 100); make the comparison protocol more explicit in the caption and text so readers do not over-read cross-budget rankings.","section":"Figure 1; Table 1"},{"comment":"Notation for advantages switches among Ai, ÂSTEPO_i,t, and Âi,t in Eqs. (10)–(13); unify symbols and state whether KL regularization is used in the reported runs.","section":"§3.2, Eqs. (10)–(13)"},{"comment":"The PRM reward-hacking observation (Fig. 7) is useful but qualitative; a short quantitative summary (e.g., PRM score vs terminal success correlation by difficulty) would strengthen §6.3.","section":"§6.3, Figure 7"},{"comment":"Appendix D lists hyperparameters but not group size G, number of tasks, total rollout steps, or wall-clock/async throughput; these are needed for reproducibility of the infrastructure claims in §5.","section":"§5; Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid engineering contribution in a fast-moving area, but the current evidence package is closer to a strong workshop/systems paper than a fully controlled journal result. I would accept after major revision if the authors add evaluation variance and at least one matched leave-one-component ablation on the 32B setting; without that, the 63.2% number risks being read as primarily a data-selection win. Scope fit is fine for an AI/systems venue that values agent RL infrastructure."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is practical, not foundational: after context management turns multi-turn CUA trajectories into step samples, naive GRPO overweighting long rollouts is a real bias, and STEPO’s Ai/|Ti| redistribution is a clean, small fix that their training curves support. The rest of the stack—policy-aware pass-rate filtering, DTAC (VAS + hard positive replay + controlled infeasibles), and staleness-aware async mini-group batching—is engineering that matches the actual bottlenecks of desktop RL (sparse binary validators, slow sandboxes, variable length).\n\nWhat is new relative to WebRL/MobileRL/DigiRL/UI-R1 is the explicit post-context-management objective analysis (Eqs. 5–12), mini-group batching so group-normalized advantages stay intact after decomposition, and the three-channel curriculum with an honest PRM-hacking warning. Ablations land where they should: STEPO beats naive multi-turn GRPO (Fig. 6), mini-group helps over fixed sample counts (Table 3), filtered high-SNR data beats full data (Table 4), DTAC helps (Table 6), and office-only RL transfers somewhat to Daily/Professional (Table 7). Cross-platform gains on Windows/macOS over the same backbone are a plus. Math is elementary but coherent; citations cover the right prior online GUI/web work; circularity is low because rewards come from executable validators.\n\nSoft spots, in proportion: the stress-test is partly right. Tables 4–5 show filtering is load-bearing and policy-dependent (an 8B-good subset slightly hurts 32B), and Table 8’s Plow/Phigh/ρinf are fixed without sensitivity. Main Table 1 is a single Pass@1 with no error bars, mixed step budgets, and incomplete leave-one-out on the final 32B run. No code/data release. That weakens transfer claims more than it invalidates STEPO’s local bias correction. Free hyperparameters exist, but they are ordinary RL knobs, not hidden fitting of the metric.\n\nThis is for people building multi-turn GUI/desktop agents who already care about GRPO-style training and sandbox throughput. Not for someone hunting a new theory of credit assignment. I would send it to peer review; the contribution is clear enough and the evidence is empirical-systems grade, not desk-reject grade. Engage if you train these agents; skim if you only need the STEPO length-bias note.","headline":"Solid systems-and-methods paper: STEPO’s length-bias fix is real, the 63.2% is competitive, and the main soft spot is policy-specific filtering plus missing variance—not a broken objective.","tokens_in":19721,"tokens_out":625,"would_cite":true,"duration_ms":9871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Online RL for multi-turn computer-use agents works when trajectory advantages, task selection, and async infrastructure are redesigned for context-managed steps.","keywords":["computer-use agents","online reinforcement learning","multi-turn GUI agents","step-level policy optimization","adaptive curriculum","asynchronous RL","OSWorld","context management"],"falsifier":"Train the same backbone and budget with STEPO, DTAC, and async mini-group batching, but replace the filtered intermediate-pass-rate task pool with unfiltered or misvalidated tasks; if OSWorld-Verified gains disappear or PRM-style hacking reappears while terminal success stays flat, the central claim fails.","tokens_in":19488,"feed_emoji":"🖥️","tokens_out":942,"duration_ms":9459,"temperature":0.7,"pith_summary":"This paper argues that computer-use agents cannot fully learn long-horizon desktop tasks from static offline traces, because each action changes the screen, the future action space, and recovery options. It extends prior offline self-evolution into online reinforcement learning inside executable sandboxes with verifiable terminal rewards. The authors claim that single-turn language RL recipes fail here: context management turns each trajectory into many step-level samples, rewards are sparse, trajectory lengths vary, and environments are slow. Their fix is a joint stack—Step-Level Policy Optimization that redistributes trajectory advantages evenly across steps, policy-aware filtering of verifiable tasks, a three-channel adaptive curriculum, and staleness-controlled asynchronous training with mini-group batching. With that stack, a 32B agent reaches 63.2% success on OSWorld-Verified, beating comparable open-weight models and approaching much larger systems. A sympathetic reader cares because the paper treats online computer-use RL as a coupled algorithm–data–system problem rather than a prompt-level objective transplant.","feed_headline":"32B computer agent hits 63.2% via online multi-turn RL","feed_subtitle":"STEPO, policy-aware tasks, and async curricula fix what single-turn RL recipes miss.","key_machinery":"Step-Level Policy Optimization (STEPO): compute a group-normalized trajectory advantage Ai, then assign Ai/|Ti| to each context-managed step so total advantage mass and zero-sum group balance are preserved after decomposition into turn-level samples.","core_discovery":"The paper establishes that multi-turn computer-use agents can be trained effectively with online RL if trajectory-level advantages are conserved after context-managed step decomposition, training data is filtered and calibrated to the current policy’s pass rates, curricula balance learnable tasks with hard positive replay and controlled infeasible exposure, and experience generation is decoupled from updates under staleness control. Under those conditions, online interaction becomes a scalable training signal and yields 63.2% Pass@1 on OSWorld-Verified for a 32B model.","pith_inferences":["If validators remain the bottleneck, scaling environments without better milestone or state-difference credit assignment will mostly add reward noise rather than capability.","Matching training-time and inference-time context management is itself a policy decision; agents that learn adaptive folding or summarization may need STEPO-style objectives redefined over those managed contexts.","GUI–CLI hybrid agents will inherit the same length-bias and group-structure issues whenever sparse terminal rewards are decomposed across mixed action streams."],"forward_implications":["Naive GRPO after multi-turn trajectory decomposition should be replaced by advantage redistribution that keeps group balance.","Online computer-use data must be recalibrated per policy; a high-SNR subset for one model need not help another.","Mini-group batching and staleness windows become first-class requirements once trajectories have variable step counts and slow sandbox feedback.","Process reward models need alignment checks against executable final success, or they can raise intermediate scores without improving completion.","Cross-domain office-only online RL can still lift Daily and Professional categories by strengthening shared atomic skills."],"fun_headline_variants":["32B multi-turn agent reaches 63.2% OSWorld via online RL","EvoCUA-1.5: online RL trains computer agents to 63.2% success","STEPO conserves trajectory advantages for 63.2% computer RL","Online multi-turn RL with DTAC yields 63.2% on OSWorld-Verified","Async curricula push 32B computer-use agent to 63.2% Pass@1"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The whole result leans on sandbox validators and policy-dependent pass-rate filters giving a clean, non-hackable signal that intermediate-difficulty tasks stay informative as the agent improves.","fun_headline_variants_meta":{"raw":{"variants":["32B multi-turn agent reaches 63.2% OSWorld via online RL","EvoCUA-1.5: online RL trains computer agents to 63.2% success","STEPO conserves trajectory advantages for 63.2% computer RL","Online multi-turn RL with DTAC yields 63.2% on OSWorld-Verified","Async curricula push 32B computer-use agent to 63.2% Pass@1"]},"model":"grok-4.5","effort":"low","cost_usd":0.006128,"raw_usage":{"total_tokens":1646,"prompt_tokens":889,"num_sources_used":0,"completion_tokens":118,"cost_in_usd_ticks":61280000,"prompt_tokens_details":{"text_tokens":889,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":639,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":889,"tokens_out":118,"duration_ms":6375,"temperature":1.0,"reasoning_tokens":639,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T15:59:55.381178+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same backbone and budget with STEPO, DTAC, and async mini-group batching, but replace the filtered intermediate-pass-rate task pool with unfiltered or misvalidated tasks; if OSWorld-Verified gains disappear or PRM-style hacking reappears while terminal success stays flat, the central claim fails.","supporting_citations":[],"review_version":1}