{"id":"ac208f51-c90e-42a8-b675-02a16d31e2bc","arxiv_id":"2606.02031","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenWebRL trains a 4B visual web agent with online RL on live sites using 0.4K init trajectories and 2.2K RL tasks to reach 67% success on Online-Mind2Web and 64% on DeepShop, outperforming prior open agents.","lead":"OpenWebRL introduces an open framework for training visual web agents via online multi-turn reinforcement learning directly on live websites, including infrastructure, context management, and policy optimization. A smart generalist might read it to see a practical route toward scalable, reproducible open agents that reduce reliance on expensive curated trajectory datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Trajectory-level success judging may not yield sufficiently low-noise rewards for stable multi-turn RL on dynamic live sites","rationale":"The reader's weakest assumption directly targets the mechanism that must hold for the small-data RL claim to be credible. No stronger internal inconsistency or missing formal step was identifiable from the abstract and stated results; the concern is empirical rather than logical and is best settled by the concrete validation check above.","tokens_in":1844,"tokens_out":364,"duration_ms":15132,"concrete_test":"Extract the exact success-judging prompt, model, and any validation metrics from the methods section on trajectory-level success judging; recompute success rates on a held-out sample of 100 trajectories using an independent human rater or a stronger judge model and measure agreement (Cohen's kappa or exact match); if agreement <0.85 or if >15% of trajectories flip label, re-train or simulate the RL loop with the observed noise level to check whether the reported benchmark deltas remain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance (67% on Online-Mind2Web, 64% on DeepShop after 2.2K RL tasks) rests on the claim that online multi-turn policy optimization succeeds with only 0.4K init trajectories. This requires the trajectory-level success judge plus live-browser setup to deliver reward signals whose noise level permits effective credit assignment across long horizons. If judging errors (false positives on partial success, false negatives on correct but non-canonical paths, or inconsistency across dynamic page states) are frequent, the PPO-style or equivalent updates would optimize against noisy or mis-specified objectives, making the reported gains with such small data volumes implausible without additional stabilizing mechanisms that are not yet shown to be sufficient.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces OpenWebRL, an open framework for training visual web agents via online multi-turn RL directly on live websites. It covers the full pipeline: scalable live-browser infrastructure, supervised initialization from 0.4K trajectories, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization. The central empirical claim is that OpenWebRL-4B, after 2.2K open-ended RL tasks, reaches 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger scale while remaining competitive with proprietary systems such as OpenAI CUA and Gemini CUA. The work also examines key design choices and how RL improves agentic reasoning.","tokens_in":2001,"tokens_out":626,"duration_ms":16505,"significance":"If the reported results hold under rigorous validation, the contribution would be significant: it demonstrates that online multi-turn RL can produce competitive visual web agents with far smaller data volumes than supervised post-training on curated trajectories, while releasing infrastructure, data, models, and code to enable reproducible open research. This directly addresses the scalability bottleneck highlighted in the abstract and provides concrete evidence on effective design choices for RL in dynamic web environments.","major_comments":[{"comment":"§4 (Experiments) and §3.4 (Success Judging): The headline performance numbers rest on the assumption that trajectory-level success judging supplies sufficiently low-noise rewards for stable multi-turn policy optimization. The manuscript must supply quantitative validation of the judge (e.g., agreement rate with human labels on a held-out set of trajectories, false-positive/false-negative rates on dynamic pages) to substantiate that credit assignment across long horizons is reliable; without this, the gains achieved with only 2.2K RL tasks remain difficult to attribute to effective RL rather than judge artifacts.","section":"§4 and §3.4"},{"comment":"Table 1 / Results: The reported success rates (67.0% and 64.0%) are presented without error bars, variance across seeds, or ablations isolating the contribution of the success judge versus other pipeline components. This omission is load-bearing because the central claim is that online RL succeeds with small data volumes; the absence of these controls leaves open the possibility that results are sensitive to judging noise or website state variability.","section":"Table 1 / Results"}],"minor_comments":[{"comment":"Abstract: The phrase 'we systematically study the key design choices' is stated without enumerating them; a brief parenthetical list or forward reference to the relevant section would improve clarity.","section":"Abstract"},{"comment":"§5 (Analysis): The discussion of how RL improves agentic reasoning would benefit from explicit comparison of pre- and post-RL trajectories on the same tasks to illustrate the claimed improvements.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments emphasizing the need for rigorous validation of the success judge and improved statistical reporting. We address each major comment below and will revise the manuscript accordingly to strengthen these aspects.","responses":[{"response":"We agree that quantitative validation of the judge is necessary to confidently attribute performance gains to the RL process. The manuscript describes the trajectory-level success judge in §3.4 but does not include human agreement metrics or error rate breakdowns. In the revised manuscript we will add a dedicated analysis in §3.4 reporting agreement rates with human labels on a held-out trajectory set together with false-positive and false-negative rates stratified by page dynamism. This addition will directly address concerns about reward noise and credit assignment reliability.","revision_made":"yes","referee_comment":"[§4 and §3.4] §4 (Experiments) and §3.4 (Success Judging): The headline performance numbers rest on the assumption that trajectory-level success judging supplies sufficiently low-noise rewards for stable multi-turn policy optimization. The manuscript must supply quantitative validation of the judge (e.g., agreement rate with human labels on a held-out set of trajectories, false-positive/false-negative rates on dynamic pages) to substantiate that credit assignment across long horizons is reliable; without this, the gains achieved with only 2.2K RL tasks remain difficult to attribute to effective RL rather than judge artifacts."},{"response":"We acknowledge that the absence of error bars, seed variance, and judge-specific ablations weakens the robustness claims. The current manuscript reports point estimates only. In the revision we will add error bars for the main results (computed from available repeated evaluations), report observed variance, and include an ablation that isolates the judge by comparing against alternative reward formulations. Due to the high cost of live-web RL runs, full multi-seed experiments are resource-intensive, so we will provide all feasible statistical controls and ablations rather than exhaustive ones.","revision_made":"partial","referee_comment":"[Table 1 / Results] Table 1 / Results: The reported success rates (67.0% and 64.0%) are presented without error bars, variance across seeds, or ablations isolating the contribution of the success judge versus other pipeline components. This omission is load-bearing because the central claim is that online RL succeeds with small data volumes; the absence of these controls leaves open the possibility that results are sensitive to judging noise or website state variability."}],"tokens_in":1640,"tokens_out":526,"duration_ms":24974,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main contribution is an end-to-end open framework that runs online multi-turn RL directly on real websites for visual agents. It includes live-browser infrastructure, multimodal context management, a trajectory-level success judge, and policy optimization. This combination has not been delivered together in prior open work, which mostly stayed with supervised training on static trajectory collections.\n\nThey demonstrate the pipeline works in practice. Starting from 0.4K initialization trajectories they run 2.2K open-ended RL tasks and produce a 4B model that hits 67% on Online-Mind2Web and 64% on DeepShop. Those numbers beat other open agents of similar or larger size and stay close to some proprietary systems. They also release the data, models, and code, and include some analysis of design choices and how RL changes reasoning.\n\nThe soft spot is the trajectory-level judge. On dynamic live sites, judging success at the full-trajectory level can easily introduce noise through false positives on partial work, missed non-canonical paths, or inconsistent verdicts across changing page states. The reported gains with such limited data volumes depend on that signal being clean enough for stable credit assignment. The paper would be stronger if it included direct validation of judge accuracy or ablations that test sensitivity to judgment errors.\n\nThis is for researchers working on web agents and RL for long-horizon visual tasks who want to move past curated datasets. A reader looking for a practical open starting point will get value from the released artifacts and the empirical setup. It deserves a serious referee because the problem it targets is real and the implementation is open enough to build on.","headline":"OpenWebRL packages a full open pipeline for online multi-turn RL on live visual websites and shows it can reach competitive numbers from small initialization data.","tokens_in":2487,"tokens_out":403,"would_cite":true,"duration_ms":17890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Online multi-turn RL on live websites trains a 4B visual web agent to 67% success with 0.4K init trajectories and 2.2K tasks.","keywords":["visual web agents","online reinforcement learning","multi-turn RL","live-browser infrastructure","trajectory-level success judging","OpenWebRL","web agent training","dynamic websites"],"falsifier":"If the trained 4B agent shows success rates below 50% when evaluated on a fresh set of live websites whose interfaces were not encountered during the 2.2K RL tasks, the claim that the online RL pipeline produces effective and stable policies would be falsified.","tokens_in":2762,"feed_emoji":"🌐","tokens_out":786,"duration_ms":21778,"temperature":0.7,"pith_summary":"The paper establishes that online multi-turn reinforcement learning can train capable visual web agents directly on dynamic real-world websites without depending on large static collections of curated trajectories. It presents an open framework covering live-browser infrastructure, supervised initialization, multimodal context handling, trajectory-level success judging, and multi-turn policy optimization to generate usable reward signals. With this setup the resulting 4B model reaches 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger size and remaining competitive with proprietary systems. A reader would care because the approach removes the main scalability bottleneck of expensive demonstration collection and shows that modest numbers of open-ended tasks suffice for effective training on the live web.","feed_headline":"Online RL trains 4B web agent to 67% success with 2K tasks","feed_subtitle":"Trajectory-level rewards from live browsers let the model beat prior open agents and compete with closed systems on real dynamic sites.","key_machinery":"The OpenWebRL framework whose live-browser infrastructure and trajectory-level success judging supply the reward signals that support stable multi-turn policy optimization on changing websites.","core_discovery":"OpenWebRL is an open framework for training visual web agents via online multi-turn RL on real websites. It supplies the full pipeline of scalable live-browser infrastructure, supervised initialization, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization. Trained with only 0.4K initialization trajectories and 2.2K open-ended RL tasks, OpenWebRL-4B achieves 67.0% success on Online-Mind2Web and 64.0% on DeepShop, establishing new open-source state-of-the-art results while remaining competitive with proprietary systems such as OpenAI CUA and Gemini CUA. The work also examines the design choices that enable effective RL and analyzes how t","pith_inferences":["The same live-environment RL loop could be adapted to train agents for other interactive interfaces such as mobile apps.","The reported data efficiency suggests the method could lower the compute and annotation cost of building new web agents in resource-constrained settings.","Further experiments that vary the judging granularity might reveal whether finer-grained rewards would accelerate learning on complex sites."],"forward_implications":["Visual web agents can be trained scalably without collecting large curated demonstration datasets.","Online RL directly on live sites improves agentic reasoning beyond what supervised post-training alone achieves.","Modest numbers of open-ended tasks suffice for effective multi-turn optimization when paired with trajectory-level rewards.","Open-source agents can reach performance levels competitive with proprietary systems through this training route."],"fun_headline_variants":["OpenWebRL trains 4B agent with 2K RL tasks to 67% success","4B OpenWebRL agent reaches 67% success via online multi-turn RL","OpenWebRL enables 4B visual agent to 64% on DeepShop benchmark","OpenWebRL-4B achieves 67% success on live web with online RL"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Trajectory-level success judging on live browsers supplies reward signals with low enough noise to support stable multi-turn policy optimization on dynamic real-world websites.","fun_headline_variants_meta":{"raw":{"variants":["OpenWebRL trains 4B agent with 2K RL tasks to 67% success","4B OpenWebRL agent reaches 67% success via online multi-turn RL","OpenWebRL enables 4B visual agent to 64% on DeepShop benchmark","OpenWebRL-4B achieves 67% success on live web with online RL"]},"model":"grok-4.3","cost_usd":0.015627,"raw_usage":{"total_tokens":6702,"prompt_tokens":849,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":156265500,"prompt_tokens_details":{"text_tokens":849,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5763,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":849,"tokens_out":90,"duration_ms":39733,"temperature":1.0,"reasoning_tokens":5763,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T15:44:25.939991+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If the trained 4B agent shows success rates below 50% when evaluated on a fresh set of live websites whose interfaces were not encountered during the 2.2K RL tasks, the claim that the online RL pipeline produces effective and stable policies would be falsified.","supporting_citations":[],"review_version":1}