{"id":"b9dd5b08-68da-44db-ace4-bb72c3696eb2","arxiv_id":"2607.07508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"SAO stabilizes asynchronous RL for LLMs by replacing group-wise sampling with single-rollout updates, token-level importance sampling, and targeted value-model training, outperforming GRPO on reasoning and coding benchmarks.","lead":"This paper introduces SAO, a method that replaces group-wise sampling with single-rollout updates for asynchronous reinforcement learning of LLMs, using token-level clipping and value-model training to maintain stability. A smart generalist might read it because it enables more efficient RL training for agentic tasks and is deployed in the GLM-5.2 model.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"SAO vs GRPO comparison confounds single-rollout effect with 8× greater prompt diversity per batch; the performance gap may stem from data diversity rather than the proposed mechanisms.","rationale":"The reader correctly identifies value model quality as an unquantified assumption, and this is a legitimate reproducibility concern. However, the more load-bearing issue for the central claim is the prompt diversity confound in the main comparison. The reader's concern affects generalization (will it work with less value pretraining?), while the diversity confound affects interpretation of the results as presented (is the gain from the proposed mechanism or from seeing more data?). Both concerns support a CONDITIONAL verdict. The paper's ablations are thorough on value model components (Table 4, Figure 4) but do not address the diversity confound. The DIS strategy, skip-observation GAE, and frozen-attention value training are well-motivated and the training stability evidence is convincing. The concern is specifically about whether the performance attribution is correct, not whether the method works at all. No code or data release is mentioned, making independent verification of this confound difficult. The GLM-5.2 deployment claim is cited without metrics, which is a minor additional concern but not load-bearing for the methodological claims. Single-backbone evaluation (Qwen3-30B-A3B only) is acknowledged in limitations and is a standard caveat.","tokens_in":13596,"tokens_out":5661,"duration_ms":424327,"concrete_test":"Train GRPO (w/ DIS) with 128 unique prompts × 8 rollouts per batch (batch size 1024, matching SAO's prompt diversity) for the same 1000 steps on AIME2025. If SAO still outperforms this variant by a similar margin (~4 points), the gain is not from prompt diversity and the paper's attribution to single-rollout mechanisms holds. If the gap narrows to ≤1 point, prompt diversity is the primary driver and the mechanistic claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that single-rollout sampling (combined with DIS and value-model improvements) reduces off-policy effects and improves generalization relative to GRPO. However, the experimental comparison confounds the single-rollout mechanism with prompt diversity. SAO uses 128 prompts × 1 rollout = 128 unique prompts per batch (§4.1: 'batch size of 128, a group size of 1'), while GRPO uses 16 prompts × 8 rollouts = 16 unique prompts per batch (§4.1: 'each training batch contains 16 prompts with 8 rollout samples per prompt, yielding the same batch size of 128'). Both have 128 total rollouts, but SAO sees 8× more unique prompts per step and over the full 1000-step run. The performance gap between SAO (97.3 AIME2025) and GRPO w/ DIS (93.5) could be partially or entirely attributable to this 8× data diversity advantage rather than to reduced off-policy effects or better value-based advantage estimation. The paper does not discuss this alternative explanation. The ablation with running-mean baseline (79.8) shows the value model is necessary, but does not isolate whether the gain over GRPO comes from single-rollout + value model or from 8× prompt diversity + value model. This matters because the paper attributes the improvement to reduced off-policy effects (§3.2), but the actual mechanism could be simpler: more diverse training data per step.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes Single-rollout Asynchronous Optimization (SAO), a method for stabilizing asynchronous RL training of LLMs on agentic tasks. SAO replaces GRPO's group-wise sampling with one rollout per prompt, introduces Direct Double-Sided Importance Sampling (DIS) that bypasses tracking old-policy checkpoints, and proposes several value-model training strategies (faster critic updates, frozen attention, skip-observation GAE, scaled pretraining). Experiments on Qwen3-30B-A3B across five benchmarks (AIME2025, BeyondAIME, HMMT Nov 2025, IMOAnswerBench, SWE-Bench Verified) show SAO outperforming GRPO variants, and an online learning simulation demonstrates adaptability to non-stationary rewards. The method is also reported as deployed in GLM-5.2 training.","tokens_in":14400,"tokens_out":1223,"duration_ms":300704,"significance":"The paper addresses a practically important problem—training stability in asynchronous RL for long-horizon agentic LLM tasks—and proposes a coherent set of mechanisms. The DIS strategy that directly uses rollout log-probabilities (dropping old-policy tracking) is a clean engineering contribution. The skip-observation GAE derivation for multi-turn agentic trajectories is well-motivated. Ablations in Tables 3–4 and training dynamics in Figure 4 provide useful evidence for each design choice. The online learning simulation (§4.5) is a nice addition showing the value of single-rollout in non-stationary settings. However, the central comparison between SAO and GRPO has a confound (see major comments) that weakens the causal attribution of gains to the proposed mechanisms.","major_comments":[{"comment":"§4.1, Tables 1–2: The SAO vs. GRPO comparison confounds single-rollout sampling with prompt diversity. SAO uses 128 prompts × 1 rollout = 128 unique prompts per batch, while GRPO uses 16 prompts × 8 rollouts = 16 unique prompts per batch (§4.1). Both have 128 total rollouts, but SAO sees 8× more unique prompts per step. The performance gap between SAO (97.3 AIME2025) and GRPO w/ DIS (93.5) could be partially or entirely attributable to greater data diversity rather than reduced off-policy effects or value-based advantage estimation. The paper attributes improvement to reduced off-policy effects (§3.2) but does not discuss this alternative explanation. A controlled comparison—e.g., GRPO with 128 prompts × 1 rollout using a group-free baseline, or SAO with 16 prompts × 8 rollouts—would isolate the single-rollout mechanism's contribution. Without this, the central claim that single-rollout ","section":null},{"comment":"§3.2 ('Scaling Value Pretraining'): The value pretraining corpus is described only as 'significantly increasing the scale' with no quantification (number of samples, tokens, or training steps). Since the paper explicitly states that 'the cold start problem in value estimation is a major bottleneck' and that scaling is 'essential,' this parameter is load-bearing for the method's reproducibility. Please provide concrete numbers and, if feasible, a sensitivity analysis showing how value pretraining scale affects final performance.","section":null},{"comment":"Abstract and §4.5: The GLM-5.2 deployment is cited as evidence of SAO's effectiveness ('SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model'), but no metrics are provided for this deployment. If the deployment is meant to support the method's claims, quantitative results (even limited) should be included; otherwise, the statement should be softened to acknowledge that GLM-5.2 used SAO without presenting it as validating evidence.","section":null}],"minor_comments":[{"comment":"Tables 3 and 4 appear to contain overlapping information (both report ablation results for value model training strategy and critic update frequency on AIME2025 and BeyondAIME). Consider merging into a single table.","section":null},{"comment":"§4.1: The clipping parameters differ between math (ε_low=0.3, ε_high=5.0) and coding (ε_low=0.8, ε_high=3.0) but no justification is provided for these choices. A brief note on how these were selected would strengthen the presentation.","section":null},{"comment":"§4.5: The online learning simulation uses only 3 stylistic phases. While illustrative, a more comprehensive evaluation with additional shifts or quantitative adaptation-speed metrics would strengthen the claim of effectiveness in non-stationary environments.","section":null},{"comment":"Appendix A.2 mentions SPO (Xu and Ding, 2025) as a single-rollout baseline but does not include it in the main results tables. Including SPO in the ablation would provide a more complete comparison to existing single-rollout methods.","section":null},{"comment":"§3.1: The relationship between DIS and the IcePop mechanism (Team et al., 2025) is mentioned but not fully elaborated. A more detailed comparison of what is shared and what differs would help readers position the contribution.","section":null},{"comment":"Figure 1: The y-axis label and tick marks are small and difficult to read. Consider enlarging or splitting into separate panels for reasoning and coding benchmarks.","section":null}],"recommendation":"major_revision","confidential_remarks":"The prompt-diversity confound (major comment 1) is the most serious concern. It is possible that the gains over GRPO come substantially from seeing 8× more unique prompts per step, which is a simpler explanation than the off-policy reduction mechanisms the paper emphasizes. The authors should be asked to address this directly, either with a controlled experiment or by acknowledging the confound and reframing the contribution. The GLM-5.2 self-citation without metrics is also worth noting to the editor as it inflates the perceived evidence base."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises three major points: (1) a confound between single-rollout sampling and prompt diversity in the SAO vs. GRPO comparison, (2) missing quantification of the value pretraining corpus, and (3) unsupported GLM-5.2 deployment claims. We address each below and commit to revisions for all three.","responses":[{"response":"The referee correctly identifies a genuine confound in our experimental design. SAO sees 8× more unique prompts per batch than GRPO (128 vs. 16), and the performance gap between SAO and GRPO w/ DIS could indeed be partially attributable to prompt diversity rather than the single-rollout mechanism itself. We agree this weakens causal attribution and must be addressed. In the revision, we will add a controlled experiment: GRPO with 128 prompts × 1 rollout (using a group-free baseline such as running-mean for advantage estimation) to isolate the effect of prompt diversity from the single-rollout mechanism. This will show whether the gains persist when both methods see the same number of unique prompts. We will also add an explicit discussion of this confound in §4.1 and acknowledge prompt diversity as a contributing factor. We cannot fully resolve this without the additional experiment, so we are committed to running it.","revision_made":"yes","referee_comment":"§4.1, Tables 1–2: The SAO vs. GRPO comparison confounds single-rollout sampling with prompt diversity. SAO uses 128 prompts × 1 rollout = 128 unique prompts per batch, while GRPO uses 16 prompts × 8 rollouts = 16 unique prompts per batch. The performance gap could be attributable to greater data diversity rather than reduced off-policy effects or value-based advantage estimation. A controlled comparison would isolate the single-rollout mechanism's contribution."},{"response":"The referee is correct that this is a reproducibility gap. The manuscript describes the value pretraining scaling as essential but provides no concrete numbers, which is insufficient for reproducibility. We will add specific details in the revision: the number of pretraining samples, total token count, and training steps for the value pretraining corpus. Regarding the sensitivity analysis, we agree this would strengthen the paper's claims about the importance of scaling. We will include a sensitivity analysis with at least two pretraining scales (e.g., 1× and the full scale used) showing the effect on final performance, subject to compute availability. If a full sensitivity sweep is not feasible within the revision timeline, we will at minimum provide the concrete numbers and a comparison between the pretraining scale used and a smaller-scale baseline.","revision_made":"partial","referee_comment":"§3.2 ('Scaling Value Pretraining'): The value pretraining corpus is described only as 'significantly increasing the scale' with no quantification (number of samples, tokens, or training steps). Since the paper states that the cold start problem is a major bottleneck and scaling is essential, this parameter is load-bearing for reproducibility. Please provide concrete numbers and a sensitivity analysis."},{"response":"The referee is right that the current phrasing presents GLM-5.2 deployment as validating evidence without providing supporting metrics. We have two options, and we will pursue both where feasible. First, if we can release even limited quantitative results from the GLM-5.2 training (e.g., training stability metrics or benchmark comparisons with and without SAO), we will include them. Second, regardless of whether metrics can be released, we will soften the statement in both the abstract and §4.5 to clarify that GLM-5.2 used SAO in its training pipeline, without presenting this as quantitative validation of the method. The revised wording will distinguish between 'SAO was deployed in production' (a statement about engineering applicability) and 'SAO improved GLM-5.2 performance by X%' (a claim requiring metrics, which we will either support or omit).","revision_made":"yes","referee_comment":"Abstract and §4.5: The GLM-5.2 deployment is cited as evidence of SAO's effectiveness but no metrics are provided. If the deployment is meant to support the method's claims, quantitative results should be included; otherwise the statement should be softened."}],"tokens_in":13623,"tokens_out":913,"duration_ms":127160,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper proposes replacing GRPO's group-wise sampling with single-rollout sampling (1 rollout per prompt instead of 8) in asynchronous RL, stabilized by a direct double-sided token-level clipping strategy (DIS), frozen-attention value training, faster critic updates, and a skip-observation GAE for multi-turn agentic trajectories. The results on Qwen3-30B-A3B are consistent and the ablations are thorough for what they cover. The GLM-5.2 deployment is mentioned but unquantified — take it as a credibility signal, not evidence.","headline":"SAO: single-rollout async RL with value-based advantages — solid empirical work with a real confound in the main comparison","tokens_in":14339,"tokens_out":637,"would_cite":false,"duration_ms":54027,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"One rollout per prompt stabilizes async RL for LLMs","keywords":[],"falsifier":"Train SAO with value models of varying pretraining corpus sizes and measure whether single-rollout training stability degrades below a threshold of value model quality, and whether frozen-attention training remains stabilizing across different backbone architectures.","tokens_in":13694,"feed_emoji":"🔄","tokens_out":1118,"duration_ms":141982,"temperature":0.7,"pith_summary":"The paper presents Single-rollout Asynchronous Optimization (SAO), a method for training large language models with reinforcement learning in asynchronous settings where rollouts of varying length arrive continuously. The core claim is that replacing group-wise sampling (as used in GRPO, which waits for multiple responses per prompt) with single-rollout sampling (one response per prompt, trained immediately upon completion) reduces off-policy drift and is structurally better suited to agentic and online tasks. To make single-rollout training stable despite its inherent high variance, the paper introduces three supporting mechanisms: direct double-sided token-level importance sampling that bypasses tracking historical policy checkpoints by using rollout log-probabilities directly with strict clipping; a value model trained with frozen attention layers and twice-per-step updates to track the rapidly changing policy; and a skip-observation GAE estimator that computes advantages across action-to-action boundaries in multi-turn agent trajectories, avoiding noise from environment-generated tokens. The paper claims SAO trains stably for approximately one thousand steps and consistently outperforms GRPO and its variants on agentic coding (SWE-Bench Verified) and math reasoning benchmarks (AIME2025, BeyondAIME, HMMT, IMOAnswerBench).","feed_headline":"One rollout per prompt makes async LLM training stable","feed_subtitle":"Replacing group sampling with single-rollout updates plus aggressive clipping and a frozen-attention value model beats GRPO on coding and","key_machinery":"SAO (Single-rollout Asynchronous Optimization) combines: (1) Direct Double-Sided Importance Sampling (DIS) — computes the probability ratio directly between the current policy and the rollout policy using saved log-probabilities, then masks tokens outside a trust region [1−ε_l, 1+ε_h] from gradient computation entirely; (2) Faster Value Update — the value model is updated K=2 times per policy gradient step; (3) Frozen-Attention Value Training — only MoE projection layers are optimized in the value model while attention weights remain fixed; (4) Skip-Observation Token-level GAE — modifies the Bellman target to bridge across environment feedback tokens, linking action-end values directly to下一个","core_discovery":"The central finding is that the structural mismatch between group-wise sampling and asynchronous training can be resolved by switching to single-rollout sampling, provided that the resulting variance is controlled through a combination of aggressive token-level clipping (using rollout probabilities directly rather than maintaining old-policy checkpoints), a value model whose attention layers are frozen during RL training to reduce gradient instability, and a GAE formulation that skips environment-observation tokens in multi-turn trajectories. The paper demonstrates that this combination enables stable training over ~1000 steps and outperforms GRPO variants, with the single-rollout approach n","pith_inferences":["The paper does not provide sensitivity analysis on value model pretraining corpus scale or quality, despite identifying it as a critical bottleneck. A natural testable extension would be to measure SAO's stability and final performance as a function of value pretraining data quantity, which would reveal whether there is a threshold below which single-rollout training becomes impractical.","The experiments use a single backbone (Qwen3-30B-A3B). Whether the frozen-attention finding generalizes to dense (non-MoE) architectures is unclear, since the stability observation is specifically attributed to MoE layers remaining stable while attention layers do not.","The online learning simulation uses binary rewards from an LLM judge with discrete stylistic shifts. Real-world non-stationary environments typically involve continuous, gradual preference drift with noisy rewards, which may stress the value model differently than the clean phase transitions tested here.","The claim that single-rollout sampling reduces off-policy effects is somewhat counterintuitive — a single sample per prompt provides less information about the reward landscape, not more. The reduction in off-policy effects likely comes from reduced latency (faster training updates) rather than from the single-rollout design itself, and disentangling these two factors would clarify the mechanism."],"forward_implications":["Asynchronous RL for LLMs can move beyond throughput-focused systems to achieve genuine training stability and task effectiveness, potentially making long-horizon agentic training more practical at scale.","The finding that single-rollout RL is naturally suited to online learning with non-stationary rewards suggests a path toward models that adapt to shifting user preferences or evolving environments in real time, without requiring group-based feedback.","The skip-observation GAE formulation could be applied more broadly to any RL setting where agent actions are interleaved with external observations, such as tool use, web browsing, or embodied agents.","The demonstration that frozen-attention value training stabilizes optimization while full-parameter training destabilizes it raises questions about which components of pretrained models are best left fixed during RL post-training."],"fun_headline_variants":["Single-rollout sampling fixes async RL stability for LLM agents","Frozen-attention value model and double-side clipping stabilize async agentic RL","One rollout per prompt beats GRPO on coding and reasoning benchmarks","Token-level clipping enables stable single-rollout async RL over 1000 steps","SAO resolves off-policy drift in asynchronous LLM training via single-rollout updates"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method depends on the value model being accurate enough to compensate for the high variance of single-sample gradient estimates. The paper acknowledges this dependency but does not quantify how good the value model needs to be or provide sensitivity analysis on value model quality beyond the ablations shown.","fun_headline_variants_meta":{"raw":{"variants":["Single-rollout sampling fixes async RL stability for LLM agents","Frozen-attention value model and double-side clipping stabilize async agentic RL","One rollout per prompt beats GRPO on coding and reasoning benchmarks","Token-level clipping enables stable single-rollout async RL over 1000 steps","SAO resolves off-policy drift in asynchronous LLM training via single-rollout updates"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":718,"prompt_tokens":624,"completion_tokens":94,"prompt_tokens_details":null},"tokens_in":624,"tokens_out":94,"duration_ms":27963,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T08:31:41.003222+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Train SAO with value models of varying pretraining corpus sizes and measure whether single-rollout training stability degrades below a threshold of value model quality, and whether frozen-attention training remains stabilizing across different backbone architectures.","supporting_citations":[],"review_version":1}