{"id":"e710fad0-0e38-4bcf-8b2d-487800a69a14","arxiv_id":"2509.08721","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Sharing rollouts between decentralized language models during RL post-training improved cumulative rewards by up to 94% over an isolated baseline in the paper's best configuration.","lead":"SAPO is a decentralized RL training algorithm that lets small language models share reasoning rollouts with each other during training. In controlled tests, the balanced sharing setup reportedly improved cumulative rewards by 94% over an isolated RL baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94% gain is confounded with zero-advantage filtering: Section 5 lets SAPO discard 0-advantage rollouts while forbidding the baseline from doing so, so sharing is not isolated as the cause.","rationale":"The reader's weakest assumption identifies the most load-bearing concern: the baseline is not allowed to filter zero-advantage samples, so the controlled comparison conflates experience sharing with a separate data-selection mechanism. This directly undermines the central causal claim that sharing rollouts drives the 94% improvement. I considered other issues, such as single-seed evaluation and the post hoc choice of the 4/4 configuration, but those are secondary: even with multiple seeds, the filtering confound remains. The absence of a held-out evaluation set is less decisive here because ReasoningGYM generates fresh instances online, making training reward a reasonable online performance measure. The proposed control experiment is straightforward and would settle whether the confound explains the result, so the appropriate verdict remains CONDITIONAL, matching the reader's assessment.","tokens_in":12404,"tokens_out":7960,"duration_ms":446811,"concrete_test":"Run a fifth controlled configuration, 8 local / 0 external with identical zero-advantage filtering, keeping the same GRPO hyperparameters and per-agent data budget as the existing experiments. Compare its cumulative reward after 2000 rounds against the 4 local / 4 external result of 1093.31. If the filtered baseline approaches that value, the headline gain is due to filtering rather than to sharing rollouts across agents; if it remains near 561.79, sharing is supported as the driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest empirical evidence, the 1093.31 vs 561.79 cumulative reward comparison in Section 5, does not isolate the effect of sharing rollouts. The authors state: 'SAPO gives agents the flexibility to subsample from a larger pool and remove the uninformative, 0-advantage samples, which the baseline cannot do.' Thus the 4 local / 4 external condition differs from the 8 local / 0 external baseline in two ways: it draws external rollouts from other agents' pools, and it filters out zero-advantage samples. In GRPO, if advantages are recomputed after filtering, dropping samples changes the group mean/std used to normalize the remaining advantages, so the optimization objective itself differs from the baseline; if advantages are computed before filtering, the discarded samples contribute no gradient, so a filtered baseline should match SAPO unless sharing adds something beyond filtering. The paper's own Section 6 finding that unfiltered uniform sampling over-represents zero-advantage rollouts and degrades performance suggests filtering may be the active ingredient. Without a baseline allowed the same filtering (and ideally a self-replay buffer to control for reuse of past data), the 94% improvement cannot be attributed to collective experience sharing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm in which each node trains its own policy while sharing decoded rollouts (plain-text question-answer samples) with other nodes. Each node assembles its training set by combining local rollouts with external rollouts sampled from the swarm, then performs a standard policy-gradient update (GRPO in the experiments). The authors report controlled experiments with eight Qwen2.5-0.5B agents trained on ReasoningGYM tasks for 2000 rounds, comparing no-sharing (8 local / 0 external) against three sharing configurations (6/2, 4/4, 2/6). They report that the 4/4 configuration achieves a cumulative reward of 1093.31 versus 561.79 for the baseline, a 94% improvement. They also present observations from a large-scale community demo involving thousands of nodes on heterogeneous hardware, where swarm-trained Qwen2.5-0.5B models outperform an isolated counterpart after roughly 175 normalized rounds, while stronger models show no clear benefit.","tokens_in":12628,"tokens_out":7331,"duration_ms":65285,"significance":"The core idea—sharing decoded rollouts across independently trained policies to bootstrap reasoning—is simple, plausible, and practically appealing. The decentralized, asynchronous formulation avoids the synchronization and communication bottlenecks of centralized distributed RL and naturally accommodates heterogeneous nodes. The paper is clearly written and the algorithm is straightforward to describe. If the empirical effect were established, SAPO would be a useful contribution to multi-agent RL post-training. However, the current evidence does not yet support the headline quantitative claim: the controlled comparison confounds experience sharing with zero-advantage filtering, the uncertainty quantification is inadequate (single run, min/max across agents as 'confidence intervals'), and the reported 94% is selected as the best of four configurations. The large-scale demo is uncontrolled and better treated as anecdotal. The conceptual contribution deserves further validation, but the experimental design must be strengthened before the central claim can be accepted.","major_comments":[{"comment":"The central comparison is confounded: the SAPO configurations filter zero-advantage rollouts before sampling external experience, while the baseline is explicitly not allowed this operation ('Note that SAPO gives agents the flexibility to subsample from a larger pool and remove the uninformative, 0-advantage samples, which the baseline cannot do.'). Consequently the headline comparison (1093.31 vs 561.79) differs in two variables: external rollouts and zero-advantage filtering. The authors' own Section 6 observation that unfiltered uniform sampling over-represents zero-advantage rollouts and degrades performance suggests filtering may be the active ingredient. A controlled baseline that is allowed the same zero-advantage filtering, and/or a SAPO variant that does not filter, is required to isolate the contribution of experience sharing.","section":"Section 5"},{"comment":"The paper reports no repeated seeds, and the 'confidence intervals' in Figure 2 are described as 'the minimum and maximum across agents,' which is a spread across agents in a single run, not an uncertainty estimate for the training procedure. The headline 94% improvement is the cumulative reward of one run of the best of four configurations, so the magnitude is also subject to selection over configurations. Multiple independent runs with proper standard errors (or full per-seed trajectory plots) are needed before a quantitative claim of 94% can be supported.","section":"Section 5, Figure 2"},{"comment":"The term 'rollout' is used ambiguously. The baseline '8 local / 0 external rollouts' could mean 8 question-level sets of 8 completions each, while 'samples J rollouts from the remaining ones' in the SAPO description suggests J individual completions. This ambiguity directly affects whether the total number of training samples is actually fixed across configurations, as claimed ('ensuring the total number of training samples was fixed across all setups'). The authors should define 'rollout' precisely (a (question, set-of-completions) object or a single completion) and report the per-agent update batch size in each configuration; otherwise the comparability of cumulative rewards is not established.","section":"Section 5"}],"minor_comments":[{"comment":"Algorithm 1 calls SampleSelf and SampleExternal without specifying the zero-advantage filtering procedure that is described only in Section 5. The algorithm description should match the experimental procedure, either by including the filter in the pseudocode or by stating that filtering is a configurable sampling strategy.","section":"Section 3.2 and Algorithm 1"},{"comment":"The label 'confidence intervals' in the Figure 2 caption is misleading; the shaded region is the min/max range across agents. Relabel it as 'range across agents' or use a proper statistical uncertainty estimate.","section":"Section 5, Figure 2"},{"comment":"The sentence 'statistical testing confirms that after roughly 175 normalized rounds, the swarm-trained variant outperforms its isolated counterpart' does not specify the test, the multiple-testing correction, or the data pre-processing. Provide details of the statistical procedure and the definition of 'adjusted p-value'.","section":"Section 6"},{"comment":"There are several typos and wording issues: 'adversely effected' should be 'adversely affected' (Section 5); 'Complimentary' should be 'Complementary' (Section 7); 'heterogenous' should be 'heterogeneous' (appears multiple times).","section":"Throughout"},{"comment":"The claim that experience sharing made a formatting reward unnecessary is anecdotal. If retained, it should be supported by a comparison or removed.","section":"Section 4.3"},{"comment":"The large-scale demo is uncontrolled: nodes voluntarily participate, differ in hardware and model configurations, and the comparison between swarm and isolated training may be affected by self-selection. The authors should frame these results as observational insights rather than evidence for the algorithm, which they mostly do, but the current wording ('Our findings show that swarm-based training can yield measurable gains') could be read too strongly.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clear and the decentralized rollouts idea is appealing, but the controlled experiment lacks the rigor expected for a quantitative claim: a single seed, a confounded baseline, and a best-of-four headline. The authors should be asked to provide the experimental code, configuration files, and multiple seeds, and to run the additional control experiments (filtered baseline and unfiltered SAPO). The community-demo section is promotional in tone and should be clearly separated from controlled evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper proposes SAPO, a decentralized RL post-training algorithm where nodes share decoded rollouts (text completions) and re-encode them for local policy updates. That is genuinely new in the LM post-training context. It is simple, clearly explained, and the design removes synchronization and hardware homogeneity assumptions, which is valuable. The controlled experiment keeps total samples fixed across conditions, which is a good touch, and the observation that heavy external sampling leads to oscillations is a real phenomenon worth reporting.\n\nThe problem is the headline claim. The 94% cumulative reward gain comes from comparing SAPO with 4 local/4 external rollouts to a baseline that cannot discard zero-advantage samples, while SAPO can. The paper states this explicitly in Section 5. As the stress-test note points out, that changes the objective in GRPO and means sharing is not isolated as the cause. The Section 6 demo, which uses no filtering and shows weaker gains (and performance degradation from uniform sampling), further suggests filtering may be the active ingredient. The experiment also appears to be single-seed; the reported \"confidence intervals\" are min/max across the eight agents, which is not a valid uncertainty estimate. Choosing the best of four configurations to headline the gain adds selection bias.\n\nNone of this is fatal to the core idea. The algorithm is coherent, and the paper is honest enough to state the asymmetry. But as published, the empirical evidence does not support the 94% claim. The fix is straightforward: run a baseline that filters zero-advantage rollouts (or, better, uses a self-replay buffer with the same subsampling), run multiple seeds, and report code and data. If the gain persists with a matched baseline, this becomes a useful contribution to distributed RL for small models.\n\nI would send this to peer review, because the idea deserves referee time and the authors have the infrastructure to run the right experiment. But the current version needs major revision before the central claim can be trusted. For me, it's a maybe for the reading group — the confound is a good teaching point.","headline":"The idea is promising but the headline 94% gain is confounded by zero-advantage filtering, and the experiment needs an equal-footing baseline and multiple seeds.","tokens_in":13245,"tokens_out":3106,"would_cite":false,"duration_ms":28636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Experience sharing across a swarm of small language models raises cumulative RL post-training reward by up to 94% over isolated training.","keywords":["decentralized reinforcement learning","experience sharing","language model post-training","swarm intelligence","GRPO","verifiable rewards","small language models","rollout sampling"],"falsifier":"Run the same eight-agent 2000-round experiment but allow the no-sharing baseline to discard zero-advantage rollouts from its own batch before the GRPO update, keeping the total number of training samples fixed; if its cumulative reward rises to roughly 1093.31, the measured benefit is attributable to sample filtering rather than to cross-agent experience sharing.","tokens_in":12176,"feed_emoji":"🤝","tokens_out":11612,"duration_ms":87051,"temperature":0.7,"pith_summary":"The paper introduces SAPO (Swarm sAmpling Policy Optimization), a fully decentralized and asynchronous reinforcement-learning post-training algorithm in which each node trains its own policy but can build its training batch by mixing its own rollouts with rollouts shared by other nodes. The central claim is that collective experience sharing lets useful discoveries—what the authors call 'aha moments'—propagate across the swarm, improving sample efficiency and final task reward. In controlled experiments with eight small language models trained on procedurally generated reasoning tasks over 2000 rounds, a balanced configuration mixing four local and four external rollouts per node accumulated 1093.31 cumulative reward versus 561.79 for a no-sharing baseline, a 94% improvement. The paper also reports that the benefit is not unlimited: relying too heavily on external rollouts produces oscillation and forgetting, and a large open demo indicates the benefit is model-dependent. If correct, the work offers a route to RL post-training on heterogeneous, loosely connected hardware without weight synchronization.","feed_headline":"Sharing rollouts lifts RL training reward by 94%","feed_subtitle":"A balanced mix of own and shared rollouts beats isolated training across eight small Qwen models.","key_machinery":"The operative mechanism is decoded-text rollout sharing with local training-batch assembly. Each node $n$ keeps its own policy $\\pi_n$ and, each round, samples questions, generates eight completions per question, and broadcasts a subset of them as (question, ground truth, rollouts, metadata) tuples. The node then builds its training set from $I_n$ self-rollouts and $J_n$ external rollouts drawn from the shared pool—discarding zero-advantage samples in these experiments—and performs a policy-gradient update (GRPO, no KL penalty, asymmetric clipping). Setting $J_n = 0$ recovers standard single-agent RL fine-tuning, making the local/external ratio the knob under study.","core_discovery":"The paper's central discovery is that in RL post-training for reasoning, sharing decoded rollouts among independently updated policies can substantially accelerate learning. Formally, each node broadcasts tuples of a question, its verifiable ground-truth answer, its rollout completions, and verification metadata; nodes then subsample $I_n$ local and $J_n$ external datapoints for GRPO updates, keeping total samples per round fixed. The empirical finding is a non-monotone effect: the 4-local/4-external mix yields 1093.31 cumulative reward, 94% above the 561.79 no-sharing baseline, while 2-local/6-external gives 945.87 with pronounced oscillations and 6-local/2-external gives 854.43. The authors attribute the gain to propagation of rare successful solution patterns and note that the flexibility to discard zero-advantage rollouts is built into SAPO, a flexibility the baseline lacks.","pith_inferences":["If zero-advantage filtering is the true driver, SAPO's contribution is better described as sample curation, and a single agent discarding its own low-reward rollouts could reproduce the gain; a filtered no-sharing control would settle this.","The oscillation in the 2-local/6-external configuration resembles coupled-oscillator dynamics, suggesting an adaptive scheme that tunes the external fraction during training as a testable stability fix.","Because rollouts are shared as plain text, the framework extends to heterogeneous modalities and even non-trained generators, offering an experimental route to study social learning in model populations.","The demo's model-dependent result hints that a policy's capacity gates the value of external experience; ranking agents by an 'absorptive capacity' measure could enable targeted sharing and fairer aggregation."],"forward_implications":["RL post-training can run over decentralized, heterogeneous, loosely connected compute without weight synchronization, lowering the infrastructure barrier for community and edge training.","Rare successful reasoning patterns ('aha moments') propagate across agents quickly, so formats and solution conventions spread without explicit reward shaping; the authors dropped their formatting reward after observing this.","There is a nontrivial optimal ratio of local to external rollouts, so practitioners should treat the mix as a hyperparameter, since too much external reliance degrades training stability.","The benefit of sharing appears strongest for mid-capacity models that absorb and re-emit diverse rollouts, while stronger models in the demo did not improve; sampling strategy and model capacity interact.","The paper predicts that with better-than-uniform sampling from the shared pool, even stronger models could benefit from swarm participation."],"supporting_citations":[{"why":"Establishes that pure RL post-training can elicit reasoning without supervised fine-tuning, the premise that motivates SAPO.","marker":"DeepSeek-AI et al., 2025"},{"why":"Provides GRPO, the policy-gradient rule every agent uses to update its policy.","marker":"Shao et al., 2024"},{"why":"Supplies the ReasoningGYM tasks and programmatic verifiers used in the controlled experiments and demo.","marker":"Stojanovski et al., 2025"},{"why":"Supplies the eight Qwen2.5-0.5B base models that form the controlled swarm.","marker":"QwenTeam, 2024"},{"why":"Motivates the training hyperparameters (no KL penalty, asymmetric clipping) used in the policy updates.","marker":"Yu et al., 2025"},{"why":"Supplies the GenRL backend that executes the decentralized experiments and the large open-source demo.","marker":"Gensyn, 2025"}],"fun_headline_variants":["Shared rollouts yield 94% higher RL reward","Collective rollout sharing accelerates LM RL training","SAPO: Decentralized RL with shared rollouts boosts rewards","Sharing RL rollouts boosts post-training reward by 94%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 94% gain could come entirely from the freedom to discard zero-advantage rollouts, because the no-sharing baseline is forced to keep all of its own samples while SAPO configurations filter a larger mixed pool.","fun_headline_variants_meta":{"raw":{"variants":["Shared rollouts yield 94% higher RL reward","Collective rollout sharing accelerates LM RL training","SAPO: Decentralized RL with shared rollouts boosts rewards","Sharing RL rollouts boosts post-training reward by 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2737,"prompt_tokens":972,"completion_tokens":1765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1697}},"tokens_in":588,"tokens_out":1765,"duration_ms":12070,"temperature":1.0,"reasoning_tokens":1697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:00:17.398827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight-agent 2000-round experiment but allow the no-sharing baseline to discard zero-advantage rollouts from its own batch before the GRPO update, keeping the total number of training samples fixed; if its cumulative reward rises to roughly 1093.31, the measured benefit is attributable to sample filtering rather than to cross-agent experience sharing.","supporting_citations":[],"review_version":1}