{"id":"075b7f77-d845-4daf-ba85-fe95c94b0660","arxiv_id":"2608.09805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Generating rollouts from multiple weight-perturbed versions of the policy, pooled inside a GRPO group, improves RLVR performance over standard GRPO on math and code benchmarks.","lead":"This paper introduces 3PO, a family of methods that add learned noise to a language model's weights during reinforcement learning, so rollouts come from slightly different versions of the model. The authors report that this parameter-space exploration beats standard GRPO on math and code benchmarks, especially hard problems, at nearly the same FLOPs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"C3PO's central results hinge on the Seq-MIS correction, whose estimator is never derived; Appendix I shows training collapses without it, so the parameter-exploration claim is conditional at best.","rationale":"The reader's weakest assumption identifies the Seq-MIS correction as the key unsupported dependency, and my reading confirms this: without the correction, C3PO does not learn (Appendix I), and the paper gives no derivation of the corrected estimator when rollouts come from N different posterior samples. This is the most load-bearing concern because it sits directly between the method definition and the headline empirical results. If the correction is ad hoc, the C3PO improvements could be an artifact of the masking heuristic rather than of parameter-space exploration. At the same time, the paper has independent supporting evidence: a 3-seed Olmo3 comparison (p = 0.027), an isotropic-noise control suggesting the learned Hessian structure matters, and consistent gains on code generation. These support the conditional verdict but do not resolve the estimator-correctness gap. I therefore agree with the reader and recommend keeping the CONDITIONAL verdict rather than upgrading or rejecting it.","tokens_in":24583,"tokens_out":4206,"duration_ms":39785,"concrete_test":"Re-derive the C3PO loss from the stated objective: replace the single-\\pi_\\text{old} ratio in Eq. 1 with per-rollout importance ratios \\prod_t \\pi_\\theta(y_t^i)/\\pi_{\\hat\\theta_{n(i)}}(y_t^i | x, y_{<t}^i), and check whether the expectation over z_1,...,z_N ~ N(0,I) matches the objective minimized by IVON in Eq. 2 with L_GRPO as the loss. Then run C3PO on Olmo3 in two configurations: (a) with the exact per-rollout correction and no mask, and (b) with the current Seq-MIS mask. Compare training reward and final AIME pass@1. If (a) trains stably and matches (b), the mask is replaceable; if (a) collapses, the mask itself is load-bearing and must be theoretically justified rather than cited to a blog post.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix I and Figure 12 (Middle) state that C3PO training is essentially flat without the Seq-MIS correction, and Appendix A describes the implemented correction as masking sequence-level importance weights that fall outside [0.5, 2.0]. This correction is load-bearing, but the paper provides no derivation tying it to Eq. 1, Eq. 2, or Algorithm 2. In C3PO, each rollout in a group is generated by a different posterior sample \\hat\\theta_n, so the GRPO ratio in Eq. 1 must be defined relative to that rollout's own generator: a proper importance ratio would be \\prod_t \\pi_\\theta(y_t^i | x, y_{<t}^i) / \\pi_{\\hat\\theta_{n(i)}}(y_t^i | x, y_{<t}^i), and the advantage baseline is computed over a mixture of N policies. The paper does not specify what \\pi_\\text{old} is in Eq. 1 under C3PO, nor does it justify that the masked, sequence-level correction remains an unbiased or otherwise controlled estimator of the intended GRPO objective. Given that the main C3PO results explicitly depend on this correction, the empirical gains could stem from the stabilization heuristic itself rather than from parameter-space exploration as the mechanism. The concern is not that the results are fabricated or that Seq-MIS is necessarily wrong; it is that a central, load-bearing component of the method is unsupported at the level of estimator correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Perturbed Parameter Policy Optimization (3PO), a family of methods that perform parameter-space exploration for reinforcement learning with verifiable rewards (RLVR) by sampling rollout policies from an approximate posterior learned with the IVON optimizer. Three variants are studied: B3PO (one weight perturbation per gradient step), M3PO (multiple perturbations with per-perturbation advantage groups), and C3PO (chunking each GRPO group across N perturbed policies and computing advantages over the mixed group). Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation report that 3PO methods, especially C3PO, improve average Pass@1 over GRPO, rescue more zero-advantage groups, and produce fewer malformed or incorrect rollouts. The paper includes ablations of the effective sample size lambda, the number of Monte Carlo samples M, the chunk size N, and the initial Hessian, and it identifies a Seq-MIS correction as necessary for C3PO training stability.","tokens_in":24884,"tokens_out":5405,"duration_ms":48272,"significance":"If the reported gains are robust, the paper makes a useful contribution: it demonstrates a compute-efficient exploration lever that is orthogonal to action-space methods and can be applied to off-the-shelf checkpoints. The manuscript has several concrete strengths: the 3-seed paired comparison of C3PO versus GRPO on Olmo3 (Appendix G), the consistent code-generation results (Section 4.4), and the isotropic-noise control (Appendix H) that indicates the learned Hessian, not merely added noise, is responsible for the late-training improvement. The public release of the implementation is also a strength. However, the central C3PO claim depends on a stabilization heuristic whose estimator is never specified, and much of the evidence consists of single runs with hyperparameters selected on the evaluation benchmarks themselves. These issues are load-bearing for the main claim that parameter-space exploration, rather than the stabilization heuristic or selection artifact, drives the improvements.","major_comments":[{"comment":"The Seq-MIS correction is load-bearing but is never defined as an estimator. In C3PO, each rollout i in a group is generated by a different perturbed policy pi_{theta_hat_{n(i)}}, so the importance ratio in Eq. (1) must be defined relative to that rollout's own generator, e.g., R_{i,t} = pi_theta(y_t^i | x, y_{<t}^i) / pi_{theta_hat_{n(i)}}(y_t^i | x, y_{<t}^i). The paper does not state what pi_old is in Eq. (1) under C3PO, and Algorithm 2 only says 'cf. Eq. 1 with Seq-MIS correction' while Appendix A describes masking sequence-level importance weights outside [0.5, 2.0]. This is not a presentation quibble: Appendix I reports that training reward stays flat without the correction, so all C3PO results depend on it. Please provide the derivation of the corrected loss, state its target objective, and validate that the masked sequence-level correction is a controlled estimator, for example by comparing against per-token importance weighting with the correct per-rollout denominator, noise replay, or ablation of the clip bounds.","section":"Section 3.2, Eq. (1), Algorithm 2, Appendix I"},{"comment":"The main hyperparameters are tuned on the same benchmarks that form Table 1. The effective sample size lambda, the chunk size N, and the Monte Carlo count M are swept in Section 5.1-5.3 on the Olmo3 math benchmarks, and the selected values (lambda=1e9, N=4, M=4) are then used in the headline comparison. For C3PO, lambda=1e9 is chosen after observing collapse at 1e8 and a lower ceiling at 1e10, so part of the reported advantage over GRPO is a selection artifact. The manuscript should either evaluate on held-out development benchmarks, report the default-configuration result, or quantify the selection effect by comparing the best-of-sweep result with the median or a prespecified configuration.","section":"Section 4.1, Section 5, Table 1"},{"comment":"Most entries in Table 1 are single runs; the only multi-seed comparison is C3PO versus GRPO on Olmo3 with three seeds (Appendix G). The claims that 'all 3PO variants outperform action-space baselines on average' and that 'C3PO has the highest average performance in both model families' therefore rest on single runs for Qwen2.5-Math and for the B3PO/M3PO comparisons. Please add at least three seeds for the key pairwise comparisons on both models, or clearly mark single-run cells and restrict the strength of the conclusions. The current paired test in Appendix G is positive but covers only one model and one method pair.","section":"Appendix G, Table 1"},{"comment":"The rescue and degeneracy comparison for M3PO is not matched on rollout budget: the footnote to Figure 3 states that GRPO's group is subsampled to 4 rollouts for its pairing with M3PO, while M3PO computes advantages on groups of 4 rollouts per perturbation and aggregates over M=4 perturbations, for a total of 16 rollouts. A GRPO group of 4 is more likely to contain no correct rollout than a group of 16, so the conclusion that 3PO 'rescues more zero-advantage groups than baselines' may be confounded by group size. Please report the comparison at matched total rollout counts or justify the subsampling.","section":"Section 4.3, Figure 3"}],"minor_comments":[{"comment":"There are typos in the first sentence: 'Hoewever, bcause' should be 'However, because'.","section":"Appendix G"},{"comment":"The sentence 'Using IVON further improves the quality of these trajectories by learning the distribution over policies jointly during training (Eq. 4)' appears twice in the same paragraph; please remove the duplicate.","section":"Section 4.3"},{"comment":"The text describes the no-Seq-MIS run as 'essentially flat' and the corrected run as 'steadily rising,' but it does not report numeric reward values or the number of steps. Please include quantitative endpoints so the reader can judge the magnitude of the effect.","section":"Appendix I, Figure 12"},{"comment":"The Seq-MIS correction is cited only to a personal blog post ([35]). Since the correction is load-bearing for C3PO, the manuscript should either provide a self-contained description and derivation or cite a peer-reviewed or archival source that defines the method.","section":"Reference [35]"},{"comment":"The abstract and conclusion emphasize 'near-identical FLOPs cost,' but Section 7 and Table 3 report a 1.5x wall-clock overhead for M3PO and C3PO. This is acknowledged as a systems artifact, but the abstract would be more precise if it stated 'near-identical FLOPs' rather than implying comparable running time.","section":"Abstract and Section 7"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The empirical evidence is suggestive and the isotropic-noise control is a good scientific check, but the C3PO results are conditional on an underspecified Seq-MIS correction, and the headline comparisons are affected by hyperparameter selection on the evaluation benchmarks and by limited seeding. I do not see evidence of fabrication or intentional misrepresentation; the issues are methodological and can be addressed within the manuscript's scope. If the authors can supply a derivation and sanity check for Seq-MIS, add seeds for the key comparisons, and either use a development split or quantify the selection effect, the paper could become acceptable. Otherwise, the central mechanism claim remains unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the most complete case I've seen so far that weight-space noise is a real lever for RLVR, and C3PO is genuinely new. But the main results for C3PO ride on the Seq-MIS correction, and the paper never derives it. Appendix I shows training is flat without it, so that is not a minor footnote.\n\nWhat's good. The idea is straightforward and orthogonal to temperature and entropy baselines. C3PO partitions a GRPO group across N independently sampled IVON weights; that partition is not in the cited prior work. B3PO is acknowledged as a variant of concurrent work, which is honest. The experiments are broad for a 7B-only study: two model families, six math benchmarks, LiveCodeBench, plus ablations on lambda, N, M, Hessian init, and an isotropic-noise control that supports the learned-noise mechanism. They release code and checkpoints. The 3-seed paired test for C3PO vs GRPO on Olmo3 (p=0.027) and the code-generation curves give the headline claim some independent grounding. I also credit them for stating the 1.5x wall-clock overhead up front.\n\nWhere I worry. The Seq-MIS correction is load-bearing, and it is cited only to a blog post. In C3PO, each rollout in a group comes from a different sampled policy. The GRPO ratio in Eq. 1 normally divides by the policy that generated the rollout; the paper never says what pi_old is in that mixed setting, nor why masking sequence-level importance weights in [0.5, 2.0] gives a controlled estimator of the intended objective. Without a derivation or a sanity check, I can't tell whether C3PO works because of parameter-space exploration or because the masking is a useful stabilizer. That's a fixable gap, but it is central. Also, lambda and the main N/M choices are tuned on the evaluation benchmarks themselves, and most Table 1 cells are single runs. Those points are less severe because the multi-seed check exists, but they cap how much I'd trust the exact gains.\n\nBottom line: the paper deserves a serious referee. The gap is not disqualifying; the empirical pattern is consistent, the artifacts are real, and the authors are upfront about limitations. I would send it out and ask for a derived or validated Seq-MIS correction, held-out hyperparameter selection, and seeds for at least C3PO on both models. I'd bring it to reading group; there's a useful debate there about whether the correction is saving the method or the method is just well-paired with the correction.","headline":"C3PO is a genuinely new and plausible weight-space exploration method for RLVR, but its headline results lean on an under-derived Seq-MIS correction that the paper itself shows is necessary.","tokens_in":25480,"tokens_out":4087,"would_cite":true,"duration_ms":32727,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sampling rollout policies from a learned weight posterior improves LLM reinforcement learning, and the multi-policy chunked variant C3PO performs best.","keywords":["parameter-space exploration","RLVR","reinforcement learning with verifiable rewards","variational learning","IVON","GRPO","weight perturbation","LLM reasoning"],"falsifier":"Run C3PO on the Olmo3 math setup with Seq-MIS replaced by a provably unbiased estimator for groups generated from $N$ policies (for example, caching the $N$ weight samples and recomputing exact per-token ratios). If the training reward stays flat under that unbiased estimator, the stability carrying the main results is not attributable to parameter-space exploration; if training recovers, the correction is validated and the mechanism confirmed.","tokens_in":24349,"feed_emoji":"🎲","tokens_out":12157,"duration_ms":108539,"temperature":0.7,"pith_summary":"This paper argues that exploration in large-language-model reinforcement learning should happen in weight space, not only in token space. The authors introduce 3PO (Perturbed Parameter Policy Optimization), a family of methods that samples rollout policies from a learned approximate posterior over the model's weights, using the IVON variational optimizer, instead of only raising the sampling temperature during generation. Across two 7B models and math and code benchmarks, all 3PO variants beat standard GRPO (group-relative policy optimization) on average at near-identical FLOPs cost, and the best variant, C3PO, mixes rollouts from several perturbed policies inside each GRPO advantage group. The gains concentrate on harder benchmarks and come with fewer zero-advantage groups and fewer malformed rollouts, which the paper reads as evidence that parameter perturbations produce genuinely more useful exploration than action-space noise. If the claim holds, parameter-space exploration is a cheap, orthogonal lever that existing temperature-based RLVR recipes can adopt without retraining.","feed_headline":"Weight noise, not temperature, drives LLM exploration gains","feed_subtitle":"Sampling policies from a learned weight posterior beats GRPO on hard math and code benchmarks at equal FLOPs cost.","key_machinery":"The engine is the IVON optimizer, which maintains a diagonal Gaussian posterior over the model weights: at each step a perturbed point $\\hat{\\theta} = m + \\sigma \\odot z$ with $z \\sim \\mathcal{N}(0, I)$ is used for rollout generation, and the per-parameter variance $\\sigma^2 = 1/(\\lambda(h + \\delta))$ is inversely proportional to a Hessian $h$ scaled by an effective sample size $\\lambda$, so the amount of exploration is learned rather than fixed. The 3PO family wraps this in three rollout-grouping strategies, and the variant that carries the paper's best results is C3PO's chunked noising: $N$ independent weight samples each generate $G/N$ rollouts, and the GRPO advantage is computed over the full group of $G$ rollouts, maximizing diversity within the advantage calculation. The third load-bearing piece is the Seq-MIS correction, a sequence-level importance-sampling and masking fix applied to the GRPO ratio because the $N$ rollout generators differ from the policy being updated; Appendix I shows the training reward stays flat without it.","core_discovery":"The paper's central claim is that sampling weights from a learned posterior at rollout time provides an additional exploration lever for RLVR that action-space controls like temperature cannot provide, because weight noise can reorder the token distribution rather than merely flatten it. Concretely, the authors claim that drawing one or more policy samples from the IVON posterior and generating rollouts with them improves downstream pass@1 over GRPO across both model families, with C3PO — which partitions each GRPO group of $G$ rollouts across $N$ independently sampled weights and computes advantages over the aggregated group — having the highest average performance in both model families. They further claim that multi-sample parameter exploration rescues zero-advantage groups throughout training and produces fewer degenerate and incorrect rollouts than GRPO and the action-space baselines tested (temperature-based Polaris, entropy regularization, and KL-covariance weighting), and that the benefit is largest on harder benchmarks such as AIME and LiveCodeBench.","pith_inferences":["If rescuing dead groups is the mechanism, the benefit should grow with model scale: larger pretrained models have denser neighborhoods of competent solutions around their weights, so a fixed noise budget should yield more diverse valid rollouts — a testable prediction on larger checkpoints.","The stability that carries the main results depends on the Seq-MIS correction being a proper estimator when a GRPO group mixes $N$ policies, which the paper neither derives nor validates; a correct unbiased estimator for that setting would settle whether the gains are genuinely due to parameter-space exploration.","The paper's analysis suggests a practical monitoring habit: track zero-advantage group rescue and malformed-rollout rates rather than entropy, since entropy falls both when a policy focuses on high-reward tokens and when it collapses.","Adaptive noise schedules are left for future work, and the paper finds opposite $\\lambda$ preferences for M3PO and C3PO, so annealing the noise scale during training could recover gains that a fixed $\\lambda$ leaves on the table."],"forward_implications":["C3PO-style chunked noising is a drop-in change to the rollout side of an existing RLVR pipeline: it lifts average pass@1 over GRPO at near-identical per-step FLOPs, so practitioners can adopt it without touching the rest of the recipe.","Parameter-space exploration helps most where the pre-RL model is weakest, with the largest gains on AIME and LiveCodeBench; the same rollout budget yields more high-reward trajectories on hard tasks.","The learned noise matters, not noise itself: a matched isotropic-noise run plateaus like GRPO, so the Hessian-scaled variance and its effective sample size $\\lambda$ are what carry the improvement.","Multi-sample variants (M3PO, C3PO) keep rescuing zero-advantage groups into late training, while single-sample B3PO and all action-space baselines lose this ability, indicating that across-policy group diversity sustains the learning signal.","The roughly 1.5$\\times$ wall-clock overhead is a systems artifact of inefficient multi-model sampling; since all 3PO variants converge faster, their effective cost to reach a target score could fall below GRPO once that infrastructure improves."],"supporting_citations":[{"why":"Defines the GRPO objective and grouped-advantage scheme that every 3PO variant modifies.","marker":"[53]"},{"why":"Supplies the IVON variational optimizer whose learned posterior generates the weight perturbations.","marker":"[55]"},{"why":"Cited as the source of the Seq-MIS correction that C3PO's stability depends on; Appendix I shows training stalls without it.","marker":"[35]"},{"why":"Provides the token-level GRPO formulation, clip bounds, and the DAPO-MATH-17k training set used in all experiments.","marker":"[73]"},{"why":"Frames the training-inference mismatch that motivates applying the Seq-MIS correction to C3PO.","marker":"[71]"},{"why":"Polaris, the temperature-based action-space exploration baseline that 3PO is measured against.","marker":"[2]"},{"why":"Supplies the KL-covariance baseline and the entropy-decline analysis used to interpret exploration.","marker":"[11]"},{"why":"OLMo-3 is one of the two base models on which the empirical claims are tested.","marker":"[46]"},{"why":"Qwen2.5-Math is the second base model on which the empirical claims are tested.","marker":"[70]"}],"fun_headline_variants":["Weight noise beats temperature for LLM exploration","Sampling policies from a weight posterior improves RLVR","Parameter-space exploration beats GRPO on math and code","Weight sampling, not temperature, drives RL gains","Learn a weight posterior to explore better in RLVR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Seq-MIS correction is the right fix for C3PO's training-inference mismatch: the paper adopts it from a blog citation without deriving or validating it as a proper estimator when one GRPO group mixes $N$ different rollout policies, and without it C3PO's training reward stays flat.","fun_headline_variants_meta":{"raw":{"variants":["Weight noise beats temperature for LLM exploration","Sampling policies from a weight posterior improves RLVR","Parameter-space exploration beats GRPO on math and code","Weight sampling, not temperature, drives RL gains","Learn a weight posterior to explore better in RLVR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1281,"prompt_tokens":969,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":585,"tokens_out":312,"duration_ms":3724,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:26:43.565753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run C3PO on the Olmo3 math setup with Seq-MIS replaced by a provably unbiased estimator for groups generated from $N$ policies (for example, caching the $N$ weight samples and recomputing exact per-token ratios). If the training reward stays flat under that unbiased estimator, the stability carrying the main results is not attributable to parameter-space exploration; if training recovers, the correction is validated and the mechanism confirmed.","supporting_citations":[{"cited_title":"Variational learning is effective for large deep networks","cited_arxiv_id":null,"evidence_quote":"Supplies the IVON variational optimizer whose learned posterior generates the weight perturbations."},{"cited_title":"When speed kills stability: Demystifying RL collapse from the training-inference mismatch","cited_arxiv_id":null,"evidence_quote":"Cited as the source of the Seq-MIS correction that C3PO's stability depends on; Appendix I shows training stalls without it."},{"cited_title":"DAPO: An open-source LLM reinforcement learning system at scale","cited_arxiv_id":null,"evidence_quote":"Provides the token-level GRPO formulation, clip bounds, and the DAPO-MATH-17k training set used in all experiments."}],"review_version":1}