{"id":"a07b71be-a74c-4bcb-8357-88b8ab623d1b","arxiv_id":"2608.09271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SoftmaxGRPO replaces GRPO's z-score normalization with softmax advantages, giving bounded prompt weighting and improved scores on math and weak-reward reasoning tasks.","lead":"The paper introduces SoftmaxGRPO, a drop-in replacement for GRPO that uses temperature-scaled softmax advantages instead of z-score group normalization, keeping learning weights bounded on easy prompts. It reports consistent gains over GRPO on math and non-verifiable reasoning benchmarks, and derives an exact finite-group objective for binary rewards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The exact finite-M binary theorem is clean, but Tables 2–4 are produced by a clipped, KL-regularized PPO surrogate that the paper explicitly excludes from the theorem; the claimed reallocation and gains could come from the trust region rather than from SoftmaxGRPO's population objective.","rationale":"Good-faith reading: the theoretical core of SoftmaxGRPO is coherent. We re-derived the M=2 case: the expected update is A_c∇p = (c-1)/(c+1)∇p, matching ω=E[Δ_S]; the low-temperature limit reproduces Eq. (5); and the non-conservative one-form in Eq. (8) is a valid obstruction. So the exact finite-M binary-reward claim, the large-group log-MGF claim, and the stated limitation all check out. This is why the attack targets the experimental bridge, not the math.\n\nThe load-bearing gap is precisely the one the reader names: Section 4's clipped, KL-regularized surrogate is what generated every table, and the paper explicitly scopes the theorems to the unclipped on-policy objective. A careful reading also shows the paper handles this honestly—Section 4 and the 'Scope of claims' paragraph concede that clipping trades objective fidelity for stability. But conceding the gap does not remove its weight: the headline 'consistently improves' claim and the gradient-allocation demonstration are only as strong as the unverified assumption that the surrogate's update direction tracks Eq. (2) closely enough. The absence of error bars and the best-cell selection in Table 5 amplify this concern, but they are secondary to the theory-to-implementation bridge. The reader's CONDITIONAL verdict is appropriate: accept the analysis, demand the bridge be tested. We do not see grounds to reject or to upgrade to accept on the current evidence.","tokens_in":23872,"tokens_out":19607,"duration_ms":202653,"concrete_test":"Rerun the GSM8K and Countdown SoftmaxGRPO-Exact configurations with clipping inactive (set PPO ε so that min(ρ,1±ε) never binds) and reference KL coefficient β=0, using a single gradient step per sampled rollout batch so the update is on-policy to first order. Compare final Pass@1 and the Table 3 prompt-difficulty allocation histogram to the reported clipped runs. If the unclipped run does not reproduce the reallocation away from p≥0.9 prompts and the accuracy gains, then the exact finite-M objective is not the mechanism behind the headline results, and the conditional verdict should stand or tighten.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 proves exact statements for the unclipped on-policy objective in Eq. (2): Eq. (4) gives E[∇J^uc] = ω_{M,τ}(p)∇p under i.i.d. groups sampled from the current θ, and Eq. (5) gives the MaxRL limit. Section 4 then replaces this with the clipped, reference-KL-regularized surrogate in Eqs. (10)–(11), trained over PPO inner steps using rollouts from θ_old, and the paper itself states that this layer 'should be viewed as a trust-region approximation... not as part of the exact theorem.' Every accuracy result in Tables 2 and 4 and every gradient-allocation number in Table 3 comes from that surrogate. This is load-bearing because the central practical claim—that SoftmaxGRPO reallocates gradient away from near-solved prompts and consistently beats GRPO—is tested only through the approximation. If clipping or KL is materially active, the measured 36.4%→10.0% shift on GSM8K and the +2.3/+0.4/+0.9 point Exact gains can be artifacts of the trust-region machinery rather than manifestations of the bounded ω_{M,τ}(p) weighting. The theory itself is not at fault; the bridge from Eq. (2) to Eqs. (10)–(11) is the insecure load path.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SoftmaxGRPO, a group-based policy-gradient variant that replaces GRPO's z-score normalized advantages with temperature-scaled softmax weights w_i ∝ exp(R_i/τ) and centered advantages A_i = M w_i − 1. For binary rewards the authors derive an exact finite-M population objective h_M,τ(p) whose derivative is ω_M,τ(p) (Eq. (4)), show that its low-temperature limit is the MaxRL weighting with T = M−1 (Eq. (5)), and prove that for bounded scalar rewards the large-M update optimizes the log moment-generating function of reward (Eq. (7)). They also give a finite-M counterexample showing that a universal scalar potential does not exist for rewards with three or more levels (Eq. (8)). Empirically the paper reports gains over GRPO on verifiable tasks (GSM8K, Countdown, DeepMath), non-verifiable tasks (Poetry, MeetingBank), and a gradient-allocation analysis (Table 3), with all practical runs using a clipped, reference-KL-regularized PPO surrogate (Eqs. (10)–(11)).","tokens_in":24079,"tokens_out":5764,"duration_ms":62274,"significance":"The theoretical core is a genuine strength: the finite-M binary-reward analysis is clean, self-contained, and explicitly derives the Bernstein-polynomial form of ω_M,τ, the closed form of h_M,τ, and the exact MaxRL limit; the large-group scalar objective and the non-conservative counterexample are also clearly proved. If the empirical bridge were solid, this would be a valuable drop-in objective family with temperature-controlled prompt-difficulty weighting, avoiding GRPO's easy-prompt blow-up. The main reservations concern the empirical support: the headline configurations are selected from an ablation sweep, and all benchmark results are obtained through a clipped/KL-regularized surrogate that the paper itself excludes from its exact theorems. These issues are fixable and do not undermine the derivations, but they need to be addressed before the practical claims can be taken at face value.","major_comments":[{"comment":"The main GSM8K and Countdown configurations are selected as the best-performing cells of a factorial sweep: Appendix B.1 states that (M, τ) = (8, 0.1) are the best-performing cells of the sweep in Table 5. GRPO, by contrast, is reported under a single matched configuration with no equivalent sweep, even though one could sweep group size, PPO clip, and KL coefficient for GRPO. The headline differences in Table 2 (75.8 vs 73.5 on GSM8K and 58.1 vs 57.7 on Countdown) may therefore reflect favorable hyperparameter selection for SoftmaxGRPO rather than the advantage computation itself. Please report the full sweep for all methods or use a pre-selected (non-best-cell) configuration for the main comparison.","section":"§6.2, Appendix B.1, Table 5"},{"comment":"The exact results of Sections 3.2–3.4 apply to the unclipped, on-policy objective in Eq. (2), whereas every benchmark result in Tables 2–4 and every gradient-allocation number in Table 3 is produced by the clipped, reference-KL-regularized surrogate of Eqs. (10)–(11). The paper acknowledges that clipping is inactive only to first order at θ = θ_old and that away from that local regime the surrogate trades objective fidelity for stability, but it never quantifies how active clipping and the KL penalty are during training. Without diagnostics such as the fraction of clipped tokens or the average KL divergence over training, the claimed reallocation and the reported gains cannot be definitively attributed to the bounded weighting ω_M,τ(p) rather than to an interaction with the trust-region machinery. Please add such diagnostics, or run a small-scale unclipped experiment that directly targets Eq. (2).","section":"§4, Eqs. (10)–(11), Tables 2–4"},{"comment":"The gradient-allocation measurement is not reproducible as reported. The table gives fractions of token-level gradient budget by pass-rate bin, but the text does not specify how the pass rate p is estimated, from which checkpoint or training phase the gradients are taken, whether the gradients are computed under Eq. (10) or Eq. (2), or how the token-level budget is aggregated across prompts and steps. Since Table 3 is the central empirical evidence for the predicted reallocation away from near-solved prompts, this methodology needs to be specified precisely, ideally with a description of the measurement protocol in an appendix and with multiple checkpoints or seeds to show stability.","section":"§6.3, Table 3"},{"comment":"The Countdown ablation shows several (τ, M) configurations with collapsed (<30 tokens) or inflated (>100 tokens) response lengths, and the Pass@1 values are non-monotonic and noisy across nearby configurations (e.g., 29.8% at τ = 0.5, M = 4 vs 57.8% at τ = 0.3, M = 4). This instability suggests a strong interaction between the softmax temperature and the PPO trust region. The paper treats this as an optimization issue, but the absence of multiple seeds and variance estimates makes it difficult to assess whether the reported improvements are robust or whether they reflect particular runs of an unstable training procedure. Please report standard deviations or at least a second seed for the main configurations.","section":"§6.5, Table 5"}],"minor_comments":[{"comment":"The abstract contains a formatting defect where several words are run together: 'universalfinite-groupscalarobjectivecannotexistwithoutadditionalassumptions'. This should be corrected.","section":"Abstract"},{"comment":"The human-judge validation uses a single human rater, so the reported human–judge correlations of 0.74 (Poetry) and 0.89 (MeetingBank) measure agreement with one rater and do not establish inter-rater reliability. This should be stated more prominently, or additional raters should be recruited.","section":"Appendix B.4"},{"comment":"The claim that the GRPO regression from 1.5B to 3B on Countdown is 'consistent with verifier-RL scaling behavior reported independently by Cai and Provilkov [3]' is a post-hoc citation rather than a quantitative comparison; please either report the cited numbers or soften the claim.","section":"Appendix B.2"},{"comment":"The adaptive-τ effective-sample-size rule is a closed-form contribution, but its behavior at the boundaries k = 0 and k = M is only mentioned in passing. A sentence explaining that the rule is undefined exactly at these boundaries and that those cases require no adaptation would be helpful.","section":"Appendix A.3"},{"comment":"The sentence 'At θ = θ_old, clipping is inactive to first order' is stated twice, once in Section 4 and once in the 'Scope of claims' paragraph. One statement would suffice, with the second reserved for a pointer.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The mathematical contribution is solid and the paper is honest about the scope of its theorems. My recommendation is driven by the gap between the exact unclipped analysis and the empirical pipeline: the practical claims are load-bearing for the paper's impact, and the current evidence is weakened by best-cell configuration selection and by the absence of clipping/KL activity diagnostics. If the authors add those diagnostics and either perform a matched sweep for GRPO or use pre-selected configurations, I would be willing to reconsider. There is also no released code or checkpoints, which would be valuable for reproducing Tables 2–4."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the theory. The finite-M binary-reward result is the real contribution: Eq. (4) gives an exact population objective via a binomial expectation, Eq. (5) recovers MaxRL as the low-temperature limit, Eq. (7) gives a clean large-group log-MGF objective for bounded scalar rewards, and Eq. (8) is a sharp, honest counterexample showing why no universal finite-group scalar objective exists beyond binary rewards. The authors correctly credit RAML, softmax policy gradient, and MPO for the exponential-tilting core, so the novelty claim is appropriately scoped to the group-level geometry. The appendix derivations are self-contained and check out. That part of the paper is solid. The soft spots are in the experimental bridge. Section 3 proves statements about the unclipped on-policy objective in Eq. (2). Section 4 optimizes a clipped, reference-KL-regularized PPO surrogate, and the paper explicitly labels that layer a trust-region approximation, not part of the exact theorem. Every accuracy number in Tables 2 and 4, and every gradient-allocation number in Table 3, comes from that surrogate. So the measured reallocation away from near-solved prompts and the accuracy gains may reflect clipping or KL regularization rather than the bounded omega weights. The paper is honest about this, but the honesty does not close the gap between what is proved and what is claimed. The stress-test note about the insecure load path is right on target. Additional empirical weaknesses are real but in proportion: the main GSM8K and Countdown configurations are the best-performing cells of the Table 5 sweep, all runs are single-seed with no variance reported, no code or data is released, and the custom Poetry dataset uses gpt-5-instant prompt inversion, which adds a layer of uncertainty. These are standard reasons for conditional rather than immediate acceptance, not fatal flaws. None of this sinks the theoretical contribution. The central argument that softmax group advantages give bounded prompt weighting under binary rewards holds up, and the connections to MaxRL and the existing literature are useful. For people working on group-based RLHF objectives, this is worth a careful read and a serious referee. The referee should push for error bars, a configuration-selection protocol that does not pick the best cell, and released artifacts. I would send it out, expecting major revision on the empirical side.","headline":"Read this for the theory, which is clean and genuinely new; the empirical claims overreach because all headline numbers come from the clipped PPO surrogate the paper itself excludes from the theorems.","tokens_in":666,"tokens_out":742,"would_cite":true,"duration_ms":29687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temperature-scaled softmax group advantages keep prompt weighting bounded and outperform GRPO under identical rewards.","keywords":["SoftmaxGRPO","group-based reinforcement learning","GRPO","prompt difficulty weighting","binary rewards","log moment-generating function","MaxRL","reasoning post-training"],"falsifier":"Simulate $M$ binary rollouts at known pass probability $p$ for a policy whose score $\\nabla_\\theta \\log \\pi_\\theta$ is tractable, average the unclipped SoftmaxGRPO update over many groups, and compare with $\\omega_{M,\\tau}(p)\\nabla_\\theta p$; a mismatch would falsify Eq. (4).","tokens_in":23581,"feed_emoji":"🧠","tokens_out":6639,"duration_ms":58821,"temperature":0.7,"pith_summary":"SoftmaxGRPO claims that swapping GRPO's z-score group normalization for a temperature-scaled softmax advantage fixes a known pathology: under binary rewards, GRPO weights easy prompts so strongly that its gradient signal diverges as pass rate approaches 1, while SoftmaxGRPO's weights stay bounded. The paper proves an exact finite-group population objective for binary rewards, identifies MaxRL as the zero-temperature limit, and shows that for bounded scalar rewards the large-group update exactly optimizes the log moment-generating function of reward. It also proves that no universal finite-group scalar objective exists once rewards take three or more levels, so binary rewards are special. Empirically, SoftmaxGRPO reallocates measured gradient budget toward moderate-difficulty prompts and improves over GRPO under identical rewards on verifiable and non-verifiable reasoning tasks.","feed_headline":"Softmax advantages fix GRPO's easy-prompt gradient blow-up","feed_subtitle":"A temperature-scaled softmax weighting stays bounded on solved prompts and beats GRPO on math and poetry.","key_machinery":"The central object is the temperature-scaled softmax group advantage, $w_i\\propto e^{R_i/\\tau}$ with centered advantage $A_i=M w_i-1$, applied to a group of $M$ rollouts per prompt. Under binary rewards the machinery reduces to a one-dimensional prompt-weighting function $\\omega_{M,\\tau}(p)$: conditioning on the number of other successes $S\\sim\\mathrm{Binomial}(M-1,p)$ gives per-sample weight gaps $\\Delta_S^{(\\tau)}$, whose binomial expectation is a Bernstein-polynomial weight that integrates to the exact objective $h_{M,\\tau}(p)$. This is what carries the argument: the same softmax construction yields the finite-$M$ binary theorem, the MaxRL low-temperature limit, and, through concentration of the softmax denominator, the large-group log-moment-generating objective for bounded scalar rewards.","core_discovery":"For binary rewards in the unclipped on-policy regime, SoftmaxGRPO's expected group update on a prompt with pass probability $p$ equals $\\omega_{M,\\tau}(p)\\nabla p$, where $\\omega_{M,\\tau}(p)=\\mathbb{E}_{S\\sim\\mathrm{Binomial}(M-1,p)}[\\Delta_S^{(\\tau)}]$ is a Bernstein polynomial with positive coefficients and is finite at $p=1$. The paper proves this weight is the derivative of an exact scalar objective $h_{M,\\tau}(p)$, recovers MaxRL's truncated weighting $\\omega_{M,0}(p)=(1-(1-p)^{M-1})/p$ as $\\tau\\to 0$, and approaches maximum-likelihood $1/p$ weighting in the joint low-temperature, large-group limit. For bounded scalar rewards, the large-group update is exactly the gradient of $\\log Z_\\tau(\\theta;x)=\\log\\mathbb{E}_{\\pi_\\theta}[e^{R/\\tau}]$. With three or more reward levels, however, the finite-group update is generically non-conservative, so no analogous scalar potential exists without extra assumptions; this is the paper's sharp boundary for clean objective-based analysis.","pith_inferences":["If the bounded-weighting mechanism is the source of the gains, SoftmaxGRPO should help most on tasks with many near-solved prompts, and should also reduce update variance there; that prediction is testable on any dataset by binning pass rates.","The appendix's effective-sample-size rule suggests an adaptive per-prompt or per-batch temperature schedule, which the paper leaves for future work; such a schedule could remove the need to hand-tune $\\tau$ to reward scale.","The non-conservativeness result implies that with fine-grained scalar rewards, comparing training methods by their 'objective value' may be meaningless, since no scalar potential exists; a testable extension would detect cyclic gradient accumulation around closed loops in reward space."],"forward_implications":["GRPO's easy-prompt gradient blow-up is avoidable without changing the reward signal: a single temperature parameter keeps prompt weights finite at every pass rate.","Adjusting $\\tau$ interpolates between REINFORCE-like uniform weighting and MaxRL-like hard-prompt emphasis, with maximum-likelihood weighting approached only in the joint low-temperature, large-group limit.","For bounded scalar rewards and large groups, SoftmaxGRPO training is exactly optimizing $\\log\\mathbb{E}[e^{R/\\tau}]$, so weak overlap rewards acquire a well-defined exponential-utility objective.","The finite-group scalar objective exists only when the reward geometry is one-dimensional (binary); with three or more reward levels, no universal potential exists and per-distribution analysis is required.","Measured gradient allocation shifts from near-solved prompts ($p\\ge 0.9$) toward moderate difficulty, which is the concrete mechanism behind the reported gains over GRPO."],"supporting_citations":[{"why":"Supplies the conditional-linear advantage form and the divergent $[p(1-p)]^{-1/2}$ prompt weighting of GRPO that SoftmaxGRPO replaces.","marker":"[6]"},{"why":"Defines MaxRL and its truncated $1/p$ weighting, which the paper proves is the zero-temperature limit of SoftmaxGRPO.","marker":"[36]"},{"why":"Introduces the GRPO z-score group-advantage objective that serves as the replacement target and experimental baseline.","marker":"[33]"},{"why":"Reward-Augmented Maximum Likelihood, the origin of exponentiated-reward weighting whose finite-$M$ tilt interpretation SoftmaxGRPO inherits.","marker":"[25]"},{"why":"MPO, which supplies the variational KL-trust-region view of the softmax weights.","marker":"[1]"},{"why":"PPO, whose clipped surrogate and reference-KL penalty form the practical optimization layer used in all experiments.","marker":"[32]"}],"fun_headline_variants":["SoftmaxGRPO tames easy-prompt gradient blow-up","Softmax advantage keeps GRPO gains bounded","Fix GRPO's easy-prompt divergence with softmax weighting","SoftmaxGRPO beats GRPO without changing rewards","SoftmaxGRPO rebalances gradient budget across prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark results assume that the PPO-clipped, KL-regularized update used in practice behaves like the unclipped on-policy objective whose exact properties are proven.","fun_headline_variants_meta":{"raw":{"variants":["SoftmaxGRPO tames easy-prompt gradient blow-up","Softmax advantage keeps GRPO gains bounded","Fix GRPO's easy-prompt divergence with softmax weighting","SoftmaxGRPO beats GRPO without changing rewards","SoftmaxGRPO rebalances gradient budget across prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000957,"raw_usage":{"total_tokens":4097,"prompt_tokens":981,"completion_tokens":3116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":3038}},"tokens_in":597,"tokens_out":3116,"duration_ms":18760,"temperature":1.0,"reasoning_tokens":3038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:22:04.709772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate $M$ binary rollouts at known pass probability $p$ for a policy whose score $\\nabla_\\theta \\log \\pi_\\theta$ is tractable, average the unclipped SoftmaxGRPO update over many groups, and compare with $\\omega_{M,\\tau}(p)\\nabla_\\theta p$; a mismatch would falsify Eq. (4).","supporting_citations":[{"cited_title":"Reward augmented maximum likelihood for neural structured prediction","cited_arxiv_id":null,"evidence_quote":"Reward-Augmented Maximum Likelihood, the origin of exponentiated-reward weighting whose finite-$M$ tilt interpretation SoftmaxGRPO inherits."},{"cited_title":"Maximum a posteriori policy optimisation","cited_arxiv_id":null,"evidence_quote":"MPO, which supplies the variational KL-trust-region view of the softmax weights."}],"review_version":1}