{"id":"5177dd00-fd8b-43a2-b392-3fc5967135fc","arxiv_id":"2607.18979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Rewarding each parallel reasoning path by Monte-Carlo-Shapley marginal contribution, scored by a generative reward model, lifts Pass@16 on AIME24/AIME25/AMC23 by 4-90% relative over Parallel-R1 with a fifth of the training steps.","lead":"This paper adds Shapley-value credit assignment to reinforcement learning for LLM parallel reasoning: each reasoning path is rewarded by its average marginal contribution to a generative reward model's score of the path set. Single-run experiments report large Pass@16 gains (AIME24 33.2→63.3) in 40 instead of 200 RL steps, while mean accuracy improves only slightly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The process reward rests on GRM subset utilities that are never validated against the actual summary; because implemented Shapley rewards sum to GRM(P) rather than the outcome reward r_out (Appendix A assumes Σφ_j=r_out), the claimed less-biased, free-rider-free signals are not established.","rationale":"I read the paper in good faith: the proposed framework is coherent, the ablations (LOO, independent-path evaluation, GRM strength, K sweep) are the right experiments, and the limitations section honestly discloses the Mean@16/Pass@16 trade-off and the lack of evaluation on larger backbones. The empirical direction is consistent across four benchmarks and several GRM choices, which is real evidence. However, the single most load-bearing assumption is the validity of the GRM-based utility function, because every path-level Shapley reward is built on it. The paper never checks whether GRM(S) measures the actual contribution of a path subset to the final <Summary>; the GRM prompt only scores path text against ground truth. Worse, Appendix A's theoretical motivation explicitly assumes Σ_j φ_j = r_out, while the implementation's Shapley values sum to GRM(P), with r_out applied separately at the final token. That mismatch means the proof does not cover the implemented reward, and the claim that the method 'fishes out free riders' goes beyond what is demonstrated. The GRM sensitivity rows weaken this concern somewhat—if the GRM were pure noise, weaker evaluators would likely not produce consistent gains—but they do not establish the required correlation with actual summary-level marginal contributions. The proposed diagnostic would settle this directly. I therefore keep the reader's CONDITIONAL verdict unchanged rather than upgrading or rejecting: the central empirical result may hold, but the core interpretative claim about path-level attribution remains conditional on a validation that has not been reported.","tokens_in":21118,"tokens_out":7417,"duration_ms":77059,"concrete_test":"Take the trained 40-step Parallel Shapley policy and 200 held-out AIME/AMC problems. For each problem, generate the K paths and the actual <Summary>; record r_out, GRM(P), and the MC-Shapley vector φ. Then for each path j, mask path j and regenerate the Summary (same temperature/decode settings), measuring Δr_out,j = r_out(Summary without j) − r_out(full Summary). Report (i) Spearman correlation between φ_j and Δr_out,j across all paths/problems, and (ii) mean absolute deviation between Σ_j φ_j and r_out. A near-zero/negative correlation or large deviation would show the process reward is not tracking actual marginal contribution to the final answer, undercutting the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is Monte-Carlo Shapley over U(S)=GRM(S) (§3.2), and every path-level reward is a difference of GRM scores. Two things make this load-bearing. First, the GRM never conditions on the policy's <Summary>; its prompt (Appendix B) asks it to score a bare subset of paths against ground truth. A path's actual marginal contribution is defined by how the summarizer uses it, so U(S) is a proxy, not the true coalition utility. If the GRM over-weights local correctness or information purity relative to what the summary actually uses, the Shapley values reward the wrong paths. Second, the theoretical justification in Appendix A assumes Σ_j φ_j = r_out (Eqs. 18-20), but the implemented reward tensor (§3.3) injects r_out only at the final token and uses normalized Shapley values summing to GRM(P); nothing in the paper shows GRM(P) tracks r_out. The GRM-sensitivity rows show robustness to evaluator strength, but that only tests the evaluator's scale, not whether the attributed path contributions correspond to the summary-generation process. If the GRM is miscalibrated, the 42.3% Pass@16 gain could still be real, but the paper's causal story—removing free-riders—would be unsupported, and the method would be an expensive function of an unvalidated proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Parallel Shapley, a reinforcement learning framework for multi-path ('parallel thinking') LLM reasoning. Each reasoning path is treated as a player in a cooperative game; a generative reward model (GRM) scores subsets of paths, and Monte Carlo Shapley values are used as token-level path rewards in GRPO, combined with a final outcome reward. Experiments on AIME24/25, AMC23, and MATH with a Qwen3-4B base report an average 42.3% relative improvement in Pass@16 over the Parallel-R1-Unseen baseline while using 40 rather than 200 RL steps, together with ablations (leave-one-out, independent evaluation, GRM strength, path count) and an appendix on multi-hop QA.","tokens_in":21393,"tokens_out":9150,"duration_ms":80677,"significance":"The idea of using Shapley-based credit assignment for parallel reasoning paths is timely and relevant to RL training for LLMs. The implementation is concrete (built on VERL, with an external GRM), and the GRM-strength ablation and multi-hop QA extension are useful sanity checks. If the attribution mechanism were properly validated, the work could influence reward design for multi-path generation. However, the central evidence is single-run benchmark numbers, and the path-level reward is computed from an unvalidated GRM proxy rather than from the actual summary-generation process, so the significance is conditional on additional validation.","major_comments":[{"comment":"The process reward is the Monte Carlo Shapley value of U(S)=GRM(S), where the GRM is prompted (Appendix B) to score a bare subset of paths against the ground-truth answer. The GRM never sees the policy's <Summary>, yet a path's true marginal contribution in parallel reasoning is defined by how the summarizer uses it. The paper provides no evidence that GRM subset utilities track the summary's use of paths, nor that GRM(P) is consistent with the outcome reward r_out. Appendix A's motivating theorem assumes Σ_j φ_j = r_out (Eqs. 18–20), but the implemented reward tensor (§3.3) sums the Shapley rewards to GRM(P) and injects r_out separately at the final token. The claimed 'less biased, free-rider-free' signal is therefore unsupported as stated. Please add a direct validation, e.g., measure the correlation between GRM(P) and r_out over rollouts, or compare Shapley path rewards against human/","section":"§3.2, Appendix B"},{"comment":"All reported results are single-run, with no confidence intervals or repeated seeds. On the 30-problem AIME sets, the AIME25 Pass@16 difference (50.0 vs. 37.8, +12.2 points) is about 1.0 binomial standard error, and the AIME24 difference (63.3 vs. 33.2, +30.1 points) is about 2.4 standard errors. The 'consistently outperforms' claim is not statistically established. Moreover, K=4 is selected from the sensitivity sweep in Table 2 using the same test benchmarks, so the reported K=4 numbers are the best case over the path count, further inflating apparent gains. Please report multiple seeds, bootstrap confidence intervals, and either a validation split for hyperparameter selection or an appropriate multiple-comparison correction.","section":"Table 1, Table 2"},{"comment":"The theoretical justification for the harm of uniform rewards contains derivational gaps. Eq. (16) converts a triangle-inequality bound into an approximate equality; Eq. (2) does not follow from Eq. (13) without an additional assumption about the sum of cosine-similarity terms; and Eq. (18)'s decomposition U(P)=Σ_j φ_j(P−j) is not the Shapley decomposition used in the implementation (which averages over all coalitions, not just P−j). Since this theorem is the paper's stated motivation for needing path-level rewards, it should be corrected or explicitly labeled as heuristic motivation rather than a formal result.","section":"Appendix A, Eqs. (2), (13), (16), (18)–(20)"}],"minor_comments":[{"comment":"The text says 'Independent path evaluation achieves the lowest performance (Avg. 46.5)' and 'LOO performs slightly worse (Avg. 46.2)', but Table 1 shows LOO at 46.2 and Independent at 46.5. The text and table contradict each other; LOO is lower.","section":"§4.3, Table 1"},{"comment":"The normalization of the two reward signals is not specified. It is unclear whether the Shapley rewards are normalized per rollout, per group, or globally, and how λ_o=0.5, λ_p=0.5 interact with that normalization. Please define 'normalized Shapley reward' explicitly.","section":"§3.3"},{"comment":"The evaluation prompt says it assesses 'the solution-path summary for {query}', but it then refers to '{path}' in the singular and does not specify how a subset of multiple paths is serialized into the prompt. This makes the GRM's exact input for a coalition S ambiguous.","section":"Appendix B"},{"comment":"The abstract and Section 4.2 claim 'consistently outperforms' strong baselines, but Table 1 shows AIME25 Mean@16 for Parallel Shapley (17.2) is below Parallel-R1-Unseen (17.7). Please qualify the claim to Pass@16 or overall average.","section":"§4.2"},{"comment":"Monte Carlo sampling uses M=5 for K=3,4,5. With such a small number of permutations, the Shapley estimates are likely to be noisy, yet no variance diagnostics or sensitivity to M are reported. Please report the variance of φ across permutations or increase M.","section":"Appendix D.1"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism is plausible, but the paper's main causal claim—that Shapley path rewards remove free-riders—rests on an unvalidated GRM proxy and single-run statistics. The internal inconsistency between Appendix A (Σφ_j = r_out) and §3.3 (Σφ_j = GRM(P)) needs to be resolved. I would support publication after the authors validate the GRM's subset utilities against the summary/outcome, add statistical rigor, and correct the theoretical appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading: it takes a real weakness in Parallel-R1's uniform outcome rewards and replaces them with Monte-Carlo Shapley values over a generative-reward-model utility. The combination is the genuinely new part; Shapley values themselves are standard. The empirical work is honest—they report the Pass@16/Mean@16 trade-off, run ablations for LOO, independent-path evaluation, GRM strength, and path count, and they use a fixed base model and training recipe.\n\nWhat holds it back is statistical fragility. All comparisons are single runs with no error bars. The headline 42.3% Pass@16 improvement is inflated by AIME24 (+90.7%), where n=30 makes the binomial noise about ±9 points. AIME25's +32.3% is about 1.3 SE. The direction of the effect is consistent across four benchmarks and three GRM strengths, and that consistency is reassuring, but the strength of the claim exceeds the evidence.\n\nThe bigger conceptual soft spot is the GRM. The Shapley values decompose U(S)=GRM(S), but the GRM never conditions on the <Summary>; it scores bare path subsets against ground truth. The paper never shows that GRM(P) tracks the outcome reward r_out, nor that the Shapley values correlate with actual path contributions. Appendix A proves free-riding under Σφ_j=r_out with factorized log-probabilities, but the implementation uses normalized Shapley values summing to GRM(P) and autoregressive paths. So the theoretical justification does not cover the implemented reward. That mismatch doesn't kill the empirical result—training toward a differentiated proxy could still be better than uniform rewards—but it means the \"free-rider\" story is not established.\n\nAlso minor: λ_o and λ_p are fixed untested at 0.5/0.5, M=5 permutation samples is small, and K=4 is selected from a subjectively chosen sweep.\n\nBottom line: this deserves a serious referee. The idea is timely, the experiments are the right ones, and the limitations are acknowledged. But acceptance should require repeated-seed CIs, a GRM-vs-outcome correlation check, and reconciliation of the appendix with the implemented reward tensor. I would bring it to reading group as a case study in how credit-assignment ideas enter LLM RL.","headline":"A sensible Shapley-based credit-assignment recipe for parallel-reasoning RL, with suggestive but statistically fragile results and a load-bearing GRM that is never validated against the summary.","tokens_in":22019,"tokens_out":2195,"would_cite":true,"duration_ms":20684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Parallel Shapley replaces uniform outcome rewards with per-path Shapley values, sharply improving multi-path math reasoning and cutting training steps.","keywords":["Shapley value","parallel reasoning","reward attribution","reinforcement learning","generative reward model","credit assignment","GRPO","mathematical reasoning"],"falsifier":"Using the authors' stored rollouts, compare GRM subset utilities against an outcome-derived ground truth: for each subset of paths, force the summarizer to write the summary from only those paths and check whether the mean GRM score orders the resulting correctness (Pass@1) monotonically. If a higher-GRM subset is not more likely to yield the correct answer, the process reward is not tracking marginal contribution to correctness, and the reported Pass@16 gains would be expected to survive even with random permutation of path rewards.","tokens_in":20861,"feed_emoji":"🧮","tokens_out":5917,"duration_ms":53432,"temperature":0.7,"pith_summary":"The paper tries to show that the free-rider problem in parallel reasoning—where redundant or misleading paths collect the same reward as truly helpful ones—is measurable and fixable at training time. Its proposed fix, Parallel Shapley, assigns each reasoning path a process reward equal to its average marginal contribution to the utility of all path subsets, where that utility is scored by a generative reward model. The authors argue this dense, attribution-based signal lets a model learn from 40 reinforcement-learning steps what a uniform-reward baseline cannot learn in 200, and that it raises the upper bound of multi-path correctness (Pass@16) by 42.3% on average across competition-math benchmarks. The central consequence, if true, is that outcome-level reward is not a necessary bottleneck for learning to reason in parallel.","feed_headline":"Path-level Shapley rewards lift parallel math reasoning by 42%","feed_subtitle":"Dense path-level credit lets a 4B model match a 200-step baseline in 40 steps and produce complementary paths.","key_machinery":"The load-bearing object is the Monte Carlo Shapley value of a reasoning path: the path's expected marginal contribution to the utility of every coalition of other paths, U(S)=GRM(S), averaged over sampled path permutations. The generative reward model supplies the utility of each path subset from a five-dimension rubric (method soundness, formula completeness, computational accuracy, result correctness, and information purity); the Shapley average turns those subset scores into a per-path credit signal. Its role is to disambiguate learning: paths that add no information or actively mislead receive low or negative rewards even when the final summary is correct, which is exactly what uniform o","core_discovery":"The central claim is that replacing the single outcome-level reward shared by all reasoning paths with a Shapley-based process reward eliminates free-riding and yields better, cheaper training. Each path is a player in a cooperative game; the utility of any subset of paths is scored in [0,5] by a generative reward model prompted against the ground-truth answer, and each path's Shapley value—its average marginal utility contribution over all subset sizes and permutations—becomes its token-level reward, injected at the path's closing tag and normalized before GRPO advantage estimation. The paper reports that on four mathematics benchmarks, a 4-billion-parameter model trained this way for 40 st","pith_inferences":["The same Shapley-over-a-proxy-utility recipe could be applied to any generation problem where a final output is assembled from multiple components (code modules, retrieved evidence, tool-call chains); the paper's mechanism is agnostic to the component content as long as a subset scorer exists.","A cheap discriminating test: swap the GRM for a non-generative, outcome-derived utility (e.g., majority-vote correctness), and keep everything else fixed; if gains persist, the credit comes from the Shapley weighting rather than from the richer rubric.","The Mean@16 vs Pass@16 gap suggests the policy may be over-specializing individual paths; a distillation or summary-reward term that rewards synthesizing across paths could convert upper-bound gains into single-generation accuracy, a next step the paper itself flags.","Because the GRM never reads the summary, the paper's measure of 'contribution' is indirect; feeding the GRM the full response (paths plus summary) would test whether the current utility is actually aligned with the final output quality that the outcome reward measures."],"forward_implications":["Redundant and misleading paths no longer receive positive gradient when the summary happens to be correct; the policy is pushed toward paths with positive marginal contribution.","Training efficiency improves: 40 GRPO steps suffice to match or beat a 200-step parallel-thinking baseline, with stable convergence around step 30.","The induced policy produces more complementary, specialized paths; masking a fraction of paths degrades Parallel Shapley-trained rollouts more sharply than uniform-reward rollouts.","Pass@16, a proxy for the upper bound of multi-path capability, rises by an average of 42.3% on three competition-math benchmarks, while Mean@16 improves modestly.","The benefit is not tied to one evaluator: the method beats the uniform-reward baseline even when the generative reward model is replaced by weaker open-source models, and preliminary multi-hop QA results show the pattern generalizes beyond mathematics."],"fun_headline_variants":["Shapley rewards fish out free riders in parallel reasoning","Path-level credit lets 4B model match 200 steps in 40","Shapley attribution stabilizes multi-path RL training","Fair per-path rewards improve parallel math reasoning","Generative reward model scores paths via Shapley values"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's correctness hinges entirely on the generative reward model's subset scores being a faithful measure of how much a set of reasoning paths helps produce the right final answer; if those scores are miscalibrated, the Shapley rewards are precise attributions of the wrong quantity.","fun_headline_variants_meta":{"raw":{"variants":["Shapley rewards fish out free riders in parallel reasoning","Path-level credit lets 4B model match 200 steps in 40","Shapley attribution stabilizes multi-path RL training","Fair per-path rewards improve parallel math reasoning","Generative reward model scores paths via Shapley values"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001013,"raw_usage":{"total_tokens":4084,"prompt_tokens":680,"completion_tokens":3404,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":3323}},"tokens_in":424,"tokens_out":3404,"duration_ms":22635,"temperature":1.0,"reasoning_tokens":3323,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:48:29.984514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using the authors' stored rollouts, compare GRM subset utilities against an outcome-derived ground truth: for each subset of paths, force the summarizer to write the summary from only those paths and check whether the mean GRM score orders the resulting correctness (Pass@1) monotonically. If a higher-GRM subset is not more likely to yield the correct answer, the process reward is not tracking marginal contribution to correctness, and the reported Pass@16 gains would be expected to survive even with random permutation of path rewards.","supporting_citations":[],"review_version":1}