{"id":"5a468c8e-9181-48db-83fb-59e9644baff4","arxiv_id":"2411.11681","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PSPO-WRS adds a Weibull-based reward that depends on reasoning chain length to process-supervised RLHF, and reports accuracy gains over baseline LLMs on six NLI-style reasoning datasets.","lead":"This paper proposes a process-supervised reinforcement learning method that scores each reasoning step and then scales the final reward by a nonlinear function of the number of steps. It reports gains over several 7B LLMs on six math-reasoning datasets, but its key ablation table contradicts the prose.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's PSPO-WRS column is offset relative to Table 2; as printed, the nonlinear-reward ablation shows w/o nonlinear winning on three datasets, so the central mechanistic claim is not currently supported.","rationale":"The reader's weakest assumption was the hand-fixed Weibull parameters, while my load-bearing concern is the internal inconsistency of the ablation table. These are related but distinct: both concern whether the nonlinear-reward mechanism is actually supported. I agree with the reader's conditional verdict, and add that the condition must include reconciling Table 4 with Table 2. The strongest claim about consistent outperformance rests on Table 2, which is internally coherent, and Table 3 provides some independent generalization evidence, which is why this is not a rejection on fraud or fabrication grounds. However, the mechanistic claim that nonlinear reward shaping is responsible for the gains is only as strong as the ablation, and that ablation is currently self-contradictory. A simple re-run with aligned labels and variance estimates would settle whether the concern is a formatting error or a substantive weakness. Until then, the paper should not be accepted as-is.","tokens_in":16346,"tokens_out":6421,"duration_ms":59215,"concrete_test":"Re-run the Table 4 ablation using the same 20% held-out split, seeds, and PPO hyperparameters as Table 2, and publish the per-dataset values in a single aligned table with the same row order. If the corrected PSPO-WRS column matches Table 2 and exceeds w/o nonlinear on all six rows, the anomaly is typographical and the central claim survives; if QQA (or any row) still does not improve, the claim that nonlinear length-aware rewards are beneficial is unsupported and the paper should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that process-supervision rewards depend nonlinearly on reasoning length is isolated in the ablation study, Table 4. As printed, the PSPO-WRS column is inconsistent with Table 2: the entries 64.91, 67.60, 71.57, 52.29, and 54.70 are Table 2's NewsNLI, RedditNLI, RTE-Quant, StressTest, and QQA values, not the values for the rows in which they appear, and StressTest shows 53.57 instead of 52.29. Taking the table at face value, the w/o-nonlinear baseline beats PSPO-WRS on AwpNLI (69.12 vs 64.91) and RTE-Quant (59.66 vs 52.29), and narrowly on QQA (54.94 vs 54.70), directly contradicting the text that nonlinear reward improves all datasets. Since this table is the only evidence that isolates the nonlinear reward contribution, the paper's key mechanistic conclusion is not reproducible from the manuscript as written. The out-of-distribution results in Table 3 support the overall method but do not isolate the nonlinearity, so they do not repair this defect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes PSPO*, a paradigm for process-supervised policy optimization, and PSPO-WRS, a concrete instance that combines geometric-mean step accuracy with an adjusted Weibull reward-shaping function. The central claims are that process-supervision effectiveness depends on both accuracy and reasoning-chain length, that the reward score is nonlinear, and that PSPO-WRS consistently outperforms current models on six mathematical reasoning datasets. The paper includes an out-of-distribution generalization check and a code link.","tokens_in":16589,"tokens_out":3438,"duration_ms":30556,"significance":"If the empirical claims hold, the paper offers a useful systematization of process supervision and draws attention to length bias in multiplicative step rewards. The out-of-distribution experiments in Table 3 are a positive check against overfitting, and the promise of released code supports reproducibility. However, the sole ablation isolating the nonlinear-reward contribution is internally inconsistent as printed, and the fixed Weibull parameters are neither tuned nor subjected to sensitivity analysis, so the central mechanistic claim is currently not established.","major_comments":[{"comment":"As printed, Table 4's PSPO-WRS column is misaligned: the entries 64.91, 67.60, 71.57, 52.29, and 54.70 are the Table 2 values for NewsNLI, RedditNLI, RTE-Quant, StressTest, and QQA, respectively, not the values for the rows in which they appear. Reading the table at face value, the w/o-nonlinear baseline outperforms PSPO-WRS on AwpNLI (69.12 vs 64.91), RTE-Quant (59.66 vs 52.29), and QQA (54.94 vs 54.70), directly contradicting the text that nonlinear rewards improve performance across all evaluated datasets. StressTest also shows 53.57 in Table 4 versus 52.29 in Table 2. Since Table 4 is the only experiment that isolates the nonlinear-reward contribution, the main mechanistic claim is not reproducible from the manuscript as written; the authors must correct the table and re-state the conclusions accordingly.","section":"Table 4 / Ablation Analysis"},{"comment":"The Weibull reward-shaping parameters C=10.735, k=1.5, and lambda=8.0 are fixed without any tuning procedure, grid search, or sensitivity analysis. Because Eq. (9) directly determines how the reward depends on step count, the observed gains could be artifacts of these specific values. Please report an ablation over these parameters or justify the choice from properties of the datasets.","section":"Experimental Setups / Eq. (9)"},{"comment":"The nonlinearity is introduced by construction: Eq. (10) length-normalizes the step-product reward and Eq. (9) applies a Weibull-shaped length prior. The comparison against a baseline without length shaping therefore tests the presence of a length-dependent reward, not the specific Weibull form or the broader claim that the underlying reward is nonlinear. I recommend comparing against alternative length reweighting schemes (e.g., linear normalization, logarithmic penalty, or a quadratic length model) and reporting whether the Weibull form gives materially better results.","section":"Eqs. (9)-(10) / Ablation Analysis"},{"comment":"All experimental comparisons appear to be single runs with no error bars, confidence intervals, or multiple training seeds. PPO training is known to be high-variance, and some reported differences in Table 4 are small (e.g., QQA 54.94 vs 54.70). Please report means and variances over at least three seeds, or explicitly justify why seed variation is negligible.","section":"Experimental Results / Metrics and Parameters"}],"minor_comments":[{"comment":"The dataset name is written inconsistently as \"RTE-Quant\" and \"RTE Quant\" across the text and tables; please standardize.","section":"Throughout"},{"comment":"The phrase \"we concrete the PSPO* paradigm\" should be \"we instantiate\" or \"we concretize the PSPO* paradigm\".","section":"Introduction"},{"comment":"The notation is overloaded: Eq. (2) writes p(zi) = sigma(zi), but sigma is later defined as softmax in Eq. (3), which is not a component-wise sigmoid; please define the activation consistently.","section":"Eqs. (2)-(3)"},{"comment":"The caption contains the typo \"Prameter settings\" and should read \"Parameter settings\".","section":"Figure 4"},{"comment":"The row \"Num. of steps\" is not defined in the caption; please state which policy's average step count is reported and over which evaluation set.","section":"Table 4"},{"comment":"The claim \"We are the first to assert that the reward score in the reasoning alignment is nonlinear\" is stronger than what the related-work discussion supports; product-of-probabilities rewards are already nonlinear in step count, so prior work implicitly involves nonlinearity even if not framed that way.","section":"Introduction / Contribution 2"}],"recommendation":"major_revision","confidential_remarks":"The Table 4 misalignment is the main blocker to acceptance; a corrected table and re-analysis may resolve it. I did not verify the GitHub repository or the experimental code. The strong novelty claim about being \"first\" to assert nonlinearity is likely to draw reviewer scrutiny and should be softened or better contextualized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the PSPO-WRS idea—normalizing step-probability products and shaping with a Weibull curve over step count—is a reasonable add-on to process-supervised RLHF. It targets a real failure mode of the standard product accumulation, which silently penalizes longer chains. The paper is worth engaging with. But the key evidence is currently not reproducible as written.\n\nWhat's actually new: the geometric-mean accumulation (1/t power) plus the Weibull reward shaper is a specific combination I haven't seen. The Weibull prior—penalizing both too-short and too-long chains—is a sensible way to encode efficiency-accuracy tradeoffs. The out-of-distribution results in Table 3 show the full method beats the product-only baseline on six unseen datasets, which is a useful sanity check.\n\nThe soft spots are real. Table 4, the only place that isolates the nonlinear reward, is internally inconsistent. The PSPO-WRS column appears to contain entries from Table 2 offset by one row, and taken at face value the 'w/o nonlinear' baseline wins on AwpNLI, RTE-Quant, and QQA. The prose says nonlinearity helps everywhere. That contradiction is load-bearing: the 'first to assert nonlinear' claim rests entirely on this table. The OOD table supports the overall method but does not isolate the nonlinear component, so it can't repair the defect. This needs a corrected table and a rerun or a clear explanation before the central claim can be evaluated.\n\nSecond, the Weibull parameters C, k, lambda are hand-set with no sensitivity analysis, and there are no error bars or multiple seeds. On six datasets, that matters. Third, the novelty claim 'first to assert nonlinear' is not credible: product accumulation is already nonlinear, and the authors cite Jin et al. on step length. The specific shaper is new; the general observation isn't.\n\nWho this is for: people working on process rewards for reasoning will want to see the method, and the geometric-mean fix is worth testing in their own pipelines. But the paper needs major revision before it is citable. I'd send it to review, with a clear request: fix Table 4, add sensitivity runs, and tone down the novelty claim. If Table 4 is a formatting error, the paper has a plausible story.","headline":"A sensible reward-shaping idea undermined by an ablation table that, as printed, contradicts the paper's main claim.","tokens_in":17129,"tokens_out":5290,"would_cite":false,"duration_ms":45115,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that process-supervised rewards should depend on both the accuracy and the length of a reasoning chain, and that the relationship is nonlinear, reporting that a method built on this principle outperforms mainstream…","keywords":["process supervision","reward shaping","chain-of-thought reasoning","reasoning alignment","policy optimization","Weibull distribution","mathematical reasoning","large language models"],"falsifier":"Take the same PSPO-WRS pipeline and run it on problems whose correct solutions require very long chains, for example 15-25 steps, while also sweeping $\\lambda$ over a grid from 4 to 16 with all else fixed: the nonlinearity claim predicts the method should keep beating the linear baseline across a range of $\\lambda$, whereas the length-prior artifact predicts performance peaks only near $\\lambda=8$ and collapses on long-chain problems.","tokens_in":16106,"feed_emoji":"🧮","tokens_out":7853,"duration_ms":69564,"temperature":0.7,"pith_summary":"The paper sets out to show that process supervision of chain-of-thought reasoning should reward not just the correctness of each step but also the length of the reasoning chain, and that the two combine nonlinearly into an overall reward score. It proposes a full workflow, PSPO*, that trains a step-level reward model and then optimizes a policy against a reward in which per-step probabilities are combined by geometric mean and then shaped by an adjusted Weibull distribution that peaks at a preferred number of steps. The paper asserts that it is the first to claim nonlinearity of the reward score in reasoning alignment, and it reports that the resulting method, PSPO-WRS, outperforms current baselines on six mathematical reasoning datasets while also improving out-of-distribution generalization. A sympathetic reading takes the central claim to be that length-aware, nonlinear process rewards make reasoning policies produce more accurate and more complete chains.","feed_headline":"Step-count-aware rewards beat six math reasoning baselines","feed_subtitle":"A geometric-mean step score plus Weibull shaping lifts a 7B model above much larger LLMs on most tested datasets.","key_machinery":"The load-bearing object is the combined reward: a geometric-mean accumulation function $F = (\\prod_{j=1}^t R^s_j)^{1/t}$, where $R^s_j$ is the reward model's probability that step $j$ is positive, multiplied by an adjusted Weibull reward-shaping function $R_s = C \\frac{k}{\\lambda}(\\frac{t}{\\lambda})^{k-1} e^{-(t/\\lambda)^k}$. The geometric mean removes the automatic penalty that plain multiplication imposes on long chains, and the Weibull factor injects the prior that chains of roughly $\\lambda = 8$ steps deserve the largest shaping reward, with $k=1.5$ controlling how sharply the bonus falls off. The policy objective is $J(\\pi) = E_\\pi[R_s F - \\beta D_{KL}(\\pi \\| \\pi_{ref})]$. This machinery carries the paper's evidence: the ablation that removes $R_s$ degenerates to the standard product-accumulation method of Lightman et al. and loses performance, while the full method shifts the policy toward longer, higher-scoring chains.","core_discovery":"Process supervision, as usually implemented, scores a reasoning chain by multiplying the per-step correctness probabilities, which silently punishes longer chains even when every step is right. The paper's central discovery claim is that this multiplicative scheme is wrong in two ways: the overall reward should depend on the number of steps as well as their accuracy, and the dependence is nonlinear rather than linear. Its concrete proposal replaces the plain product with the geometric mean $F = (\\prod_{j=1}^t P(y_j=1 \\mid x, y_{<j}))^{1/t}$, which removes the length bias, and then multiplies $F$ by a Weibull-based shaping factor $R_s = C \\frac{k}{\\lambda}(\\frac{t}{\\lambda})^{k-1} e^{-(t/\\lambda)^k}$ with $C=10.735$, $k=1.5$, $\\lambda=8.0$, encoding a prior that chains of moderate length are best. In experiments on the six MATH-derived datasets, PSPO-WRS beats the Abel-7B baseline by between 6.56 and 30.94 percentage points and often matches or exceeds much larger models such as GPT-3.5 and Qwen2-72B; ablation shows the nonlinear module also raises the average number of generated steps from 2.624 to 3.051. The paper concludes that nonlinear, length-aware rewards are what make process supervision work.","pith_inferences":["The paper tests exactly one shaping curve, so its experiments do not yet separate 'nonlinearity helps' from 'this particular Weibull shape helps'; our inference is that a comparison against other nonlinear shapes, such as a log-length bonus or a bounded length cap, is needed to pin down the active ingredient.","The chosen peak at eight steps is likely tuned to the difficulty of the six datasets; our inference is that portable versions of the method would need a procedure for setting $\\lambda$ per task or per problem, otherwise the same method could hurt on puzzles whose correct chains are much shorter or much longer.","The geometric mean already removes the main length penalty by itself; our inference is that an ablation keeping the geometric mean but dropping the Weibull factor would reveal how much of the gain is due to normalization versus the nonlinear prior."],"forward_implications":["Process supervision pipelines should stop scoring chains with a raw product of step probabilities, because that construction systematically discourages longer reasoning.","Step-count-normalized rewards plus a length prior can be added to any PRM-trained policy, not just the specific base model used here.","Models trained this way generate more complete reasoning chains on average, which is the behavior the paper ties to higher accuracy.","Because the gain persists on out-of-distribution datasets, the improvement is attributed to stronger reasoning rather than to memorizing the evaluation sets."],"supporting_citations":[{"why":"Supplies the standard process-supervised reward model and product accumulation that PSPO-WRS modifies and ablates against.","marker":"Lightman et al. 2023"},{"why":"Establishes process versus outcome feedback, the task formulation that process supervision builds on.","marker":"Uesato et al. 2022b"},{"why":"Provides the six MATH-derived datasets and their expansion used for reward-model training and evaluation.","marker":"Chen et al. 2023"},{"why":"Provides Abel-7B, the base policy model and the main baseline that PSPO-WRS improves upon.","marker":"Chern et al. 2023"},{"why":"Motivates chain-of-thought reasoning and the efficiency-accuracy tradeoff behind the length prior.","marker":"Wei et al. 2022"},{"why":"Provides the CN-PPO baseline, the prior state of the art that PSPO-WRS compares against.","marker":"Liang et al. 2024"}],"fun_headline_variants":["Length-aware rewards fix process supervision's chain bias","Geometric-mean rewards eliminate chain-length penalty","Weibull-shaped rewards improve process supervision","Step-count-aware rewards beat six math reasoning sets","Nonlinear length-aware rewards beat linear scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-chosen Weibull curve, which gives the largest shaping reward to chains of about eight steps, correctly captures the true relationship between step count and reward quality; a single bad curve could erase the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Length-aware rewards fix process supervision's chain bias","Geometric-mean rewards eliminate chain-length penalty","Weibull-shaped rewards improve process supervision","Step-count-aware rewards beat six math reasoning sets","Nonlinear length-aware rewards beat linear scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001407,"raw_usage":{"total_tokens":5713,"prompt_tokens":998,"completion_tokens":4715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":4646}},"tokens_in":614,"tokens_out":4715,"duration_ms":35692,"temperature":1.0,"reasoning_tokens":4646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:14:44.813604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same PSPO-WRS pipeline and run it on problems whose correct solutions require very long chains, for example 15-25 steps, while also sweeping $\\lambda$ over a grid from 4 to 16 with all else fixed: the nonlinearity claim predicts the method should keep beating the linear baseline across a range of $\\lambda$, whereas the length-prior artifact predicts performance peaks only near $\\lambda=8$ and collapses on long-chain problems.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the six MATH-derived datasets and their expansion used for reward-model training and evaluation."},{"cited_title":"H.; Le, Q","cited_arxiv_id":null,"evidence_quote":"Motivates chain-of-thought reasoning and the efficiency-accuracy tradeoff behind the length prior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CN-PPO baseline, the prior state of the art that PSPO-WRS compares against."}],"review_version":1}