{"id":"629ff4c2-d8ce-463a-8503-fb9af4e93175","arxiv_id":"2506.17533","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A reward model that multiplies stepwise correctness and potential scores improves best-of-N verification accuracy for math reasoning.","lead":"DuaShepherd combines two kinds of step-level feedback for math problem solving: whether each step is correct, and whether the solution is still on track to reach a right answer. The authors show that multiplying these two scores, trained together in one model, improves how well a verifier picks good solutions on MATH500 and ProcessBench.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ProcessBench comparison may be invalid: authors fix the threshold at 0.5 instead of the official GSM8K-tuned threshold, so the Table 2 state-of-the-art claim is not comparable to published baselines.","rationale":"The reader's stated weakest assumption is that Eq. (5) is not a valid compound probability because the two rewards are not conditionally independent, with the 0.812 correlation as evidence. I agree that the 'chain rule' framing is unsupported, but I do not think that is the most load-bearing concern: multiplication can still work as a heuristic even if it is not a true probability. The more decisive weakness is that the ProcessBench evaluation protocol deviates from the official instructions in a way that breaks comparability with every published baseline in Table 2. This is an explicit, locatable statement in Section 4.1, and it directly affects the central claim of state-of-the-art performance. The reader's rationale also mentions this fixed-threshold issue, so there is partial agreement, but it was not identified as the weakest assumption. Because the concern is about evaluation validity rather than the internal soundness of the reward combination, and because the MATH500 results provide some independent support, the appropriate outcome remains conditional acceptance: the empirical claims should be re-verified under the official ProcessBench protocol and with significance or sensitivity analysis before the state-of-the-art statement is accepted.","tokens_in":17454,"tokens_out":5191,"duration_ms":63193,"concrete_test":"Re-run the ProcessBench evaluation of Table 2 under the official protocol: for each scalar-output PRM (DuaShepherd, its correctness and potential heads, PRM800K, Math-Shepherd, and the marked baselines), select the threshold on the GSM8K subset that maximizes F1 and report per-split and average scores. If DuaShepherd's average drops from 65.5 toward the baseline range, or its margin over Qwen2.5-Math-7B PRM800K at 57.5 narrows materially, the state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is 'consistent performance improvements across multiple benchmarks' and 'state-of-the-art under comparable resource constraints,' with the strongest evidence in Table 2 where DuaShepherd reaches 65.5% average on ProcessBench versus 57.5% for the PRM800K model and 42.1% for Skywork-PRM-7B. Section 4.1 states a critical deviation: 'for models that directly output a scalar value, we do not select the threshold based on the GSM8K subset of the data. Instead, we fix it at 0.5.' The baselines marked with * in Table 2 are taken directly from the ProcessBench paper, which uses the official protocol of selecting the threshold on the GSM8K subset. Fixing the threshold at 0.5 changes the operating point for every scalar-output model, and F1 for first-error detection can shift substantially when score distributions are not centered at 0.5. Thus the headline margin over baselines may be an artifact of comparing a favorable fixed threshold against baselines evaluated at their own tuned thresholds, rather than evidence that the compound reward is superior. The independence critique of Eq. (5) is a conceptual issue, but this protocol mismatch directly threatens the empirical claim that motivated the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DuaShepherd, a reward-modeling framework that combines two step-level signals: a correctness reward (trained on PRM800K, with neutral and positive labels collapsed to correct) and a potential reward (trained on Math-Shepherd's Monte-Carlo-derived labels). The two heads share a Qwen2.5-Math-7B base and are trained with multi-task binary cross-entropy; at inference the final score is the product R_correctness * R_potential, presented as a compound probability via a chain-rule argument (Eq. 5). The authors construct a pseudo-labeled training set by annotating Math-Shepherd steps with both a correctness model and a potential model, and evaluate on MATH500 best-of-64 selection and ProcessBench first-error detection, reporting gains over either single reward and over the separately trained teachers, and claiming state-of-the-art under comparable resource constraints (Tables 1 and 2). The paper also reports analyses of reward correlation, alternative mixing approaches, and a weak-to-strong scaling experiment.","tokens_in":17658,"tokens_out":2628,"duration_ms":32691,"significance":"The core idea—combining a backward-looking correctness signal with a forward-looking potential signal—is timely and practically motivated, and the automated pipeline that reuses PRM800K and Math-Shepherd without additional sampling is a genuine strength. The multi-head multi-task formulation is simple and plausible, and the head-level comparisons in Table 1 provide some evidence that joint training helps. The paper also contains useful analyses, including the reward-scatter diagnostics of Section 6.2 and the comparison of mixing strategies in Table 4. However, the headline ProcessBench claim is weakened by a protocol deviation, the compound-probability derivation in Eq. (5) is not actually derived, the potential-RM label validation in Table 3 is circular, and the reported numbers lack error bars or significance tests. If the ProcessBench comparison is re-run under the official protocol and the product rule is reframed as a heuristic combination, the contribution can be salvageable, but as written the central empirical claim is not fully supported.","major_comments":[{"comment":"The ProcessBench evaluation protocol is not comparable to the published baselines. The paper fixes the threshold at 0.5 for all models that output a scalar, whereas the starred baselines taken from the ProcessBench paper use the official protocol of tuning the threshold on the GSM8K subset. First-error-detection F1 is threshold-dependent, so comparing a fixed-threshold DuaShepherd against tuned-threshold baselines can inflate or deflate the reported margin. The authors should re-evaluate all scalar-output models, including their own, under the official GSM8K-tuned protocol, or report the threshold sweep curve; without this, the state-of-the-art claim in the Abstract and Section 5.1 is not supported.","section":"§4.1, Table 2"},{"comment":"Eq. (5) is presented as following from the chain rule of probability, but no conditioning event or independence assumption is stated. Moreover, Section 6.2 reports an overall Pearson correlation of 0.812 between the two reward scores, so the two signals are far from conditionally independent, and each score is a sigmoid output rather than a probability over a well-defined joint event. The product is a heuristic fusion, not a derived compound probability. The authors should either state the simplifying assumption explicitly and test it, or present the product as a heuristic aggregation with appropriate justification.","section":"§3.3, Eq. (5)"},{"comment":"The validation of the potential reward model's pseudo-labels is circular. The potential RM was trained on Math-Shepherd binary labels and is then used to generate soft pseudo-labels on Math-Shepherd; comparing those pseudo-labels to the original Math-Shepherd labels measures consistency with the training target, not label quality. This part of Table 3 should be removed or replaced with an evaluation against an independent reference (e.g., the o1-mini annotations used for the correctness RM, or human judgments).","section":"§6.1, Table 3"},{"comment":"No error bars, multiple seeds, or significance tests are reported, yet the central claim is 'consistent performance improvements' and several reported margins are small (e.g., 0.498 vs 0.526 on MATH500 with Mistral, and 63.0% vs 65.5% on ProcessBench). Given the stochasticity of best-of-N evaluation and PRM training, the authors should provide at least bootstrap confidence intervals over the 500 MATH500 problems or averaged results over independent training runs.","section":"Tables 1 and 2"},{"comment":"The claimed consistency of the compound reward is not always observed. For DeepSeekMath-Instruct-7B, the DuaShepherd correctness head alone (0.626) outperforms the full DuaShepherd compound (0.622), and for Qwen-2.5-Math-Instruct-7B the potential head alone matches the compound (0.824 vs 0.824). The text should acknowledge these exceptions and discuss when the product helps versus when it merely tracks the stronger head.","section":"§5.1, Table 1"}],"minor_comments":[{"comment":"There is a typo in 'DuaShepherd!over majority voting' in the paragraph beginning 'As shown in Table 1'.","section":"§5.3"},{"comment":"The introduction contains a duplicated sentence: the two sentences beginning 'Although PRM800K requires steps labeled as positive...' are repeated verbatim.","section":"§1"},{"comment":"The caption refers to 'Pcorrectness' and 'Ppotential', while the axis labels in the figure and the text use 'PRM Score' and 'Shepherd Score'; the notation should be unified to match Eqs. (2)–(3).","section":"Figure 3"},{"comment":"The paper alternates between 'MATH500' and 'Math500'; standardize the capitalization.","section":"§4.1"},{"comment":"The contribution list has a typo: 'final answer..' should be 'final answer.'","section":"§1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something simple and mostly honest: it trains a two-headed reward model to predict step-level correctness (from PRM800K-style labels) and forward potential (from Math-Shepherd-style labels), then multiplies the two scores to rank solutions. The multi-task training is a sensible extension of ArmoRM, and the MATH500 best-of-N results with Mistral and DeepSeek generators show the product beating either head alone. That is a real, if incremental, finding.\n\nThe new part is the pseudo-labeling pipeline that creates soft labels for both signals on Math-Shepherd data, and the observation that joint training helps both heads. I believe those results. The paper is clear and the method is easy to replicate.\n\nThe soft spots are mostly in the evaluation, not the idea. First, for ProcessBench they change the protocol: instead of selecting the decision threshold on the GSM8K subset, they fix it at 0.5 for all scalar-output models. The baselines marked with * are taken from the ProcessBench paper, which uses the tuned threshold. Comparing your fixed-threshold numbers to tuned-threshold baselines is not “state-of-the-art under comparable conditions.” It may be the reason DuaShepherd shows a 65.5 vs 42.1 margin over Skywork-PRM-7B. That claim should be re-run with the official protocol before being taken seriously.\n\nSecond, Eq. (5) is not a chain-rule derivation. It is a product of two sigmoid outputs, and the paper’s own reported Pearson correlation of 0.812 between the rewards shows the independence assumption is false. The product may still be a useful heuristic, but calling it a “compound probability” is misleading.\n\nThird, the label-quality check for the potential RM in Table 3 compares its pseudo-labels to Math-Shepherd’s binary labels — the very labels used for training. That’s a consistency check, not an accuracy check. The correctness RM check against o1-mini is somewhat better, but the o1-mini F1 is itself lower than the reported DuaShepherd ProcessBench score, so there’s a ceiling concern.\n\nMinor: no error bars, no significance tests, no released code or data. Also the “consistent improvements” claim is overstated: on the Qwen generator the product ties the better head (0.824).\n\nWho is this for? Anyone working on process reward models or test-time verification for math reasoning. The method is cheap to try and the paper is worth reading and discussing. It deserves a serious referee, but the ProcessBench protocol issue must be fixed before the SOTA claim can be trusted.","headline":"A simple, plausible combination of two PRM signals that is undercut by a non-standard ProcessBench threshold choice; worth a referee but not the SOTA claim as written.","tokens_in":18224,"tokens_out":3304,"would_cite":true,"duration_ms":36300,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multiplying a step-correctness score by a forward-potential score yields a compound reward that beats either signal alone in LLM math verification.","keywords":["process reward models","mathematical reasoning","correctness reward","potential reward","compound probability","multi-task learning","best-of-N verification","pseudo-labeling"],"falsifier":"On the same MATH500 candidates, re-rank the solutions with the product $R_c R_p$, the minimum, and a weighted sum of the two heads; if the product does not beat the weighted sum specifically on trajectories where the two scores disagree (for example, a wrong step followed by a correct answer), then the claimed chain-rule mechanism is not what is carrying the gain. A more direct probe is to train a version of the model on synthetic data where correctness and potential are deliberately decoupled and see whether the product's advantage tracks that decoupling.","tokens_in":17215,"feed_emoji":"🧮","tokens_out":12055,"duration_ms":116418,"temperature":0.7,"pith_summary":"The paper sets out to establish that process reward models for mathematical reasoning judge better when each reasoning step gets two scores: one for whether the step itself is correct, and one for whether the path still leads to a correct final answer. The two scores are trained as separate heads on a shared base model and combined by multiplication, which the authors treat as a compound probability via the chain rule. On MATH500 best-of-64 with a Mistral-7B generator, the combined reward scores 0.526, versus 0.472 for either the correctness-only or the potential-only reward; on ProcessBench the full DuaShepherd model averages 65.5 percent, up from 57.5 percent for the correctness-only model. The point of caring is practical: the training labels come from two existing datasets plus pseudo-labeling, so the improvement does not require new human step annotations. If the claim is right, a cheap dual-signal recipe gives better verifiers for selecting among candidate math solutions.","feed_headline":"Multiplying correctness and potential beats single rewards","feed_subtitle":"Step-level verifier hits 65.5% on ProcessBench and 0.526 best-of-64 on MATH500 with a Mistral generator.","key_machinery":"The load-bearing identity is Eq. (5), $R_{\\mathrm{DuaShepherd}} = R_{\\mathrm{correctness}} \\cdot R_{\\mathrm{potential}}$, applied to every step. Each factor is a sigmoid output from a linear head on a shared base model's hidden state; the heads are trained with summed binary cross-entropy on a dataset built by taking Math-Shepherd trajectories, keeping the Monte Carlo potential labels, and pseudo-labeling correctness with a reward model trained on PRM800K. The product is justified as a chain-rule compound probability, and the paper compares it against using either head alone, distilling the product into a single head, and multiplying two independently trained models' scores.","core_discovery":"The central claim is that step correctness and forward potential are complementary, measurable properties of a reasoning step, and that their product, $R_{\\mathrm{DuaShepherd}} = R_{\\mathrm{correctness}} \\, R_{\\mathrm{potential}}$, is a more reliable stepwise reward than either factor alone. Correctness looks backward, flagging whether any step so far is erroneous; potential looks forward, estimating whether the solution still ends at the right answer. The paper shows that multiplying two separately trained rewards already improves on both teachers, and that jointly training both heads in one multi-head model improves further, with each DuaShepherd head outperforming its single-task teacher. It also reports a weak-to-strong pattern in which a larger model trained on the product-style pseudo-labels does even better, which the authors take as evidence that the multiplication is surfacing a genuine underlying signal about reasoning quality.","pith_inferences":["Because the paper reports an overall Pearson correlation of 0.812 between the two rewards, the chain-rule interpretation is not strictly supported; a plausible reading is that the product acts as an AND-like gate that rejects candidates scoring low on either signal, and this heuristic is what yields the gains.","A testable consequence is that the product's edge over its factors should grow when the two signals are deliberately decorrelated; building a probe set where correctness and potential disagree and measuring the gap would separate the complementarity story from a simple weighting effect.","The weak-to-strong result suggests the pseudo-labeled product data may encode a generalizable preference, but the reported gains for stronger generators shrink; training on generator-aligned samples is the direct experiment that would show whether the dual-signal advantage survives distribution shift."],"forward_implications":["Best-of-N solution selection improves on MATH500 without new human annotation, because the dual labels come from existing datasets plus pseudo-labels.","Multi-task training strengthens each individual head relative to its single-task teacher, so the benefit is not only in the final product but in the shared representation.","ProcessBench error-detection accuracy also rises, showing the dual signal helps locate the first wrong step, not just pick final answers.","The automated pipeline is scale-expandable, so the same dual-label recipe can be regenerated for newer generators as their solution styles drift from the original training data."],"supporting_citations":[{"why":"supplies the PRM800K human step-correctness labels that the correctness reward head is trained on.","marker":"Lightman et al. (2024)"},{"why":"supplies the Math-Shepherd Monte Carlo potential labels and the trajectory pool used to build the DuaShepherd dataset.","marker":"Wang et al. (2024b)"},{"why":"provides ProcessBench, the data and protocol used to measure first-error detection accuracy.","marker":"Zheng et al. (2024)"},{"why":"supplies the MATH benchmark from which the MATH500 test set is drawn for best-of-N evaluation.","marker":"Hendrycks et al. (2021)"},{"why":"provides the Qwen2.5-Math-7B base model used for the reward heads and for generators.","marker":"Qwen (2024a)"},{"why":"motivates the shared multi-head multi-task architecture for learning both rewards.","marker":"Wang et al. (2024a)"},{"why":"frames the weak-to-strong generalization interpretation of the 72B improvement.","marker":"Burns et al. (2023)"},{"why":"supplies LoRA, used to fine-tune the 72B model under computational constraints.","marker":"Hu et al. (2022)"}],"fun_headline_variants":["Correctness times potential: better step rewards","Dual reward heads beat single ones for math","Multiplying two signals beats each alone","Stepwise correctness plus potential wins","New reward combo boosts math reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise the whole construction rests on is that 'this step is correct' and 'this step leads to the right answer' are separate enough that their scores can simply be multiplied into one probability; the paper's own reported correlation of 0.812 between the two scores shows that independence is not actually there, so the compound probability is a heuristic rather than a derived chain-rule product.","fun_headline_variants_meta":{"raw":{"variants":["Correctness times potential: better step rewards","Dual reward heads beat single ones for math","Multiplying two signals beats each alone","Stepwise correctness plus potential wins","New reward combo boosts math reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1159,"prompt_tokens":870,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":486,"tokens_out":289,"duration_ms":3379,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:29:55.089926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same MATH500 candidates, re-rank the solutions with the product $R_c R_p$, the minimum, and a weighted sum of the two heads; if the product does not beat the weighted sum specifically on trajectories where the two scores disagree (for example, a wrong step followed by a correct answer), then the claimed chain-rule mechanism is not what is carrying the gain. A more direct probe is to train a version of the model on synthetic data where correctness and potential are deliberately decoupled and see whether the product's advantage tracks that decoupling.","supporting_citations":[],"review_version":1}