{"id":"2b488613-7792-4d38-b391-3967a6028f16","arxiv_id":"2412.01981","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A process reward model can be obtained from an outcome reward model by parameterizing rewards as beta log(pi_theta/pi_ref) and taking per-token log-ratio differences as step scores.","lead":"This paper shows that a step-by-step process reward model can be pulled out of a model trained only on final answer correctness, by writing the reward as a log-likelihood ratio. This could make fine-grained reward modeling much cheaper for math reasoning, since expensive step-level annotations are no longer needed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The identity in Prop. 3.1 is algebraically correct, but the claim that per-step differences are faithful process rewards is unvalidated; BoN accuracy alone does not distinguish a PRM from a dense token-level score.","rationale":"The proposition is correct: for any normalized πθ and πref, the prefix sum telescopes to β log E_{πref} exp(r/β). The proof in Appendix A is valid because ∑ πθ(y_{t+1}|y≤t)=1. Thus the theoretical contribution is an identity, not a constructive result. The load-bearing uncertainty is whether the per-step difference is a meaningful process reward. The paper adopts the advantage definition from Lu et al. and validates only BoN accuracy, which cannot discriminate between a step-correctness signal and a dense score that happens to rank completions well. A direct step-label correlation test would settle this. Until then, the verdict CONDITIONAL is appropriate; our stress-test does not change it.","tokens_in":17744,"tokens_out":7550,"duration_ms":73276,"concrete_test":"Evaluate the implicit PRM's per-step scores against step-level correctness labels on a benchmark such as PRM800K or the Math-Shepherd step labels used in Section 5.3. For each response, segment tokens into the same steps as the labels, compute the per-step score rt = qt - q_{t-1} (or the sum of token log-ratios within a step), and measure AUROC or Spearman correlation with the binary correctness label. If the AUROC is near chance while BoN accuracy remains high, the per-step scores are not faithful process rewards and the central claim should be weakened; if the AUROC is high, the process-reward interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 (Eq. 2) shows that any log-ratio-parameterized πθ yields a prefix sum that is an exponential expectation under πref; this is a true algebraic identity. However, the paper's central claim that this delivers a 'process reward' depends on the convention (Section 2, following Lu et al., 2024) that the advantage rt = qt - q_{t-1} is a faithful step-level correctness signal. The proposition provides no support for that premise: the per-token difference is simply β log(πθ(y_t|y<t)/πref(y_t|y<t)). The empirical protocol (Section 4.1) scores steps and takes the minimum for best-of-N, an outcome-level ranking metric that can be improved by a dense token-level filter even if step scores do not correspond to correct/incorrect steps. No experiment in the paper measures agreement with human or automated step labels, and Section 5.3's finding that extra step labels do not help is ambiguous. If the advantage definition is not accepted as a process reward, the contribution reduces to a free dense outcome score, and the title's 'process rewards' claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that an outcome reward model (ORM) trained with the reward parameterization r_θ(y) = β log(π_θ(y)/π_ref(y)) implicitly yields a process reward model (PRM). The key identity is Proposition 3.1: the prefix sum q_t^θ(y_{≤t}) = Σ_{i=1}^t β log(π_θ(y_i|y_{<i})/π_ref(y_i|y_{<i})) equals β log E_{π_ref(y|y_{≤t})} exp(r_θ(y)/β), so q_t can be interpreted as a Q-value. The per-step difference q_t − q_{t−1} is then presented as a process reward obtainable without step labels. The paper instantiates this idea with DPO, KTO, NCA, and CE losses, trains on 33K math instructions with eight responses each, and evaluates best-of-N accuracy on MATH-500 with three different generation models. It reports that the implicit PRMs outperform reimplemented Math-Shepherd and AutoPSV at substantially lower FLOPs, and it analyzes scaling behavior, data composition, majority voting, and the role of the reference model.","tokens_in":17896,"tokens_out":5945,"duration_ms":49223,"significance":"If the process-reward interpretation is accepted, the contribution is significant: it removes the need for step-level annotations, connects standard preference-learning objectives to dense token-level rewards, and generalizes the Q-function insight of Rafailov et al. (2024) beyond DPO. The algebraic identity in Proposition 3.1 is correct and elegantly derived, and the empirical study is broad in terms of objectives, data scaling, and inference-cost accounting. The paper also releases code and data, which supports reproducibility. However, the significance is conditional: the claim that the per-step difference is a faithful step-level correctness signal is not directly validated. All experiments use outcome-level best-of-N ranking, which can be improved by a dense token-level filter even if the per-step scores do not correspond to correct or incorrect steps. The paper's central novelty therefore needs either a step-level validation or a more cautious framing as a free dense outcome reward.","major_comments":[{"comment":"The claim that a 'process reward' is obtained for free rests on the convention (Section 2) that r_t^θ = q_t^θ − q_{t−1}^θ is the process reward. This is a definitional choice, not a consequence of Proposition 3.1: by construction the difference equals β log(π_θ(y_t|y_{<t})/π_ref(y_t|y_{<t})). The paper does not test whether these per-step values agree with human or automatic step-level labels, and the best-of-N protocol in Section 4.1 scores each step and takes the minimum, a procedure that any dense token-level score can improve. The central 'free process reward' claim is therefore unvalidated; the authors should either add a step-level evaluation (e.g., agreement with PRM800K or ProcessBench labels) or explicitly reframe the contribution as a free dense outcome reward.","section":"Section 3, Eq. (3)"},{"comment":"The experiment augmenting an implicit PRM with Math-Shepherd step labels is the only direct evidence about the process-reward interpretation, and it is inconclusive. It uses a single second-stage objective (step-level KTO) on MCTS-generated labels, and the paper itself acknowledges annotation noise and potential suboptimality of the training algorithm as confounds. This result cannot establish that process labels are generally unnecessary; it only shows that this particular recipe provides no gain. The conclusion that 'we can already train a strong PRM without process label' goes beyond what this experiment supports.","section":"Section 5.3, Table 2"},{"comment":"Proposition 3.2 is stated without proof, and its notation is inconsistent with Eq. (1): the lower bound is written as E_{π_ref(y|y<t)} r_θ(y) in the proposition, while Eq. (1) defines q_t^s as E_{π_ref(y|y≤t)} r_θ(y). The inequality chain and the equality conditions (β→∞ and β→0) are plausible, but because the proposition is load-bearing for the claim that the implicit Q-value is more accurate than MCTS-based hard and soft estimates, it needs a correct statement and a proof or at least a derivation in the appendix.","section":"Proposition 3.2, Eq. (4)"}],"minor_comments":[{"comment":"The manuscript contains many typos, including 'requies' (Introduction), 'parameterizin' (Introduction), 'perfroms' (Section 2), 'traiinng' (Section 5.2), 'Balanceed' (Table 1), 'real-worl' (Section 5.2), 'the the' (Section 5.5), and 'b larger' (Section 5.5.1). The paper would benefit from a careful proofreading pass.","section":"Throughout"},{"comment":"The table caption says 'Implicit PRM' but the text in Section 5.3 says the results are for 'implicit PRM (DPO)'; the caption should state the objective explicitly, since DPO, KTO, NCA, and CE have different performance profiles.","section":"Table 2"},{"comment":"The phrase 'the only assumption' overstates the case: in addition to the log-ratio parameterization, the method requires a reference model, a choice of β, and a policy that is a causal LM. These are not all stated as assumptions in the introduction.","section":"Abstract and Section 1"},{"comment":"The statement that the proposition holds under a uniform reference model is only true for relative comparisons in best-of-N selection, not for the absolute Q-value interpretation; the text later makes this clear, but the wording in the first paragraph of Section 5.5.2 could be sharper.","section":"Section 5.5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be influential because the algebraic identity is clean and the empirical results are strong when viewed as an outcome-level dense scorer. The main risk is overclaiming: without step-level validation, the 'process reward' terminology is not earned by the evidence. I would encourage the editor to require a step-level evaluation or a revised framing before publication. The paper is within scope for a general machine-learning venue, but the title and abstract currently promise more than the experiments deliver."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper proves an identity that was already implicit in Rafailov et al. (2024) and turns it into a practical recipe. If you parameterize the outcome reward as beta * log(pi_theta/pi_ref), the prefix sum of per-token log-ratios is exactly the exponential expectation under pi_ref, and the per-token difference is automatically a dense score. The algebra in Prop 3.1 is correct, and the proof is clean. What is actually new is the concrete instantiation with CE loss on unpaired, imbalanced data, including the one-response-per-instruction case, plus the scaling and reference-ablation study. Those are genuinely useful contributions, and the authors cite the concurrent entropy-regularized PRM work and Rafailov's Q-function paper honestly.\n\nThe soft spots are real but not fatal. First, the step from \"dense score\" to \"process reward\" rests on the advantage convention q_t - q_{t-1}, which is definitionally the per-token log-ratio. No experiment tests whether those step scores agree with human or automated step-correctness labels. Best-of-N with the minimum step score is an outcome-level ranking metric that a generic token-level filter can improve without step scores being meaningful. So the title slightly overclaims: you get a free dense score, and calling it a process reward depends on adopting the convention. Second, the empirical evidence is single-benchmark (MATH-500), no error bars, and beta is hand-picked. Third, the step-label augmentation result (Sec 5.3) is ambiguous, though the authors honestly flag the noise in MCTS labels and the limited algorithm choice.\n\nStill, the central practical claim holds up: training an ORM with CE on outcome labels gives you a strong reranker, cheap to train and data-efficient. The math is solid, the experiments are careful within their scope, and the negative results are reported without spin.\n\nWho is this for: people building reward models for math reasoning, especially under data scarcity, and anyone working on process supervision. It deserves a serious referee, not a desk reject. I would accept it for review, then push for a title that says \"dense token-level rewards\" rather than \"process rewards,\" and for an explicit step-level validation—e.g., correlation with step-wise correctness on a small labeled set—before the claims are taken at face value.","headline":"A correct, useful identity that turns outcome labels into a dense token-level score; whether that score is a true process reward is asserted, not yet validated.","tokens_in":18532,"tokens_out":3127,"would_cite":true,"duration_ms":32564,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Step-by-step reward models for free, no step labels needed","keywords":["implicit process reward model","outcome reward model","log-likelihood ratio reward","direct preference optimization","cross-entropy loss","best-of-N sampling","mathematical reasoning","process supervision"],"falsifier":"Train an implicit PRM on a dataset with gold step-level correctness labels, then compare its per-step scores with those labels; if correct steps do not reliably receive higher scores than incorrect steps, the claimed process reward is not a usable correctness signal.","tokens_in":17470,"feed_emoji":"🧮","tokens_out":8554,"duration_ms":74378,"temperature":0.7,"pith_summary":"This paper sets out to show that a dense, step-by-step reward model, normally trained with labels on every intermediate reasoning step, can instead be obtained at no extra cost from an outcome-level reward model. The key move is to parameterize the outcome reward as the log-likelihood ratio $\\beta \\log(\\pi_\\theta(y)/\\pi_{\\mathrm{ref}}(y))$ and then read per-step scores off the same language model. The paper proves that the cumulative sum of per-token log-ratios equals an expectation of the outcome reward under the reference model, so the difference between consecutive prefix sums is a process reward. If this is right, the largest cost of building process reward models—step-level annotation—disappears, and the resulting models match or beat Monte-Carlo-tree-search baselines on math reasoning.","feed_headline":"Step-by-step reward models for free, no step labels needed","feed_subtitle":"A reward parameterization turns outcome-only training into a dense process verifier, beating costly baselines on MATH.","key_machinery":"The load-bearing object is the telescoping identity for prefix log-likelihood-ratio sums. Because $q_t^\\theta$ is an exact expectation of the outcome reward under the reference distribution, subtracting consecutive prefix sums isolates the incremental contribution of step $t$, turning any outcome-trained model with the reward parameterization into a dense verifier. The paper uses this identity to define process rewards as advantages, the convention it adopts from prior work.","core_discovery":"The central claim is Proposition 3.1: with the reward $r_\\theta(y) = \\beta \\log(\\pi_\\theta(y)/\\pi_{\\mathrm{ref}}(y))$, the prefix score $q_t^\\theta(y_{<t}, y_t) = \\sum_{i=1}^t \\beta \\log(\\pi_\\theta(y_i|y_{<i})/\\pi_{\\mathrm{ref}}(y_i|y_{<i}))$ is exactly $\\beta \\log \\mathbb{E}_{\\pi_{\\mathrm{ref}}(y|y_{\\le t})} e^{r_\\theta(y)/\\beta}$, the Q-value of the outcome reward at step $t$. The per-step difference $q_t - q_{t-1}$ therefore gives a process reward without any step-level supervision. The paper also argues that this implicit Q-value lies between the soft and hard Monte-Carlo estimates and that it can be instantiated with DPO, KTO, NCA, or cross-entropy loss. On MATH best-of-N reranking, the resulting models outperform MCTS-style step-labeling baselines while consuming a small fraction of the data-collection FLOPs.","pith_inferences":["Editorial inference: if the identity transfers beyond math, any outcome-labeled dataset—code, dialogue, tool use—could yield dense token-level rewards without annotation; a direct test would compare the per-step scores with human step judgments.","Editorial inference: the observation that the reference model can be dropped suggests its role is mainly a training-time KL anchor, not a required inference component; this could be tested on more diverse tasks.","Editorial inference: the null result for extra step labels may reflect the noise of MCTS labels and the advantage convention rather than a general uselessness of step supervision; a cleaner test uses human-annotated steps."],"forward_implications":["Process reward models can be trained from outcome labels alone, eliminating step-label annotation and the look-ahead trajectory sampling it requires.","Any preference-learning objective that uses the log-likelihood-ratio reward parameterization, plus cross-entropy on unpaired data, yields a dense verifier at no extra training cost.","On MATH best-of-N reranking, the implicit PRM outperforms MCTS-based step-labeling baselines while using less than 1/38 of the data-collection FLOPs.","Scaling the number of responses per instruction helps more than scaling the number of instructions, and cross-entropy remains useful even with one response per instruction.","Adding step labels from the MCTS pipeline does not further improve the outcome-trained implicit PRM in the reported experiments."],"supporting_citations":[{"why":"Gives the implicit reward parameterization $r_\\theta(y)=\\beta\\log(\\pi_\\theta(y)/\\pi_{\\mathrm{ref}}(y))$ that the whole construction rests on.","marker":"Rafailov et al., 2023"},{"why":"Shows DPO training learns an implicit Q-function; the paper generalizes this property to other objectives.","marker":"Rafailov et al., 2024"},{"why":"Defines the MCTS-based step-labeling method and the hard-estimate Q-value that serve as the main comparison baseline.","marker":"Wang et al., 2023"},{"why":"Defines process rewards as advantages $q_t-q_{t-1}$ and supplies the AutoPSV baseline.","marker":"Lu et al., 2024"},{"why":"Defines process reward models and the best-of-N evaluation protocol used on MATH.","marker":"Lightman et al., 2023"},{"why":"Supplies the KTO objective used as one instantiation of the implicit PRM.","marker":"Ethayarajh et al., 2024"},{"why":"Argues for using advantages as process rewards, supporting the per-step difference convention.","marker":"Setlur et al., 2024"}],"fun_headline_variants":["Outcome rewards secretly deliver process rewards for free","No step labels? No problem: outcome-only training yields step rewards","Implicit process rewards from outcome-only training: free lunch","Process rewards without process labels: cheaper, better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the change in the prefix score from one step to the next is a meaningful measure of how good that step is, rather than just a number that happens to be computable.","fun_headline_variants_meta":{"raw":{"variants":["Outcome rewards secretly deliver process rewards for free","No step labels? No problem: outcome-only training yields step rewards","Implicit process rewards from outcome-only training: free lunch","Process rewards without process labels: cheaper, better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1622,"prompt_tokens":1116,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":732,"tokens_out":506,"duration_ms":5673,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:57:46.212366+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an implicit PRM on a dataset with gold step-level correctness labels, then compare its per-step scores with those labels; if correct steps do not reliably receive higher scores than incorrect steps, the claimed process reward is not a usable correctness signal.","supporting_citations":[],"review_version":1}