{"id":"11cc6912-087c-4d03-a803-788a9cd4b7e6","arxiv_id":"2510.00492","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Across 14 domains, a generative outcome reward model (gORM) outperforms both discriminative and generative process reward models for selecting best-of-N LLM reasoning chains.","lead":"This paper compares four ways to score AI reasoning chains—checking only final answers versus checking each step, and using discriminative classifiers versus generative judges—across 14 knowledge domains. It finds that a generative judge that checks only the final answer is the most reliable, contrary to the common preference for step-by-step process reward models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theory-experiment mismatch: PRM length-error lower bounds assume product aggregation, while experiments use min (dPRM) and final-verdict (gPRM) aggregation.","rationale":"The reader's weakest assumption identifies exactly this theory-experiment aggregation mismatch. The central claim that PRMs fail due to length-compounding errors depends on theorems that analyze product aggregation, yet the controlled experiments use min for dPRM and final-verdict for gPRM. This is an internal consistency issue, not a disagreement with consensus. The proposed concrete test would settle whether the mismatch matters. The reader's verdict of CONDITIONAL is appropriate; my read does not move it, so I mark UNCHANGED. Other concerns (e.g., overclaiming 'consistent gains' without error bars, gPRM training-set size) are secondary but reinforce the conditional status.","tokens_in":32522,"tokens_out":1952,"duration_ms":19202,"concrete_test":"On MMLU-Pro, evaluate dPRM with both min and product aggregation, and gPRM with both final-verdict normalized probability and product-of-step normalized probabilities, using the same trained models. Plot F1 vs. CoT-length bins. If min/product or final-verdict/product show different length trends (e.g., product degrades but min does not), then Theorems 4.2/4.3 do not explain the reported results. Conversely, if both aggregation schemes exhibit the same length-dependent degradation, the mismatch is inconsequential.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central explanation for why PRMs fail in multi-domain TTS rests on Theorems 4.2 and 4.3, which prove that the log-error of dPRM and gPRM grows linearly with CoT length T. However, the theory in Appendix A.1 defines f_hat_dPRM(x) := prod_t u_hat_t(x_{1:t}) and treats gPRM as a product of step-correctness probabilities, whereas the experiments (§2.2, §3) use minimum aggregation for dPRM and normalized final-verdict probability for gPRM (Eqs. 8–9). If product and min (or product and final-verdict) have different error-vs-length behavior, then the theorems do not explain the empirical degradation of PRMs in Figures 12 and 6. The empirical support (Fig. 12) is correlational and does not distinguish aggregation scheme from other confounds (e.g., label noise, length distribution shift). Since the paper's headline claim — that stepwise aggregation compounds errors — is precisely this theoretical explanation, the mismatch is load-bearing. If the theory only applies to aggregation schemes not used in the experiments, the attribution of gORM's robustness to the length-error mechanism is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a unified comparison of four reward-model variants for best-of-N test-time scaling: discriminative outcome (dORM), discriminative process (dPRM), generative outcome (gORM), and generative process (gPRM) reward models. The evaluation spans a math domain (PRM800K/ProcessBench) and a 14-domain multi-domain setting (MMLU-Pro), with multiple generator LLMs and shared reward backbones. The authors report that, consistent with prior work, dPRM beats dORM and generative variants beat discriminative ones in math; in the multi-domain setting, dORM is on par with dPRM, gPRM is not competitive, and gORM is overall the most robust. They propose two explanations: (i) stepwise scoring/aggregation compounds errors as reasoning chain length grows, supported by log-error lower bounds (Theorems 4.1–4.3); and (ii) label noise and consensus-filtering-induced length distribution shift hurt gPRM. Code, data, and checkpoints are released.","tokens_in":32868,"tokens_out":10103,"duration_ms":80942,"significance":"If the empirical ranking holds, the paper challenges the common assumption that fine-grained process supervision always improves verifier performance in multi-domain test-time scaling, and it provides a practical comparison that is more controlled than much prior work (shared backbones, four variants, multiple generator LLMs). The release of code, datasets, and checkpoints is a concrete strength that will facilitate follow-up work. The theoretical analysis, however, is not tightly connected to the experimental protocols: the dPRM theorem analyzes product aggregation while the experiments use minimum aggregation, and the gPRM theorem concerns single-sample product scores while the deployed estimator is an M-sample average of final-verdict probabilities. The headline abstract claim of consistent significant gains across every domain is also stronger than the per-domain results support. The empirical study itself is valuable and likely correct in its main qualitative trends, but the explanatory narrative needs substantial revision.","major_comments":[{"comment":"Theorem 4.2 analyzes a product-aggregated dPRM: Appendix A.1 defines f_hat_dPRM(x) := ∏_{t=1}^T û_t(x), and the lower bound E[Δ²_dPRM] ≥ (σ²−2γ)T is derived for this product. In contrast, §2.2 states that at test time dPRM 'aggregat[es] the step rewards with the minimum' (following Zeng et al., 2025). Min aggregation has different length behavior: it is dominated by the worst step rather than by the accumulation of all step errors, so the theorem does not bound the error of the estimator actually evaluated in Figures 3–7 and 12. The empirical support in Fig. 12 (Left) uses min-aggregated dPRM and therefore cannot rescue the mismatch. This is load-bearing: the paper's central explanation for why PRMs underperform in multi-domain TTS is precisely that 'stepwise aggregation compounds errors as reasoning length increases,' and that claim is not established for the dPRM variant used in the ex","section":"§4.1 / Theorem 4.2 vs. §2.2, Appendix A.1"},{"comment":"Theorem 4.3 bounds the log-error of Δ_gPRM defined in Appendix A.1 as the log of a single-sample product of step-correctness probabilities, ∏_t F_t(x, v_{≤t}). The test-time estimator in Eq. 8 averages M samples of the normalized final-verdict probability p_gPRM(y=1 | x, v_{1:L+}), not the log of a single product. While p_gPRM(y=1|x,v) factors as a product when the final 'Yes' is deterministically equivalent to all steps being correct, the paper does not supply a rigorous bridge from the single-sample lower bound to the M-sample average used with M=10. Theorem A.4 addresses a mean predictor (E[e^L]) rather than the finite-M sample average, and the main text's pointer to 'cf. Eq. 5' is incorrect — Eq. 5 is the gORM estimator, not the gPRM estimator of Eq. 8. The theoretical support for gPRM's empirical degradation is therefore incomplete, and the stated linear-in-T lower bound is not dire","section":"§4.1 / Theorem 4.3 vs. Eqs. 8–9, Appendix A.1"},{"comment":"The abstract claims that gORM yields 'significant and consistent gains across every tested domain.' This is not supported by the paper's own per-domain results in Fig. 6. In Psychology, dORM (81.9) and dPRM (81.7) both exceed gORM (81.3); in Health, dPRM (79.2) exceeds gORM (78.2). The weaker statement in §3.2 that gORM outperforms the others 'without notable degradation in any domain relative to the others' is accurate, but the abstract overstates the finding. Furthermore, §D reports a single run for the multi-domain setting, so the word 'significant' is not backed by error bars or statistical tests. The manuscript should either soften the abstract to match the per-domain results or provide significance testing that justifies the stronger claim.","section":"Abstract / §3.2 / Fig. 6"}],"minor_comments":[{"comment":"Typo: 'Byeond Khalifa et al. (2025)' should be 'Beyond Khalifa et al. (2025)'.","section":"§2.2"},{"comment":"The sentence referencing 'Theorem. A.4' for gPRM with Monte Carlo estimation cites 'cf. Eq. 5'; this should be Eq. 8, since Eq. 5 defines the gORM estimator.","section":"Main text, after Theorem 4.3"},{"comment":"The correlation values in Fig. 12 (Right) are reported without confidence intervals or p-values. Given only 8 binned length categories (or 6 on the right), these correlations should be interpreted cautiously; adding a scatter plot with per-bin standard errors would strengthen the claim.","section":"§4.1 / Fig. 12"},{"comment":"The Wasserstein distance histograms in Fig. 14 aggregate over all domains; per-domain distances in Table 3 show large variation (e.g., Chemistry 6.922 vs. History 0.581 for gPRM). A per-domain scatter with domain labels would make the 'length shift hurts gPRM' mechanism more transparent than the single pooled number.","section":"§4.2 / Fig. 14"},{"comment":"The table reports average CoTs per question for training and test splits, but the test-time generation uses N=128 for the main evaluation and N=16 for some additional generator LLMs. Clarify in the caption which N applies to which figure.","section":"§C / Table 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is substantial and the released artifacts are a strong asset. The main risk is the disconnect between the formal theory (product aggregation, single-sample gPRM) and the experimental protocols (min aggregation, M-sample final-verdict probability). If the authors cannot close this gap, they should reframe the theoretical contribution as a stylized model of stepwise scoring rather than a direct explanation of the reported results. The abstract overclaim is readily fixable but needs attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time: this is the first controlled four-way comparison of dORM, dPRM, gORM, and gPRM across 14 domains, and the headline result—gORM comes out ahead, with PRMs losing their math-domain edge—looks real. The authors release code, data, and checkpoints, and they run sensible sanity checks: shuffled CoTs to probe ORM overfitting, injected label noise, and length-distribution analyses. The empirical pattern holds across four different generator LLMs and two reward-model backbones. That is genuine, useful work.\n\nThe soft spots are in the explanation, not the raw ranking. Theorems 4.2 and 4.3 prove length-error lower bounds for a product-aggregated dPRM and a product-of-step-probabilities gPRM, but the experiments use min aggregation for dPRM and a normalized final-verdict probability for gPRM. The theory does not cover those estimators, and Fig. 12 is correlational. So the claim that stepwise aggregation compounds errors is plausible but not actually supported by the theorems as stated. That is a load-bearing gap because the paper leans on it for the causal story.\n\nTwo smaller issues. The abstract says significant and consistent gains across every tested domain, while the figure caption says without notable degradation in any domain—those are not the same claim, and there are no error bars or significance tests on the multi-domain results. Also, gPRM's multi-domain training set is 94k verification CoTs versus gORM's 172k, so gPRM's failure is confounded with dataset size. A matched-size gPRM baseline would tighten the story. The math-versus-multi-domain comparison also conflates domain and label source (human vs. auto-labeled), though that is more a limitation than a flaw.\n\nBottom line: the empirical contribution deserves peer review. It changes the default recommendation for multi-domain test-time scaling, and the artifacts will be reused. The theory needs either to be realigned with the actual aggregation functions or reframed as a heuristic. I would send it to review with a request for major revision: align theory and experiments, report uncertainty, add a matched-size gPRM control. The core finding will probably survive, but the current framing overreaches.","headline":"Solid empirical comparison with a real finding—gORM beats PRMs across 14 domains—but the theoretical explanation analyzes aggregation schemes the experiments don't use, so the causal story is provisional.","tokens_in":740,"tokens_out":1444,"would_cite":true,"duration_ms":39009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative outcome reward model, not a stepwise process verifier, is the most robust choice for multi-domain test-time scaling.","keywords":["reward models","process reward models","outcome reward models","generative verifiers","test-time scaling","best-of-N","multi-domain reasoning","chain-of-thought verification"],"falsifier":"Take a set of long CoTs with known outcome labels and compare dPRM scores computed by minimum aggregation versus product aggregation; if product aggregation does not reproduce the observed F1 decline with chain length, then the linear-in-length bound in the theory does not explain the empirical dPRM result. Alternatively, train gPRM without consensus filtering and match its training length distribution to the test distribution; the paper's account predicts gPRM would then match gORM.","tokens_in":32368,"feed_emoji":"✅","tokens_out":3928,"duration_ms":32096,"temperature":0.7,"pith_summary":"The paper tries to establish that, for test-time scaling across many domains, fine-grained step-level verification is not the right default. In a unified comparison of four verifier types on 14 domains, a generative outcome reward model—one that writes a verification rationale and a final Yes/No verdict—consistently outperforms discriminative outcome models, discriminative process models, and generative process models. Process reward models lose because they compound per-step errors as reasoning chains lengthen, and because auto-labeled step data carries noise that long chains amplify. If correct, the result overturns the common assumption that finer-grained supervision is always better, and points to outcome-level generative verification for multi-domain deployment.","feed_headline":"Outcome verifiers beat stepwise reward models across 14 domains","feed_subtitle":"Process reward models fail on long, self-correcting chains; the generative outcome verifier wins every domain.","key_machinery":"The comparison rests on four verifier variants—discriminative vs generative, and outcome vs process—and the central mechanism is the difference between scoring the whole chain once (outcome) and aggregating per-step scores (process: minimum aggregation for dPRM in experiments, product aggregation in the theory, and sampling with a final 'Yes' probability for gPRM). Theorems 4.1–4.3 give log-error bounds: ORM error is independent of chain length T, while PRM error lower bounds grow linearly in T. The length-distribution shift from consensus filtering, measured by Wasserstein distance, explains gPRM's extra degradation in the multi-domain setting.","core_discovery":"Across 14 domains of MMLU-Pro, with controlled backbones and four verifier variants, the paper finds the conventional math-domain ranking inverts: discriminative outcome and process models perform on par, a generative process reward model is not competitive, and a generative outcome reward model (gORM) is the most robust, improving best-of-N selection consistently across every domain. The cause is stepwise scoring: per-step errors and label noise compound as reasoning chains lengthen, so PRMs miss correct answers reached after an early mistake. The paper backs this with log-error bounds showing ORM error independent of chain length while PRM error grows at least linearly, and with empirical","pith_inferences":["The same error-compounding argument implies that stepwise reward signals used in RL training will increasingly bias against long, self-correcting trajectories; outcome-level verification may be a safer reward signal for long-horizon tasks.","A testable extension: train gPRM without consensus filtering, matching its training length distribution to the test set; the paper's account predicts gPRM would then recover most of the gap against gORM, isolating length shift from stepwise scoring itself.","If gORM's robustness comes from judging the whole chain at once, a similar design could carry over to open-ended generation once a verifiable outcome is defined, though the paper does not test that setting."],"forward_implications":["For multi-domain deployment, the default verifier should be a generative outcome reward model; it yields consistent Best-of-N gains over dORM, dPRM, and gPRM without notable per-domain degradation.","The PRM-over-ORM advantage is not universal: it appears in math but disappears or reverses across broader domains such as law, engineering, and health.","Long reasoning chains, especially those with self-correction ('aha' moments), are systematically penalized by stepwise aggregators, so verifiers for such chains should avoid per-step scoring.","Multi-domain training benefits generative models more than discriminative ones: per-domain training sharply degrades dORM/dPRM, while gORM/gPRM stay close to their multi-domain performance.","When compute or latency is tight, dORM is a serviceable fallback; when labels are noisy, outcome-level verification is safer than process-level scoring."],"fun_headline_variants":["Outcome verifiers outlast process rewards on all 14 domains","Generative outcome verifier wins test-time scaling across 14 domains","Reward models flipped: outcome beats process in every domain","PRMs fail long chains; gORM claims every domain","Why process rewards lose: error compounding in long reasoning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The theory's error-compounding result assumes that the way step scores are combined in the proof (multiplying them) matches how the experiments combine them (taking the minimum), and that per-step mistakes do not cancel as chains grow; if either fails, the linear-in-length bound would not explain the empirical results.","fun_headline_variants_meta":{"raw":{"variants":["Outcome verifiers outlast process rewards on all 14 domains","Generative outcome verifier wins test-time scaling across 14 domains","Reward models flipped: outcome beats process in every domain","PRMs fail long chains; gORM claims every domain","Why process rewards lose: error compounding in long reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2562,"prompt_tokens":828,"completion_tokens":1734,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1660}},"tokens_in":572,"tokens_out":1734,"duration_ms":11653,"temperature":1.0,"reasoning_tokens":1660,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T13:23:24.640492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of long CoTs with known outcome labels and compare dPRM scores computed by minimum aggregation versus product aggregation; if product aggregation does not reproduce the observed F1 decline with chain length, then the linear-in-length bound in the theory does not explain the empirical dPRM result. Alternatively, train gPRM without consensus filtering and match its training length distribution to the test distribution; the paper's account predicts gPRM would then match gORM.","supporting_citations":[],"review_version":1}