REVIEW 3 major objections 5 minor 14 references
Rethinking Reward Models for Multi-Domain Test-Time Scaling
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A generative outcome reward model, not a stepwise process verifier, is the most robust choice for multi-domain test-time scaling.
desk verdict Solid empirical comparison with a real finding—gORM beats PRMs across 14 domains—but the theoretical explanation analyzes aggregation schemes the experiments don't use, so the causal story is provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison rests on four verifier variants—discriminative vs generative, and outcome vs process—and the central mechanism is the difference between scoring the whole chain once (outcome) and aggregating per-step scores (process: minimum aggregation for dPRM in experiments, product aggregation in the theory, and sampling with a final 'Yes' probability for gPRM). Theorems 4.1–4.3 give log-error bounds: ORM error is independent of chain length T, while PRM error lower bounds grow linearly in T. The length-distribution shift from consensus filtering, measured by Wasserstein distance, explains gPRM's extra degradation in the multi-domain setting.
What would settle it
Take a set of long CoTs with known outcome labels and compare dPRM scores computed by minimum aggregation versus product aggregation; if product aggregation does not reproduce the observed F1 decline with chain length, then the linear-in-length bound in the theory does not explain the empirical dPRM result. Alternatively, train gPRM without consensus filtering and match its training length distribution to the test distribution; the paper's account predicts gPRM would then match gORM.
Extended reading notes
Core claim
Across 14 domains of MMLU-Pro, with controlled backbones and four verifier variants, the paper finds the conventional math-domain ranking inverts: discriminative outcome and process models perform on par, a generative process reward model is not competitive, and a generative outcome reward model (gORM) is the most robust, improving best-of-N selection consistently across every domain. The cause is stepwise scoring: per-step errors and label noise compound as reasoning chains lengthen, so PRMs miss correct answers reached after an early mistake. The paper backs this with log-error bounds showing ORM error independent of chain length while PRM error grows at least linearly, and with empirical
Load-bearing premise
The theory's error-compounding result assumes that the way step scores are combined in the proof (multiplying them) matches how the experiments combine them (taking the minimum), and that per-step mistakes do not cancel as chains grow; if either fails, the linear-in-length bound would not explain the empirical results.
Editorial extensions
If this is right
- For multi-domain deployment, the default verifier should be a generative outcome reward model; it yields consistent Best-of-N gains over dORM, dPRM, and gPRM without notable per-domain degradation.
- The PRM-over-ORM advantage is not universal: it appears in math but disappears or reverses across broader domains such as law, engineering, and health.
- Long reasoning chains, especially those with self-correction ('aha' moments), are systematically penalized by stepwise aggregators, so verifiers for such chains should avoid per-step scoring.
- Multi-domain training benefits generative models more than discriminative ones: per-domain training sharply degrades dORM/dPRM, while gORM/gPRM stay close to their multi-domain performance.
- When compute or latency is tight, dORM is a serviceable fallback; when labels are noisy, outcome-level verification is safer than process-level scoring.
Reading between the lines
- The same error-compounding argument implies that stepwise reward signals used in RL training will increasingly bias against long, self-correcting trajectories; outcome-level verification may be a safer reward signal for long-horizon tasks.
- A testable extension: train gPRM without consensus filtering, matching its training length distribution to the test set; the paper's account predicts gPRM would then recover most of the gap against gORM, isolating length shift from stepwise scoring itself.
- If gORM's robustness comes from judging the whole chain at once, a similar design could carry over to open-ended generation once a verifiable outcome is defined, though the paper does not test that setting.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a unified comparison of four reward-model variants for best-of-N test-time scaling: discriminative outcome (dORM), discriminative process (dPRM), generative outcome (gORM), and generative process (gPRM) reward models. The evaluation spans a math domain (PRM800K/ProcessBench) and a 14-domain multi-domain setting (MMLU-Pro), with multiple generator LLMs and shared reward backbones. The authors report that, consistent with prior work, dPRM beats dORM and generative variants beat discriminative ones in math; in the multi-domain setting, dORM is on par with dPRM, gPRM is not competitive, and gORM is overall the most robust. They propose two explanations: (i) stepwise scoring/aggregation compounds errors as reasoning chain length grows, supported by log-error lower bounds (Theorems 4.1–4.3); and (ii) label noise and consensus-filtering-induced length distribution shift hurt gPRM. Code, data, and checkpoints are released.
Significance. If the empirical ranking holds, the paper challenges the common assumption that fine-grained process supervision always improves verifier performance in multi-domain test-time scaling, and it provides a practical comparison that is more controlled than much prior work (shared backbones, four variants, multiple generator LLMs). The release of code, datasets, and checkpoints is a concrete strength that will facilitate follow-up work. The theoretical analysis, however, is not tightly connected to the experimental protocols: the dPRM theorem analyzes product aggregation while the experiments use minimum aggregation, and the gPRM theorem concerns single-sample product scores while the deployed estimator is an M-sample average of final-verdict probabilities. The headline abstract claim of consistent significant gains across every domain is also stronger than the per-domain results support. The empirical study itself is valuable and likely correct in its main qualitative trends, but the explanatory narrative needs substantial revision.
major comments (3)
- [§4.1 / Theorem 4.2 vs. §2.2, Appendix A.1] Theorem 4.2 analyzes a product-aggregated dPRM: Appendix A.1 defines f_hat_dPRM(x) := ∏_{t=1}^T û_t(x), and the lower bound E[Δ²_dPRM] ≥ (σ²−2γ)T is derived for this product. In contrast, §2.2 states that at test time dPRM 'aggregat[es] the step rewards with the minimum' (following Zeng et al., 2025). Min aggregation has different length behavior: it is dominated by the worst step rather than by the accumulation of all step errors, so the theorem does not bound the error of the estimator actually evaluated in Figures 3–7 and 12. The empirical support in Fig. 12 (Left) uses min-aggregated dPRM and therefore cannot rescue the mismatch. This is load-bearing: the paper's central explanation for why PRMs underperform in multi-domain TTS is precisely that 'stepwise aggregation compounds errors as reasoning length increases,' and that claim is not established for the dPRM variant used in the ex
- [§4.1 / Theorem 4.3 vs. Eqs. 8–9, Appendix A.1] Theorem 4.3 bounds the log-error of Δ_gPRM defined in Appendix A.1 as the log of a single-sample product of step-correctness probabilities, ∏_t F_t(x, v_{≤t}). The test-time estimator in Eq. 8 averages M samples of the normalized final-verdict probability p_gPRM(y=1 | x, v_{1:L+}), not the log of a single product. While p_gPRM(y=1|x,v) factors as a product when the final 'Yes' is deterministically equivalent to all steps being correct, the paper does not supply a rigorous bridge from the single-sample lower bound to the M-sample average used with M=10. Theorem A.4 addresses a mean predictor (E[e^L]) rather than the finite-M sample average, and the main text's pointer to 'cf. Eq. 5' is incorrect — Eq. 5 is the gORM estimator, not the gPRM estimator of Eq. 8. The theoretical support for gPRM's empirical degradation is therefore incomplete, and the stated linear-in-T lower bound is not dire
- [Abstract / §3.2 / Fig. 6] The abstract claims that gORM yields 'significant and consistent gains across every tested domain.' This is not supported by the paper's own per-domain results in Fig. 6. In Psychology, dORM (81.9) and dPRM (81.7) both exceed gORM (81.3); in Health, dPRM (79.2) exceeds gORM (78.2). The weaker statement in §3.2 that gORM outperforms the others 'without notable degradation in any domain relative to the others' is accurate, but the abstract overstates the finding. Furthermore, §D reports a single run for the multi-domain setting, so the word 'significant' is not backed by error bars or statistical tests. The manuscript should either soften the abstract to match the per-domain results or provide significance testing that justifies the stronger claim.
minor comments (5)
- [§2.2] Typo: 'Byeond Khalifa et al. (2025)' should be 'Beyond Khalifa et al. (2025)'.
- [Main text, after Theorem 4.3] The sentence referencing 'Theorem. A.4' for gPRM with Monte Carlo estimation cites 'cf. Eq. 5'; this should be Eq. 8, since Eq. 5 defines the gORM estimator.
- [§4.1 / Fig. 12] The correlation values in Fig. 12 (Right) are reported without confidence intervals or p-values. Given only 8 binned length categories (or 6 on the right), these correlations should be interpreted cautiously; adding a scatter plot with per-bin standard errors would strengthen the claim.
- [§4.2 / Fig. 14] The Wasserstein distance histograms in Fig. 14 aggregate over all domains; per-domain distances in Table 3 show large variation (e.g., Chemistry 6.922 vs. History 0.581 for gPRM). A per-domain scatter with domain labels would make the 'length shift hurts gPRM' mechanism more transparent than the single pooled number.
- [§C / Table 1] The table reports average CoTs per question for training and test splits, but the test-time generation uses N=128 for the main evaluation and N=16 for some additional generator LLMs. Clarify in the caption which N applies to which figure.
Circularity Check
No significant circularity; the empirical ranking is externally evaluated and the theoretical bounds are conditional on stated assumptions rather than fitted to the outcome.
full rationale
The central empirical claim (gORM outperforms dORM/dPRM/gPRM across MMLU-Pro domains) is an externally measured ranking of four independently trained verifier variants on held-out benchmarks; nothing in the training objectives, aggregation rules, or evaluation protocol forces this ordering by construction. The theoretical results (Theorems 4.1-4.3, A.1-A.4) are conditional variance-of-sum bounds: under per-step variance floors and weak anti-correlation, the log-error of a product-aggregated PRM grows linearly in T. These are theorem statements with explicit assumptions, not fitted parameters renamed as predictions, and the paper does not claim to have estimated sigma^2 or gamma from the benchmark results. The Appendix's use of product aggregation while experiments use min aggregation for dPRM and final-verdict probability for gPRM is an internal-validity mismatch, but it is not a reduction of the conclusion to its inputs. No load-bearing self-citation chain exists: prior work (Zeng et al., Khalifa et al., Zhang et al.) is cited as external methodology, and the authors invoke no uniqueness theorem or prior result of their own to force the choice of gORM. Hence no circular step meets the evidentiary bar; the score is 0.
Assumptions & free parameters
free parameters (6)
- M (number of verification CoTs sampled) =
16 (math), 10 (multi-domain)
- Sampling temperature τ for verification CoT generation =
0.6
- Consensus-filtering thresholds =
token limits 4096/8192; discard unparsable, Chinese, label-inconsistent CoTs
- LoRA rank and alpha =
r=16/32, α=32/16
- Decision threshold for F1 =
0.5
- Theoretical constants σ², τ², γ, κ =
unspecified
assumptions (6)
- domain assumption A correct final step implies all previous steps are correct (y=z_T=1 ⇒ z_1:T=1).
- ad hoc to paper Variance floors and weak anti-correlation: Var(ξ_t|x) ≥ σ², covariance sums ≥ −γT, and σ² > 2γ.
- ad hoc to paper ORM conditional variance is bounded independent of T: Var(ξ̄|x) ≤ τ²_orm.
- ad hoc to paper Product aggregation of step scores behaves like min aggregation for dPRM.
- domain assumption Llama-3.1-70B auto-generated process labels on MMLU-Pro are sufficiently reliable for training dPRM/dORM.
- domain assumption MMLU-Pro multiple-choice answers are valid verifiable outcome labels.
Cite this review
Pith. "Pith review of Rethinking Reward Models for Multi-Domain Test-Time Scaling." pith.science (2026). https://pith.science/paper/MWPBOOML
@misc{pith2026251000492,
author = {Pith},
title = {Pith review of: Rethinking Reward Models for Multi-Domain Test-Time Scaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/MWPBOOML}},
note = {Machine review of arXiv:2510.00492}
}
read the original abstract
The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic. Prior work has studied both outcome reward models (ORMs), which assess only the final answer, and process reward models (PRMs), which score intermediate reasoning steps. Although PRMs are often viewed as advantageous due to their finer-grained supervision, much of the supporting evidence comes from math-adjacent settings, and their relative benefits across broader domains remain unclear. We present the first unified evaluation of four reward model variants, discriminative ORM and PRM (dORM, dPRM) and generative ORM and PRM (gORM, gPRM), across 14 diverse domains. Contrary to conventional wisdom, we find that (i) dORM performs on par with dPRM, (ii) gPRM is not competitive, and (iii) overall, gORM is the most robust, yielding significant and consistent gains across every tested domain. We attribute the worse performance of gPRM to the stepwise scoring process, which inherits label noise from LLM-based automatic labeling, leading to difficulties in evaluating long reasoning trajectories, including those involving self-correcting reasoning. Both our theoretical analysis and empirical observations indicate that stepwise aggregation compounds errors as reasoning length increases. These findings challenge the common assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. Our \href{https://github.com/db-Lee/Multi-RM}{\underline{code}} is publicly available to facilitate future research in multi-domain settings.
Figures
Figures from the paper (36 more)
Reference graph
Works this paper leans on
-
[1]
(Variance floors)Var(ξ t |x)≥σ 2,Var(ξ (g) t |x)≥σ 2 +τ 2
-
[2]
(Weak anti-correlation) P 1≤s<t≤T Cov(ξs,ξt |x)≥ −γT, P 1≤s<t≤T Cov(ξ(g) s ,ξ(g) t |x)≥ −γT
-
[3]
ForgPRMwithsampledverification CoTs, sampling contributes per-step noise:Var(ξ (g) t |x)≥ σ2 +τ 2 for someτ 2 >0
(Positive slope)σ 2 >2γ. ForgPRMwithsampledverification CoTs, sampling contributes per-step noise:Var(ξ (g) t |x)≥ σ2 +τ 2 for someτ 2 >0. For ORMs, assumeVar( ¯ξ|x)≤τ 2 orm <∞(noT-dependence). Theorem A.1(Log-error lower bound ofdPRM).Let∆ dPRM := log ˆfdPRM(x)−ζ(x). Under the assumptions above, E ∆2 dPRM ≥(σ 2 −2γ)T. Theorem A.2(Log-error bound ofdORMor...
-
[4]
Finally, Jensen’s inequality gives(max{0,E[∆ mean]})2 ≤E[∆ 2 mean], so the MSE bound follows
Expectations and MSE.Taking expectations overXand applying Cauchy–Schwarz to E[B(g)(X)]yields E[∆mean]≥ κ 2 (σ2 +τ 2 −2γ)T+ (σ 2 A −2γ A) − q E[B(g)(X) 2]. Finally, Jensen’s inequality gives(max{0,E[∆ mean]})2 ≤E[∆ 2 mean], so the MSE bound follows. In the calibrated caseB (g) ≡0, the stated simplified bounds hold. B PROMPTS In this section, we present pr...
2021
-
[5]
URLhttps://aclanthology.org/2025
doi: 10.18653/v1/2025.findings-acl.547. URLhttps://aclanthology.org/2025. findings-acl.547/. Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. GenPRM: Scaling test-time compute of process reward models via generative reasoning.arXiv preprint arXiv:2504.00891, 2025. Chujie Zheng, Zhenru Z...
arXiv 2025
-
[6]
SinceK X (0) = 0andK ′ X (0) =E[L|X], we obtain logµ(X) =E[L|X] + Z 1 0 (1−θ) Var θ(L|X)dθ
Jensen-gap identity.Taylor with integral remainder atθ= 0gives KX (1) =K X (0) +K ′ X (0) + Z 1 0 (1−θ)K ′′ X (θ)dθ. SinceK X (0) = 0andK ′ X (0) =E[L|X], we obtain logµ(X) =E[L|X] + Z 1 0 (1−θ) Var θ(L|X)dθ. By definition of the mean predictor, ∆mean(X) = logµ(X)−ζ A(X),whereµ(X) =E[e L |X]. Plugginglogµ(X) = ∆ mean(X) +ζ A(X)withB (g)(X) :=E[L|X]−ζ A(X)...
-
[7]
Moreover, sinceL=ζ A + ∆g-prm =ζ A +B (g) +N (g) withE[N (g) |X] = 0, and sinceζ A and B(g)(X)are constants when conditioning onX, we have Var(L|X) = Var(N (g) |X)
Lower bound onδ J and variance linkage.By tilt-stability, δJ (X)≥ κ 2 Var(L|X). Moreover, sinceL=ζ A + ∆g-prm =ζ A +B (g) +N (g) withE[N (g) |X] = 0, and sinceζ A and B(g)(X)are constants when conditioning onX, we have Var(L|X) = Var(N (g) |X). Expanding and using the variance floors and weak anti-correlation conditions (as in Theorem A.3), Var(N(g) |X)≥σ...
-
[8]
•Verifiable: The step can be verified using common knowledge, simple calculations, or a quick reference (e.g., recalling a basic theorem)
GOOD StepA step is classified as GOOD if it meets all of these criteria: •Correct: Everything stated is accurate and aligns with known principles or the given problem. •Verifiable: The step can be verified using common knowledge, simple calculations, or a quick reference (e.g., recalling a basic theorem). If verifying requires extensive effort (e.g., deta...
Show all 14 references
-
[9]
Good job!
OK StepA step is classified as OK if it is: •Correct and Verifiable: Contains no errors and can be verified. •Unnecessary or Redundant: Adds little value, such as restating prior information or provid- ing basic encouragement (e.g., “Good job!”). •Partially Progressing: Makes ...
-
[10]
•Is Hard to Verify: Requires significant effort to confirm due to poor explanation
BAD StepA step is classified as BAD if it: •Is Incorrect: Contains factual errors, misapplies concepts, derives an incorrect result, or con- tradicts the ground truth answer. •Is Hard to Verify: Requires significant effort to confirm due to poor explanation. •Is Off-Topic: Inc...
-
[11]
A Multiple Choice Question
-
[12]
A Ground Truth Answer
-
[13]
Verification: Is the answer correct (Yes/No)? X
A Student’s Step-by-Step Solution, where each step is enclosed with tags and indexed from 0. Once you identify a BAD step, return the index of the earliest BAD step. Otherwise, return the index of -1 (which denotes all steps are GOOD or OK). Please put your final answer (i.e.,...
2025
-
[2024]
Verification: Is the answer correct (Yes/No)? X
for training, where the process labelsz 1:T are human-annotated. For training ORMs, we set the outcome labely=1 z1:T =1 T (rather thany=1(ˆa(r T ) =a)), since PRM800K provides high-quality ground-truth process labels. As a testbed, we useProcessBench(Zheng 22 Preprint et al., ...
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.