{"id":"7a028a19-8529-4655-bd78-0a25dd65631b","arxiv_id":"2502.06737","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"VersaPRM, fine-tuned on LLM-generated and auto-labeled multi-domain chain-of-thought data, beats math-only process reward models and majority voting on non-math MMLU-Pro categories.","lead":"VersaPRM is a process reward model trained on synthetic step-by-step reasoning data from multiple domains, and it improves answer selection accuracy beyond math, for example by about 8 points in law questions. The paper matters because it shows that test-time verification with reward models can be extended from math to law, philosophy, biology, and other fields.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decisive unknown is whether VersaPRM's step labels encode process-level correctness or merely final-answer consistency; no same-data ORM baseline is reported, so the 'process' part of the central claim is not yet isolated.","rationale":"The reader's weakest assumption identifies auto-labeling quality, and my reading converges on the same point with a sharper operationalization: the labeler's access to the ground-truth answer makes the labels ambiguous between process correctness and final-answer consistency. This is the most load-bearing assumption because every downstream claim about cross-domain process supervision inherits it. The paper's own ablation in Section 5.3 shows that removing the ground-truth answer from the labeling prompt drastically degrades both label quality and downstream performance, which is consistent with the labeler leaning on the answer rather than on step-level reasoning. The absence of any ORM baseline on the same training data is the key gap: WMV and BoN can be powered by a scalar correctness score, so the reported gains do not yet prove that step-level labels matter. I credit the paper's open artifacts, the hold-out domain ablation, and the generator-swap experiments, which partially address distributional leakage, but these do not resolve the process-versus-outcome ambiguity. My proposed test is inexpensive: train an ORM on the same CoTs and compare reranking gains. If the ORM matches VersaPRM, the central claim should be downgraded to an outcome-reranking result; if not, the process claim is substantially strengthened. Until that test is run, CONDITIONAL remains the right verdict, so I recommend no change to the reader's decision.","tokens_in":31907,"tokens_out":5298,"duration_ms":53982,"concrete_test":"Train an outcome reward model on the same MMLU-Pro-CoT-Train (Labeled) data, using the final-answer match to ground truth as the label for each CoT, and evaluate it with WMV and BoN under the same protocol as Table 4. If the ORM's WMV/BoN gains match or exceed VersaPRM's across the non-math categories, then the step-level labels are not the operative signal and the process-reward claim fails. As a complementary check, have domain experts label the first erroneous step on a few hundred held-out CoTs and measure whether VersaPRM's step scores localize that step better than an answer-only baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a PRM can transfer beyond math by scoring reasoning steps across domains. This requires the synthetic step labels to carry genuine process-level signal, not just information about whether the final answer matches the ground truth. The auto-labeler is given the ground-truth answer and instructed to find the first BAD step, so for a CoT with a wrong final answer it will typically locate some earlier step to blame; for a correct CoT it labels all steps as 1. Consequently, the training signal may largely reduce to 'this CoT reaches the correct answer or not', making VersaPRM a learned outcome filter with step-level tokenization. The manual validation on 64 questions (Section 5.2) reports 80% agreement on correct CoTs and 71% on incorrect CoTs, but both numbers are compatible with an answer-matching labeler. The paper does not compare VersaPRM against an ORM trained on the same data, so the observed WMV/BoN gains could be achievable by weighting solutions with final-answer correctness alone. If that is true, the process-reward contribution is not demonstrated, and the headline claim reduces to 'a reranker trained on MMLU-Pro CoTs improves MMLU-Pro reranking', which is a weaker and less novel statement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether process reward models (PRMs) trained on mathematical reasoning generalize to non-mathematical domains. It first shows that open-source math PRMs provide only marginal gains over majority voting in domains such as Law, Philosophy, and Biology. It then introduces VersaPRM, a PRM initialized from a Llama-based math PRM (LlamaPRM800K) and fine-tuned with LoRA on MMLU-Pro-CoT-Train (Labeled), a new synthetic dataset of roughly 84k chain-of-thought solutions auto-labeled by Llama-3.1-70B with step-level GOOD/OK/BAD judgments. VersaPRM is evaluated using weighted majority voting, best-of-N, beam search, and MCTS on a 14-domain MMLU-Pro subset, with consistent gains over majority voting and over math PRMs, particularly in non-math domains. The paper also ablates training-data diversity, holdout domains, base-model choice, aggregation rules, generator size, and augmentations, and releases the data, code, and model checkpoints.","tokens_in":32145,"tokens_out":5302,"duration_ms":44924,"significance":"If the results hold, the paper would provide evidence that step-level reward models can be trained cheaply for diverse domains and that domain diversity is an important driver of PRM generalization. The open-sourced dataset, code, and model are a strong community contribution, and several ablations are well designed: the holdout-domain experiment (Section 6.2, Figure 5) and the random-versus-math-subset experiment (Figure 4) directly address alternative explanations based on in-distribution format learning. However, the central 'process' interpretation is not yet isolated from outcome-based reranking because the auto-labeler is given the ground-truth answer, and the small per-domain evaluation samples limit the strength of the headline claims.","major_comments":[{"comment":"The paper does not compare VersaPRM against an outcome reward model (ORM) trained on the same synthetic data. Because the auto-labeler is given the ground-truth answer and labels every step before the first detected error as 1, the training signal is consistent with a CoT-level outcome filter: a CoT whose final answer matches the ground truth receives all-1 labels, and a CoT with a wrong final answer almost always receives a −1 at some earlier step. Without a same-data ORM baseline, or a PRM trained on labels produced without the ground-truth answer while holding the step-definition prompt fixed, the observed WMV/BoN gains do not demonstrate that the step-level process signal, rather than final-answer matching, drives the improvement. The ablated-prompt experiment in Appendix B.4 changes two variables simultaneously (the ground-truth answer and the step-definition instruction), so it does not isolate this factor.","section":"Section 5.2 and Section 6.2"},{"comment":"Per-domain accuracy is measured on approximately 145–150 questions, but the paper reports no confidence intervals or significance tests for the headline differences. For example, the Law gain of +7.93 points corresponds to roughly 11.5 questions out of about 145, and with a baseline accuracy of 35.93%, the standard error is approximately 4 points; this gain is borderline by conventional significance levels. The authors should report bootstrap confidence intervals or paired McNemar tests for the WMV-versus-MV and VersaPRM-versus-math-PRM comparisons.","section":"Table 4 and Section 4.2"},{"comment":"The manual validation of the auto-labels is based on only 64 questions, with 80% agreement for CoTs labeled correct and 71% agreement for CoTs labeled incorrect, and the paper estimates that 'approximately 75% of the CoTs in the entire dataset are correctly labeled' by averaging these two rates without accounting for class proportions or uncertainty in the point estimates. More importantly, the validation protocol checks agreement with human judgments but does not test whether the labeler identifies process-level errors independently of final-answer consistency; a labeler that simply rewards final-answer matching could plausibly achieve similar agreement numbers. The authors should provide a validation that separates process errors from outcome errors, for example by reporting the proportion of cases where the first BAD step occurs in a CoT whose final answer is correct.","section":"Section 5.2"}],"minor_comments":[{"comment":"The abstract states a '7.9% performance gain' and Section 1 repeats '7.9%', but Table 4 reports +7.93 absolute percentage points over the majority-voting baseline; please use 'percentage points' to avoid implying a relative improvement.","section":"Abstract and Table 4"},{"comment":"The sentence 'Based on those findings, our final, our final multi-domain PRM' contains a duplicated phrase and should be edited.","section":"Section 6.1"},{"comment":"The text refers to 'MVW' in the comparison of VersaPRM configurations; this should be 'WMV' (weighted majority voting).","section":"Figure 6 caption and Section 6.3"},{"comment":"Please clarify whether the 150 questions per domain were sampled with or without replacement and how duplicate questions were handled across domains when 41 questions were discarded.","section":"Section 4.2"},{"comment":"Step labels are described as 1 and −1 in Section 5.2, while Appendix D.2 describes training with '+' and '−' classification tokens; the notation should be unified.","section":"Section 5.2 and Appendix D.2"},{"comment":"The Pass@1 comparison of iterative refinement discards responses that timed out or failed parsing in the refined set, but it is unclear whether the same filtering is applied to the original CoT baseline; please state the filtering procedure consistently for both conditions.","section":"Appendix G.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a top-tier ML venue and the open-sourced artifacts are valuable. The main risk is that the 'process' interpretation may be an artifact of outcome leakage through the ground-truth answer in the labeling prompt; the same-data ORM baseline and per-domain confidence intervals are needed before the central claim can be accepted. The revision should also temper the title-level emphasis on 'process' if the ORM control is not added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: VersaPRM is a real, reproducible empirical result. Math PRMs do not transfer to law, biology, or philosophy, and fine-tuning on synthetic multi-domain CoT data with auto-labels gives consistent WMV and BoN gains across MMLU-Pro categories. The paper is honest, open-sources everything, and runs the right ablations.\n\nWhat is actually new is the labeled multi-domain CoT dataset and the finding that domain diversity, not data volume, drives generalization. The random-subset-versus-math-subset ablation is the strongest piece of evidence: equal-size training data, but the diverse set wins in both math and non-math. The hold-out domain experiments (law, biology, philosophy, psychology, CS) are also convincing. I credit the open data, code, and model checkpoints as real contributions.\n\nThe paper does several things well. It compares aggregation rules, base models, LoRA versus full fine-tuning, generator sizes (8B and 70B), and search-based methods. The manual error analysis in Appendix F shows VersaPRM sometimes catches answer-choice mismatches that math PRMs miss, which is genuine step-level behavior. The ablated auto-labeling prompt without ground truth is a good control: label quality drops sharply, and the trained PRM loses most of its lift. That control tells you the ground-truth answer is doing critical work in the labeler, which cuts both ways.\n\nNow the soft spots. Per-domain evaluation is roughly 150 questions, with no confidence intervals on the headline numbers. Training and evaluation both come from MMLU-Pro, and both use Llama-3.1 generators, so the distribution overlap is real. The auto-label quality check on 64 questions (80% agreement on correct CoTs, 71% on incorrect) is thin. Most importantly, there is no ORM baseline trained on the same data. Given that correct CoTs are labeled all-1 and incorrect CoTs get an early -1, the PRM could be functioning largely as a learned outcome filter. I think that concern is partially right but overstated: the MCTS and beam-search gains depend on intermediate step scores, and the manual examples show step-level discrimination beyond final-answer matching. Still, the paper would be much stronger with an ORM trained on the same synthetic data.\n\nWho gets value from this: people working on PRMs, test-time compute, or synthetic data labeling. It deserves a serious referee, and the referee should ask for an ORM baseline, confidence intervals, and ideally an evaluation on a different benchmark or generator family to break the overlap. I would engage with it and probably cite it.","headline":"A solid empirical demonstration that multi-domain synthetic CoT tuning makes PRMs useful beyond math, but the process-vs-outcome question is under-tested by one missing ORM baseline.","tokens_in":32721,"tokens_out":2303,"would_cite":true,"duration_ms":23753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A step-scoring model trained on synthetic multi-domain data transfers beyond math, with weighted majority voting gaining 7.9 accuracy points in law and 4.6 in biology.","keywords":["process reward models","multi-domain reasoning","synthetic data generation","auto-labeling","test-time compute","chain-of-thought","MMLU-Pro","domain generalization"],"falsifier":"Take a fixed set of MMLU-Pro questions and generate 16 chains per question, then in each chain rewrite exactly one middle step to be factually wrong while keeping the final answer and all other steps unchanged; if VersaPRM's score for that step does not drop clearly below the scores of the unmodified steps in the majority of cases, the model is encoding answer consistency rather than process correctness.","tokens_in":31728,"feed_emoji":"🧠","tokens_out":7099,"duration_ms":60059,"temperature":0.7,"pith_summary":"This paper tries to establish that process reward models—scorers that grade each step of a chain-of-thought—can work outside mathematics, contrary to the current practice of training them only on math. It first documents that existing math PRMs give near-zero reranking gains in law, biology, philosophy, and history. It then claims that fine-tuning a math PRM on a synthetically generated, auto-labeled multi-domain dataset, MMLU-Pro-CoT-Train (Labeled), yields a model, VersaPRM, that improves weighted majority voting and best-of-N selection across all fourteen MMLU-Pro domains. The headline evidence is that VersaPRM's weighted-majority-voting gain over majority voting is +7.93 accuracy points in law and +4.62 in biology, versus at most +1.31 and +0.31 for math-only PRMs. If correct, this shows that step-level correctness is a learnable, domain-general signal rather than a math-specific artifact.","feed_headline":"Multi-domain step scorer lifts law accuracy by 7.9 points","feed_subtitle":"Math-trained step scorers stall in law and biology; multi-domain synthetic fine-tuning unlocks the gains.","key_machinery":"The load-bearing mechanism is the auto-labeling pipeline that turns unlabeled chains of thought into step-level reward targets. A 70B Llama judge is given the question, the ground-truth answer, and a three-level rubric (GOOD/OK/BAD) and asked to find the first incorrect step; the labels are converted to a 1/−1 sequence, and training uses a classification token appended to each step with cross-entropy loss on top of an 8B base model. This pipeline matters because it is what lets the PRM learn 'which step is wrong' in law and biology, not just 'which answer is right'; the paper's own ablation shows that removing the ground-truth answer or the rubric definitions collapses label quality and most of the downstream gain.","core_discovery":"The central claim is that a process reward model can be made domain-general by continuing the training of a math PRM (LlamaPRM800K) on synthetic step-level supervision drawn from 5,750 MMLU-Pro questions across fourteen domains. Chains of thought are generated by Llama-3.1-8B-Instruct and labeled by Llama-3.1-70B-Instruct, which receives the ground-truth answer and a GOOD/OK/BAD rubric and returns the index of the first BAD step; all steps before it become 1, the bad step becomes −1, and later steps are discarded. The resulting VersaPRM beats open-source math PRMs and the majority-voting baseline under weighted majority voting, best-of-N, beam search, and MCTS, and hold-out-domain experiments show the gain does not depend on having seen that domain's questions during training. The paper also reports modest gains when reranking DeepSeek-R1's own chains in law and philosophy.","pith_inferences":["The same generation-and-label recipe should transfer to open-ended generation beyond MMLU-Pro; the paper's preliminary law experiment is consistent with this, and a cleaner test would push free-form biology or history answers through VersaPRM reranking.","Because the ground-truth answer is essential to label quality, the learned signal may be partially answer-conditional; a variant trained with the answer removed from the labeler prompt would show how much of the gain is genuine step-level detection.","The roughly 75% label-agreement rate implies substantial noise in the training set; modeling that noise explicitly, rather than self-filtering on the model's own scores, might sharpen the process signal further.","The random-subset result hints at positive transfer from diverse domains into math; testing VersaPRM on harder held-out math benchmarks would reveal whether multi-domain training improves math verification or only MMLU-Pro-style questions."],"forward_implications":["Weighted majority voting and best-of-N with VersaPRM improve accuracy in non-math domains where math-trained PRMs are flat, so test-time compute can be spent productively beyond math.","Domain diversity, not merely question-format familiarity, drives the gain: a random-subset model beats a math-subset model of the same size, and held-out domains still benefit.","The gains persist across inference methods (WMV, BoN, beam search, and MCTS) and across generators of different sizes, and even DeepSeek-R1's chains are reranked better than majority voting.","The 84,098 labeled chains were produced at a total cost under $100, so the generation-and-labeling pipeline is cheap enough to reproduce at scale.","Prior exposure to math data helps the base model leverage multi-domain fine-tuning, while full fine-tuning can overfit; this points to initialization and regularization as important design choices."],"supporting_citations":[{"why":"Defines process reward models and the step-level verification paradigm, and supplies the PRM800K math dataset used to initialize the base PRM.","marker":"(Lightman et al., 2024)"},{"why":"Supplies the MMLU-Pro questions used both for evaluation and for sampling the training questions.","marker":"(Wang et al., 2024c)"},{"why":"Provides the Llama-3.1 generator, labeler, and base model used throughout the pipeline.","marker":"(Dubey et al., 2024)"},{"why":"Inspires the critic-based auto-labeling prompt and provides the Qwen-2.5-Math-PRM baseline that VersaPRM is benchmarked against.","marker":"(Zheng et al., 2024)"},{"why":"Provides the Math-Shepherd baseline and the rollout-style synthetic labeling method the paper contrasts with its own.","marker":"(Wang et al., 2024b)"},{"why":"Supplies the weighted majority voting procedure and the process-versus-outcome-feedback framing used in the inference methods.","marker":"(Uesato et al., 2022)"},{"why":"Supplies the test-time compute scaling argument and the beam-search inference algorithm evaluated in Section 6.3.","marker":"(Snell et al., 2024)"},{"why":"Provides DeepSeek-R1, the strong reasoning model whose generated chains VersaPRM reranks in the Section 6.4 experiment.","marker":"(Guo et al., 2025)"},{"why":"Provides the LoRA fine-tuning method used for the final VersaPRM model.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["VersaPRM: Process reward model that excels beyond math","Synthetic step labels make PRMs work for law and biology","Math PRMs fail outside math; VersaPRM fixes that","Open-source PRM gains 7.9 points in law via multi-domain training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that Llama-3.1-70B, when given the correct answer, marks the first genuinely wrong reasoning step accurately enough that roughly three-quarters of the training labels are correct; if those labels mostly reward whether the chain ends at the right answer, VersaPRM would be little more than an answer-consistency reranker.","fun_headline_variants_meta":{"raw":{"variants":["VersaPRM: Process reward model that excels beyond math","Synthetic step labels make PRMs work for law and biology","Math PRMs fail outside math; VersaPRM fixes that","Open-source PRM gains 7.9 points in law via multi-domain training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1600,"prompt_tokens":911,"completion_tokens":689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":613}},"tokens_in":527,"tokens_out":689,"duration_ms":7048,"temperature":1.0,"reasoning_tokens":613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:28:09.523489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of MMLU-Pro questions and generate 16 chains per question, then in each chain rewrite exactly one middle step to be factually wrong while keeping the final answer and all other steps unchanged; if VersaPRM's score for that step does not drop clearly below the scores of the unmodified steps in the majority of cases, the model is encoding answer consistency rather than process correctness.","supporting_citations":[],"review_version":1}