REVIEW 9 cited by
Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Process supervision, using a trained verifier to evaluate the intermediate steps generated by a reasoner, has demonstrated significant improvements in multi-step problem solving. In this paper, to avoid the expensive effort of human annotation on the verifier training data, we introduce Model-induced Process Supervision (MiPS), a novel method for automating data curation. MiPS annotates an intermediate step by sampling completions of this solution through the reasoning model, and obtaining an accuracy defined as the proportion of correct completions. Inaccuracies of the reasoner would cause MiPS underestimating the accuracy of intermediate steps, therefore, we suggest and empirically show that verification focusing on high predicted scores of the verifier shall be preferred over that of low predicted scores, contrary to prior observations on human curated data. Our approach significantly improves the performance of PaLM 2 on math and coding tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with an output supervision trained verifier). Additionally, our study demonstrates that the verifier exhibits strong generalization ability across different reasoning models.
Forward citations
Cited by 9 Pith papers
-
Exploring Expert Failures Improves LLM Agent Tuning
EEF improves LLM agent fine-tuning by extracting beneficial action segments from failed expert trajectories via rollouts from intermediate states, reporting a 62% WebShop win rate and a ScienceWorld score above 81.
-
Evolving Deeper LLM Thinking
Mind Evolution, an LLM-driven evolutionary search guided by a programmatic scorer, solves over 98% of TravelPlanner and Natural Plan instances with Gemini 1.5 Pro, outperforming Best-of-N and sequential revision at co...
-
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
A self-training method scores individual reasoning steps with MCTS rollouts and trains the LLM with a weighted loss, improving accuracy on MATH and GSM8K.
-
Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
A tree-based, inference-only framework combining execution metrics with LLM self-critique improves code generation correctness and efficiency across models and benchmarks.
-
RRO: LLM Agent Optimization Through Rising Reward Trajectories
RRO samples LLM agent actions until a step shows a rising process reward, converts those steps into DPO preference pairs, and reports modest gains on WebShop and InterCode-SQL with fewer samples.
-
Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning
A process reward model trained with Error Propagation and Error Cessation labels from an o1 judge outperforms existing PRMs on math selection and step-level scoring.
-
MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
A process reward model trained on 700k automatically generated step-level labels from only 10k seed math problems improves Best-of-N accuracy on multimodal math benchmarks.
-
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
ReARTeR improves RAG multi-step reasoning by combining a process reward model, an aligned explanation model, temporal-difference lookahead, and MCTS-based iterative preference training.
-
Understanding Chain-of-Thought in LLMs through Information Theory
A chain-of-thought step's information gain, estimated by a fine-tuned supervisor model, can identify the step where an LLM's reasoning first diverges from the correct answer.
Discussion (0). Continue with ORCID to comment.