REVIEW 10 cited by
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large language models (LLMs) often struggle with maintaining accuracy throughout multiple multiple reasoning steps, especially in mathematical reasoning where an error in earlier steps can propagate to subsequent ones and it ultimately leading to an incorrect answer. To reduce error propagation, guided decoding is employed to direct the LM decoding on a step-by-step basis. We argue that in guided decoding, assessing the potential of an incomplete reasoning path can be more advantageous than simply ensuring per-step correctness, as the former approach leads towards a correct final answer. This transforms the task into a $\textit{value estimation}$ problem in planning. Inspired by the findings that $\textit{outcome supervision for guided decoding essentially acts as a value model}$, we propose Outcome-supervised Value Model (OVM) that employs outcome supervision for training a value model, which prioritizes steps that lead to accurate conclusions. Furthermore, the OVM eliminates the need for labor-intensive annotations of step-level correctness, thereby significantly enhancing its scalability. Our experiments on two multi-step mathematical reasoning datasets, GSM8K and Game of 24, demonstrate the superior performance of the OVM model. Notably, in GSM8K, our $\textbf{OVM-7B model achieves state-of-the-art results among LLMs up to 13B parameters}$; especially it does not utilize GPT-4 or code execution. These findings offer a novel perspective on the role of outcome supervision in training value models for multi-step reasoning tasks and provide theoretical justification for its advantage in value estimation for guided decoding.
Forward citations
Cited by 10 Pith papers
-
SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation
SCOPE compresses sampled math reasoning traces into a prefix tree of AST-normalized code steps, yielding PRM training labels at O(N) cost and improving Best-of-N and ProcessBench scores over automated baselines.
-
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
BMMR provides a 110k-question bilingual, multimodal, college-level dataset across 300 subjects where state-of-the-art models score at most about 50%.
-
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
UI-Genie uses a specialized reward model and iterative self-improvement to generate synthetic training trajectories, achieving state-of-the-art results for mobile GUI agents on AndroidControl, AndroidLab, and Android Arena.
-
ProgRM: Build Better GUI Agents with Progress Rewards
ProgRM, a per-step progress reward model trained with LCS-based self-annotated labels, improves RL-trained GUI agent success rates on WikiHow relative to outcome reward models.
-
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
A reward-guided threshold decides whether to accept a draft model's step or call a larger target model, cutting inference cost up to 4.4x while slightly improving accuracy on reasoning benchmarks.
-
MultiLingPoT: Enhancing Mathematical Reasoning with Multilingual Program Fine-tuning
Training one model on Program-of-Thought solutions in four programming languages raises math accuracy for each language, and answer mixing outperforms single-language augmented training by up to about 6 percentage points.
-
MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision
A process reward model trained on 700k automatically generated step-level labels from only 10k seed math problems improves Best-of-N accuracy on multimodal math benchmarks.
-
GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets
SFT plus RL with post-hoc selection of the best training and inference prompts improves generative text classification accuracy by about 3.5% relative to a naive SFT baseline.
-
Lightweight Latent Verifiers for Efficient Meta-Generation Strategies
LiLaVe, an XGBoost verifier trained on hidden states of a base LLM, predicts answer correctness with AUC comparable to large LLM-based verifiers, and enables conditional majority voting and self-correction that improv...
-
CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis
A small CoT-based synthesizer model trained on candidate-response analysis improves LLM reasoning accuracy, including cases where all sampled candidate answers are incorrect.
Discussion (0). Continue with ORCID to comment.