Pith. sign in

REVIEW 10 cited by

Step-level Value Preference Optimization for Mathematical Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.10858 v2 pith:3ISFIZ7Q submitted 2024-06-16 cs.CL cs.AI

Guoxin Chen , Minpeng Liao , Chengxi Li , Kai Fan This is my paper · ORCID
classification cs.CLcs.AI
keywords preferencemodelreasoningoptimizationvaluemathematicalrewardstep-level
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks. Our code is available at \url{https://github.com/MARIO-Math-Reasoning/Super_MARIO}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning

    cs.CL 2025-01 conditional novelty 7.0 of 10

    Step-aligned in-context learning with a first-try retrieval strategy improves LLM mathematical reasoning over problem-level few-shot prompting on multiple benchmarks.

  2. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  3. Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Qwen-LookAgain copies or routes visual tokens back into the context at learned reflection steps, cutting hallucinations and raising visual QA accuracy.

  4. Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms

    cs.LG 2025-02 reject novelty 6.0 of 10

    The authors show that DPO, IPO, DRO, PPO and SAC can be viewed through one loss-construction framework, but their claim that DPO's target distribution differs from the standard DPO target rests on an incomplete argument.

  5. Progressive Multimodal Reasoning via Active Retrieval

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.

  6. CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CheMatAgent uses hierarchical Monte Carlo tree search with separate policy and execution models, plus trained reward models, to improve tool selection and parameter filling on a new chemistry benchmark, ChemToolBench.

  7. Efficient Pretraining Length Scaling

    cs.CL 2025-04 conditional novelty 5.0 of 10

    PHD-Transformer repeats input tokens K times during pretraining with a custom attention mask, then drops the repeated tokens' KV cache, yielding average accuracy gains over a vanilla baseline at 1.2B scale.

  8. Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization

    cs.AI 2024-12 conditional novelty 5.0 of 10

    DAPO trains a step-level value critic and regresses the policy log-ratio to the critic-derived advantage, improving LLM math and code reasoning over the base model on several benchmarks.

  9. DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A reward model that multiplies stepwise correctness and potential scores improves best-of-N verification accuracy for math reasoning.

  10. AI Agent Behavioral Science

    q-bio.NC 2025-06 conditional novelty 4.0 of 10

    AI agents should be studied as behavioral entities shaped by context and interaction, not only as trained models.

Pith tools