REVIEW 36 cited by
AlphaMath Almost Zero: Process Supervision without Process
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy, step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.
Forward citations
Cited by 36 Pith papers
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
-
Neuro-Symbolic Data Generation for Math Reasoning
A neuro-symbolic generator that mutates math problems in SMT-LIB form with solver validation and LLM informalization produces training data that improves LLM math reasoning over MetaMath and similar baselines.
-
ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.
-
Output-Space Search: Targeting LLM Generations in a Frozen Encoder-Defined Output Space
Retrieval-grounded reinforcement learning makes an autoregressive LLM hit requested coordinates in a frozen encoder-defined PCA space, giving an outer loop a low-dimensional target to sweep or optimize.
-
Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.
-
Teaching Language Models To Gather Information Proactively
Rewarding questions for eliciting genuinely new information trains a small model to outperform larger models at proactive clarification and downstream writing quality.
-
RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
RoboMonkey shows that test-time sampling with Gaussian perturbation and a VLM-based action verifier improves the success rate of vision-language-action models on manipulation tasks.
-
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.
-
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents
UI-Genie uses a specialized reward model and iterative self-improvement to generate synthetic training trajectories, achieving state-of-the-art results for mobile GUI agents on AndroidControl, AndroidLab, and Android Arena.
-
Accelerating Large Language Model Reasoning via Speculative Search
SpecSearch speeds up tree-search LLM reasoning by drafting thoughts with a small model, rejecting low-quality thoughts with a PRM-based threshold, and correcting them with a large model, achieving up to 2.12x speedup ...
-
An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
Under strict output token budgets, LLM rankings change: smaller instruction-tuned models can beat larger or reasoning-specialized models, and mid-sized models are latency-optimal.
-
PIPA: Preference Alignment as Prior-Informed Statistical Estimation
A unified maximum-likelihood framework with prior constraints that recovers DPO and KTO as special cases and yields new PIPA-M/PIPA-N losses with 3-10% gains on GSM8K and MATH.
-
Iterative Deepening Sampling as Efficient Test-Time Scaling
ID-Sampling inserts self-correction trigger sentences at geometrically increasing token budgets during LLM generation, improving pass rates on MATH-500 and AIME over vanilla Best-of-N and majority voting.
-
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.
-
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
A reward-guided threshold decides whether to accept a draft model's step or call a larger target model, cutting inference cost up to 4.4x while slightly improving accuracy on reasoning benchmarks.
-
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
A self-training method scores individual reasoning steps with MCTS rollouts and trains the LLM with a weighted loss, improving accuracy on MATH and GSM8K.
-
Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
A tree-based, inference-only framework combining execution metrics with LLM self-critique improves code generation correctness and efficiency across models and benchmarks.
-
Progressive Multimodal Reasoning via Active Retrieval
AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.
-
AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models
Wrist-only VLA manipulation with persistent world and ego memory reports state-of-the-art success on LIBERO, RLBench, and real-world long-horizon tasks.
-
CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning
CheMatAgent uses hierarchical Monte Carlo tree search with separate policy and execution models, plus trained reward models, to improve tool selection and parameter filling on a new chemistry benchmark, ChemToolBench.
-
Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agents
A knowledge-graph cognitive prototype plus beam-search self-refinement lets LLM agents simulate students' imperfect programming solutions more accurately than plain prompting.
-
Reward Model Generalization for Compute-Aware Test-Time Reasoning
The paper claims PRM generalization error controls test-time compute efficiency and that the proposed CATS controller improves accuracy on MATH-500 and AIME24.
-
Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning
A process reward model trained with Error Propagation and Error Cessation labels from an o1 judge outperforms existing PRMs on math selection and step-level scoring.
-
On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.
-
Holistically Guided Monte Carlo Tree Search for Intricate Information Seeking
HG-MCTS guides Monte Carlo tree search with an adaptive checklist and self-generated rewards to improve multi-hop information seeking, claiming better benchmark accuracy.
-
Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning
Verifier-guided beam search helps at small sample sizes but falls behind repeated sampling as the sample size increases, and the deficit worsens on difficult and out-of-distribution problems.
-
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
A trained critique model providing step-level feedback improves an LLM actor's mathematical reasoning at test time and during self-improvement, with gains concentrated on hard problems.
-
StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
StreamAdapter compresses a demonstration cache into low-rank parameter updates at test time, matching or beating in-context learning on several benchmarks while keeping generation time constant.
-
CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation
A three-stage training pipeline (synthetic problems, multi-agent chain-of-thought data, step-DPO self-improvement) raises LiveCodeBench pass@1 for Qwen2.5-Coder bases to 21.88 (7B) and 35.08 (32B).
-
Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
A structured survey of reinforcement-learning-based reasoning methods for multimodal large language models, including a taxonomy, reward design review, benchmark tables, and open challenges.
-
Evolution of Thought: Diverse and High-Quality Reasoning via Multi-Objective Optimization
EoT applies multi-objective evolutionary search with crossover, mutation, and clustering to MLLM reasoning and reports improved Pass@K accuracy on MathVista, Math-Vision, and GSM8K.
-
Enhancing LLM Reasoning with Reward-guided Tree Search
A reward-guided tree search framework improves LLaMA-3.1-8B math reasoning accuracy on four benchmarks over CoT and best-of-N baselines.
-
Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering
A survey and position paper that reframes post-training of foundation models as a search, verify, and feedback pipeline called verifier engineering.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
-
Generative AI Act II: Test Time Scaling Drives Cognition Engineering
Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.
-
MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Tree
MC-NEST adds a constant probability term to MCTSr's node selection and reports improved AIME pass@1 for GPT-4o, but the numbers are weakened by test-set rollout tuning and internal inconsistencies.
Discussion (0). Continue with ORCID to comment.