REVIEW 46 cited by
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to $81.8\%$ (+$5.9\%$), $34.7\%$ (+$5.8\%$), and $76.4\%$ (+$15.8\%$), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO.
Forward citations
Cited by 46 Pith papers
-
Thought-Level Beam Search for Reasoning
Gambit performs thought-level beam search during reasoning-model inference, pruning weak traces and branching from strong prefixes under a fixed memory budget, improving accuracy while cutting token use by up to 68.5%.
-
Theoretical Foundations of $\max$@$k$ Reinforcement Learning
For max@k (best-of-K) finite-horizon MDPs, Markovian policies are suboptimal, a compact (previous-best, current-cumulative) state augmentation restores optimality, exact planning is NP-hard but an FPTAS exists, and th...
-
ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning
ReSum trains LLMs via RLVR to self-summarize reasoning trajectories, yielding 4% average performance gains and 18.6% shorter rollouts through contrastive rollout branches.
-
Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.
-
Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation
In a metastable Markov model of reasoning, search and RL improve hitting times by boosting rare hard edges, and distilling clusters into a meta-chain preserves escape dynamics.
-
ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
APPO: Agentic Procedural Policy Optimization
APPO refines branching and credit assignment in agentic RL via a Branching Score and procedure-level scaling, improving baselines by nearly 4 points on 13 benchmarks.
-
Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
Selectively attacking the visual input with a gradient of response entropy during GRPO sampling increases policy entropy and improves visual reasoning accuracy by roughly 1–2 points.
-
When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.
-
Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm
E2C decouples LLM reasoning into a stochastic planning phase and a deterministic execution phase, achieving similar or better accuracy with far fewer generated tokens.
-
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...
-
Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models
Interactive Reasoning, instantiated as Hippo, lets users view and edit an LLM's chain-of-thought as a tree, and a 16-person study reports improved perceived control, sense-making, and assumption awareness.
-
Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments
Dialogue agents aligned via DPO on preference pairs mined from simulated conversations improve engagement scores against the same simulator, with smaller and partially inconsistent human evaluation evidence.
-
PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.
-
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.
-
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.
-
SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation
Sibling-guided critique and revision of MCTS reasoning traces yields a 30K-sample dataset that matches or beats 590K-sample baselines on the MATH benchmark for 7B models.
-
Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
A new RL penalty, scaled inversely with per-prompt solve rate, trains reasoning models to use few tokens on easy problems and more on hard ones, cutting average inference tokens by roughly 50 percent on math benchmarks.
-
Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidence
Guided by Gut lets an LLM guide its own search using token-level confidence and step novelty, with a RL fine-tuning step to calibrate that confidence, achieving PRM-level accuracy at far lower compute and memory.
-
EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning
EquivPruner detects mathematically equivalent reasoning steps during LLM tree search and keeps only one per group, cutting token use by up to half on GSM8K and MATH-500 without hurting accuracy.
-
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.
-
Adversarial Reasoning at Jailbreaking Time
A loss-guided 'reason, verify, search' loop with three LLM modules surpasses prior semantic jailbreak methods on several defended models.
-
Aligning Instruction Tuning with Pre-training
AITP selects pre-training texts that are underrepresented relative to instruction-tuning data, rewrites them into instruction-response pairs, and improves average SFT performance on three open LLMs.
-
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
A self-training method scores individual reasoning steps with MCTS rollouts and trains the LLM with a weighted loss, improving accuracy on MATH and GSM8K.
-
System-2 Mathematical Reasoning via Enriched Instruction Tuning
Enriched Instruction Tuning (EIT) uses GPT-4 to add planning and missing reasoning steps to human-annotated math solutions, and fine-tuning LLaMA-2 on this data yields 84.1% on GSM8K and 32.5% on MATH.
-
A Systematic Examination of Preference Learning through the Lens of Instruction-Following
Across 48K synthetic instruction prompts, DPO models train best on preference pairs with shared prefixes, high chosen-rejected contrast, and moderate prompt difficulty.
-
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
A single MLLM-based agent, finetuned with cross-domain supervision and online RL, achieves strong zero-shot generalization across manipulation, navigation, games, UI control, and planning.
-
Towards Adaptive Mechanism Activation in Language Agent
ALAMA trains Llama-3-8B to choose among five agent mechanisms per task via self-exploration and KTO-based preference learning, improving accuracy over fixed-mechanism baselines on math and QA tasks.
-
Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Models
Monte Carlo tree search can pick better slot-filling orders for masked diffusion language models, but the claimed advantage over autoregressive models depends on truncating the autoregressive baselines.
-
Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
RL fine-tuning of LLMs improves clean-benchmark accuracy while degrading accuracy under three injected-distractor evaluation scenarios, though one of the three scenarios contradicts the headline claim.
-
Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
ADAPT, a diversity-aware prefix fine-tuning method, improves best-of-N sampling efficiency for a 1.5B reasoning model, reaching 80% accuracy at N=32 versus N=256 for the baseline.
-
Control-R: Towards controllable test-time scaling
Control-R-32B is a fine-tuned 32B model that takes 11 control-field scores in the prompt, claiming to let users trade reasoning depth for efficiency and reaching SOTA on AIME2024 and MATH500 at that scale.
-
Reward Model Generalization for Compute-Aware Test-Time Reasoning
The paper claims PRM generalization error controls test-time compute efficiency and that the proposed CATS controller improves accuracy on MATH-500 and AIME24.
-
Policy Guided Tree Search for Enhanced LLM Reasoning
PGTS trains a graph-transformer policy via PPO to guide tree search over LLM reasoning steps, reporting improved accuracy and lower token use than chain-of-thought and MCTS baselines.
-
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
ReARTeR improves RAG multi-step reasoning by combining a process reward model, an aligned explanation model, temporal-difference lookahead, and MCTS-based iterative preference training.
-
Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search
Using a group of MLLMs to search reasoning trees and training on the resulting paths improves MLLM reasoning, with Mulberry models beating their base models by up to 7.5 points on average.
-
Formal Mathematical Reasoning: A New Frontier in AI
Machine-checkable formal proof should become the backbone of AI mathematics, and a five-task, five-level capability roadmap can measure progress toward that goal.
-
AI Agent Behavioral Science
AI agents should be studied as behavioral entities shaped by context and interaction, not only as trained models.
-
First Finish Search: Efficient Test-Time Scaling in Large Language Models
First Finish Search launches n parallel reasoning traces and returns the shortest one, improving accuracy on AIME benchmarks while cutting token use.
-
The Process of Categorical Clipping at the Core of the Genesis of Concepts in Synthetic Neural Cognition
Words that strongly activate both a lower-layer neuron and its strongly connected upper-layer neuron in GPT-2XL form more semantically similar clusters, which the paper interprets as a clipping process.
-
Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning
Adding a self-correction RL stage before MCTS-DPO improves arithmetic reasoning accuracy by 2.00 to 4.94 points on GSM8K and MATH.
-
Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering
A survey and position paper that reframes post-training of foundation models as a search, verify, and feedback pipeline called verifier engineering.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
-
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
ASPO's adaptive sentence-level loss, by the paper's own definitions, reduces exactly to the standard DPO loss, leaving no difference in the optimization objective.
Discussion (0). Continue with ORCID to comment.