Pith. sign in

REVIEW 30 cited by

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.14283 v4 pith:NAUDZCYG submitted 2024-06-20 cs.AI

Chaojie Wang , Yanchen Deng , Zhiyi Lyu , Liang Zeng , Jujie He , Shuicheng Yan , Bo An This is my paper · ORCID
classification cs.AI
keywords llmsreasoningmulti-stepdeliberativeheuristicimprovinglanguageperformance
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. However, the auto-regressive generation process makes LLMs prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. In this paper, by casting multi-step reasoning of LLMs as a heuristic search problem, we aim to alleviate the pathology by introducing Q*, a general, versatile and agile framework for guiding LLMs decoding process with deliberative planning. By learning a plug-and-play Q-value model as heuristic function for estimating expected future rewards, our Q* can effectively guide LLMs to select the most promising next reasoning step without fine-tuning LLMs for the current task, which avoids the significant computational overhead and potential risk of performance degeneration on other tasks. Extensive experiments on GSM8K, MATH and MBPP demonstrate the superiority of our method, contributing to improving the reasoning performance of existing open-source LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 30 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making

    cs.CL 2025-06 conditional novelty 6.0 of 10

    UProp estimates extrinsic uncertainty in multi-step LLM decision-making by approximating pointwise mutual information over trajectory samples, improving AUROC for identifying incorrect agent outputs by 2.3 to 11% over...

  2. Reasoning LLMs are Wandering Solution Explorers

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Six current reasoning LLMs, including commercial systems, exhibit structured-search failures on verifiable computation tasks and degrade as the solution space grows.

  3. Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SiGIR trains a language model to decompose multi-hop questions, self-critique each retrieval and reasoning step, and use cumulative self-rewards in a beam search, beating prior methods by up to 14.4 F1 points on MuSiQue.

  4. Enhancing Large Language Models with Reward-guided Tree Search for Knowledge Graph Question and Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RTSoG combines question decomposition, LLM-reward-guided Monte Carlo Tree Search with a self-critic stop signal, and weighted path stacking to achieve new state-of-the-art KGQA accuracy, though without code or error bars.

  5. A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworks

    cs.AI 2025-01 conditional novelty 6.0 of 10

    A survey that unifies LLM search-based inference frameworks under an MDP-based taxonomy and modular search procedures.

  6. HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Proving

    cs.AI 2024-12 conditional novelty 6.0 of 10

    HunyuanProver, a 7B Lean4 prover trained on autoformalized and self-generated tactic data with a coarse-to-fine distance critic, achieves 68.4% on miniF2F-test.

  7. Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Think&Cite uses self-guided Monte Carlo tree search and progress reward modeling to improve attributed text generation, reporting gains on ASQA, QAMPARI, and ELI5.

  8. Progressive Multimodal Reasoning via Active Retrieval

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.

  9. RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A Monte Carlo tree search reasoning framework where each node is a sub-question and its answer, and retrieval-augmented rewards guide the search toward verifiable answers.

  10. Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks

    cs.AI 2024-12 conditional novelty 6.0 of 10

    Token-level MCTS with CoT prompting lifts Qwen2.5-Coder-32B-Instruct to a 0.351 pass rate on LiveCodeBench-Hard, approaching O1-Mini's pass@1.

  11. Engineering Trustworthy Agentic AI for Critical Systems

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A survey claiming that agentic AI trustworthiness is a single cross-domain problem and outlining a framework for graded, certifiable assurance.

  12. DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent

    cs.AI 2026-03 conditional novelty 5.0 of 10

    A synthetic benchmark of 9,000 multi-hop web-research questions with difficulty tiers and teacher-generated search trajectories, plus an open-source RL training framework that reportedly lets 3B-parameter agents beat ...

  13. GPO: Learning from Critical Steps to Improve LLM Reasoning

    cs.AI 2025-09 conditional novelty 5.0 of 10

    Guided Pivotal Optimization improves LLM reasoning post-training by resetting rollouts at an advantage-estimated critical step, beating each of five base optimizers on seven benchmarks in 34 of 35 comparisons.

  14. Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.

  15. FreePRM: Training Process Reward Models Without Ground Truth Process Labels

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A weakly supervised PRM training method using outcome-only pseudo-labels plus a buffer probability reaches 53.0% F1 on ProcessBench, beating supervised baselines in the paper's comparisons.

  16. One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A pipeline converts short-CoT LLM outputs into o1-style long chain-of-thought rationales using 1K seed reasoning flows, and SFT on the resulting dataset improves downstream RLVR cold-start.

  17. Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SASR dynamically mixes SFT and GRPO training steps using a gradient-norm ratio, and reports improved average accuracy over static SFT-then-RL schedules on GSM8K, MATH, and Knight-and-Knives.

  18. Convert Language Model into a Value-based Strategic Planner

    cs.CL 2025-05 conditional novelty 5.0 of 10

    straQ* uses an LLM's token logits as Q-values to select emotional support strategies, improving long-term response quality and out-of-domain generalization in experiments.

  19. On the Emergence of Thinking in LLMs I: Searching for the Right Intuition

    cs.AI 2025-02 conditional novelty 5.0 of 10

    A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.

  20. Holistically Guided Monte Carlo Tree Search for Intricate Information Seeking

    cs.IR 2025-02 reject novelty 5.0 of 10

    HG-MCTS guides Monte Carlo tree search with an adaptive checklist and self-generated rewards to improve multi-hop information seeking, claiming better benchmark accuracy.

  21. Policy Guided Tree Search for Enhanced LLM Reasoning

    cs.LG 2025-02 conditional novelty 5.0 of 10

    PGTS trains a graph-transformer policy via PPO to guide tree search over LLM reasoning steps, reporting improved accuracy and lower token use than chain-of-thought and MCTS baselines.

  22. Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation

    cs.IR 2025-02 conditional novelty 5.0 of 10

    An LLM recommender that decomposes preference reasoning into summarization, matching, and prediction steps, trained on review text, improves rating accuracy and explanation quality on Amazon and Yelp benchmarks.

  23. Zero-Shot Verification-guided Chain of Thoughts

    cs.CL 2025-01 conditional novelty 5.0 of 10

    COT STEP, a zero-shot prompt that forces numbered reasoning steps, enables competitive accuracy and step-level self-verification, but verifier-guided search does not beat self-consistency.

  24. Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking

    cs.CL 2025-01 reject novelty 5.0 of 10

    HaluSearch reduces LLM hallucinations by generating responses through MCTS-based tree search with a reward model, outperforming CoT, self-consistency, and best-of-N baselines.

  25. Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization

    cs.AI 2024-12 conditional novelty 5.0 of 10

    DAPO trains a step-level value critic and regresses the policy log-ratio to the critic-derived advantage, improving LLM math and code reasoning over the base model on several benchmarks.

  26. Bag of Tricks for Inference-time Computation of LLM Reasoning

    cs.AI 2025-02 conditional novelty 4.0 of 10

    An empirical study showing that inference-time LLM reasoning methods are highly sensitive to sampling hyperparameters and reward-model choices, with recommended defaults of temperature 0.8 and top-p 0.9.

  27. QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search

    cs.LG 2025-02 conditional novelty 4.0 of 10

    QLASS trains a stepwise Q-value model on self-explored trajectories and uses it to guide language agents action by action, outperforming baselines on WebShop, SciWorld, and ALFWorld.

  28. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

  29. Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

    cs.MA 2024-11 conditional novelty 3.0 of 10

    A survey that proposes the Generalist Virtual Agent concept and taxonomies for agent environments, tasks, perceptions, actions, models, and evaluation, concluding that real-world-like environments favor human-like int...

  30. Building Trustworthy AI: Transparent AI Systems via Large Language Models, Ontologies, and Logical Reasoning (TranspNet)

    cs.AI 2024-11 reject novelty 3.0 of 10

    TranspNet is a proposed neural-symbolic pipeline that uses ontologies, RAG, and ASP to verify and explain LLM outputs, but it provides no empirical validation.

Pith tools