Pith. sign in

REVIEW 13 cited by

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.15028 v3 pith:6XBWKF7W submitted 2023-09-26 cs.CL cs.AIcs.LG

Jiacheng Liu , Andrew Cohen , Ramakanth Pasunuru , Yejin Choi , Hannaneh Hajishirzi , Asli Celikyilmaz This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords textnetworkpolicysearchvaluedecodingdemonstrategeneration
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Inference-time search algorithms such as Monte-Carlo Tree Search (MCTS) may seem unnecessary when generating natural language text based on state-of-the-art reinforcement learning such as Proximal Policy Optimization (PPO). In this paper, we demonstrate that it is possible to get extra mileage out of PPO by integrating MCTS on top. The key idea is not to throw out the value network, a byproduct of PPO training for evaluating partial output sequences, when decoding text out of the policy network. More concretely, we present a novel value-guided decoding algorithm called PPO-MCTS, which can integrate the value network from PPO to work closely with the policy network during inference-time generation. Compared to prior approaches based on MCTS for controlled text generation, the key strength of our approach is to reduce the fundamental mismatch of the scoring mechanisms of the partial outputs between training and test. Evaluation on four text generation tasks demonstrate that PPO-MCTS greatly improves the preferability of generated text compared to the standard practice of using only the PPO policy. Our results demonstrate the promise of search algorithms even on top of the aligned language models from PPO, and the under-explored benefit of the value network.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  2. Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

    cs.LG 2025-10 conditional novelty 6.0 of 10

    MAHALO aligns LLMs to multiple objectives in one model via per-objective action heads and PRM-guided decoding, improving math, value, and tutoring metrics jointly.

  3. It's Not That Simple. An Analysis of Simple Test-Time Scaling

    cs.LG 2025-07 conditional novelty 6.0 of 10

    The apparent scaling behavior of the s1 model is mostly an artifact of truncating long outputs, while appending 'Wait' does not reliably improve answers.

  4. Breaking the Myth: Can Small Models Infer Postconditions Too?

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Fine-tuning Qwen2.5-Coder-7B on 1.5K reasoning examples yields postcondition generation close to GPT-4o and better than 32B open models on Defects4J.

  5. Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models

    cs.HC 2025-06 conditional novelty 6.0 of 10

    Interactive Reasoning, instantiated as Hippo, lets users view and edit an LLM's chain-of-thought as a tree, and a 16-person study reports improved perceived control, sense-making, and assumption awareness.

  6. Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A systematic benchmark shows multi-agent debate's value depends on task difficulty, model scale, and agent diversity: limited for math unless problems are hard or models weak, but useful for safety when agents are diverse.

  7. Iterative Deepening Sampling as Efficient Test-Time Scaling

    cs.CL 2025-02 conditional novelty 6.0 of 10

    ID-Sampling inserts self-correction trigger sentences at geometrically increasing token budgets during LLM generation, improving pass rates on MATH-500 and AIME over vanilla Best-of-N and majority voting.

  8. Large Language Models to Diffusion Finetuning

    cs.CL 2025-01 conditional novelty 6.0 of 10

    L2D finetunes a small parallel diffusion path on a frozen pretrained LLM so that running more diffusion steps at inference monotonically improves task accuracy.

  9. Progressive Multimodal Reasoning via Active Retrieval

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.

  10. First Finish Search: Efficient Test-Time Scaling in Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    First Finish Search launches n parallel reasoning traces and returns the shortest one, improving accuracy on AIME benchmarks while cutting token use.

  11. Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning

    cs.LG 2024-12 conditional novelty 4.0 of 10

    Adding a self-correction RL stage before MCTS-DPO improves arithmetic reasoning accuracy by 2.00 to 4.94 points on GSM8K and MATH.

  12. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

  13. Reinforcement Learning from Human Feedback

    cs.LG 2025-04 unverdicted novelty 2.0 of 10

    The book introduces the origins, mathematical setup, and optimization stages of RLHF including reward modeling, reinforcement learning, rejection sampling, and direct alignment algorithms.

Pith tools