Pith. sign in

REVIEW 9 cited by

GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.10963 v2 pith:2AEWBDFU submitted 2024-02-13 cs.CL cs.LG

Alex Havrilla , Sharath Raparthy , Christoforus Nalmpantis , Jane Dwivedi-Yu , Maksym Zhuravinskyi , Eric Hambro , Roberta Raileanu This is my paper
classification cs.CLcs.LG
keywords modelswhenormspredictreasoningrefinetrainedcorrectness
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

State-of-the-art language models can exhibit impressive reasoning refinement capabilities on math, science or coding tasks. However, recent work demonstrates that even the best models struggle to identify \textit{when and where to refine} without access to external feedback. Outcome-based Reward Models (\textbf{ORMs}), trained to predict correctness of the final answer indicating when to refine, offer one convenient solution for deciding when to refine. Process Based Reward Models (\textbf{PRMs}), trained to predict correctness of intermediate steps, can then be used to indicate where to refine. But they are expensive to train, requiring extensive human annotations. In this paper, we propose Stepwise ORMs (\textbf{SORMs}) which are trained, only on synthetic data, to approximate the expected future reward of the optimal policy or $V^{\star}$. More specifically, SORMs are trained to predict the correctness of the final answer when sampling the current policy many times (rather than only once as in the case of ORMs). Our experiments show that SORMs can more accurately detect incorrect reasoning steps compared to ORMs, thus improving downstream accuracy when doing refinements. We then train \textit{global} refinement models, which take only the question and a draft solution as input and predict a corrected solution, and \textit{local} refinement models which also take as input a critique indicating the location of the first reasoning error. We generate training data for both models synthetically by reusing data used to train the SORM. We find combining global and local refinements, using the ORM as a reranker, significantly outperforms either one individually, as well as a best of three sample baseline. With this strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned with RL) on GSM8K from 53\% to 65\% when greedily sampled.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new 1,012-question benchmark shows LLMs often fail instructions that deliberately invert common training conventions, revealing a measurable gap in counterintuitive instruction following.

  2. SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    An RL framework that trains vision-language models to self-correct captions via a scene-graph-based reward outperforms SFT and DPO on caption quality.

  3. Decomposing Elements of Problem Solving: What "Math" Does RL Teach?

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning (GRPO) on math LLMs primarily increases execution robustness on already-solvable problems, not planning or coverage of new problems.

  4. Hallucination Detection with Small Language Models

    cs.CL 2025-06 reject novelty 5.0 of 10

    A multi-small-model ensemble with sentence splitting, z-score normalization, and harmonic mean detects hallucinations in RAG answers with a reported 10% F1 gain over single-model baselines.

  5. CORE-KG: An LLM-Driven Knowledge Graph Construction Framework for Human Smuggling Networks

    cs.CL 2025-06 conditional novelty 5.0 of 10

    CORE-KG reduces node duplication by 33.28% and legal noise by 38.37% versus a GraphRAG baseline on 20 human smuggling court cases, through type-aware LLM coreference resolution and domain-filtered extraction prompts.

  6. Boosting LLM Reasoning via Spontaneous Self-Correction

    cs.AI 2025-06 conditional novelty 5.0 of 10

    SPOC trains LLMs to interleave self-verification and solution attempts in a single pass, reporting gains on math benchmarks, though most gains come from stronger first attempts.

  7. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  8. Data Diversification Methods In Alignment Enhance Math Performance In LLMs

    cs.AI 2025-07 reject novelty 4.0 of 10

    DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.

  9. AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

    q-bio.BM 2025-06 reject novelty 4.0 of 10

    DPO with contrastive sequence-annotation alignment improves GO term prediction by 2 to 4 percent relative F1-Max over supervised fine-tuning alone.

Pith tools