REVIEW 11 cited by
Iterative Reasoning Preference Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
Forward citations
Cited by 11 Pith papers
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization
Pseudocode-structured plans paired with preference optimization improve LLM agent success rates and generalization across interactive benchmarks.
-
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
PCPO selects preference pairs by combining correct-answer status with token-level probability consistency, then trains with a weighted DPO+NLL loss, yielding small and partly inconsistent gains over outcome-only metho...
-
Online Knowledge Distillation with Reward Guidance
A preference-based knowledge distillation framework uses a confidence-set reward model in a min-max imitation game, with offline, online, and white-box variants, and outperforms prior KD baselines on LLM benchmarks.
-
From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.
-
Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
BDPO replaces the rejected response probability in the DPO loss denominator with a mixture of the learned and reference policies, yielding better chosen-response probability and better benchmark scores.
-
Control-R: Towards controllable test-time scaling
Control-R-32B is a fine-tuned 32B model that takes 11 control-field scores in the prompt, claiming to let users trade reasoning depth for efficiency and reaching SOTA on AIME2024 and MATH500 at that scale.
-
Frictional Agent Alignment Framework: Slow Down and Don't Break Things
FAAF aligns an LLM with a two-part loss, one part conditioned on a detected belief-misalignment state, to generate reflection-prompting interventions, and reports higher win rates than DPO, IPO, and PPO on collaborati...
-
Technical Report of TeleChat2, TeleChat2.5 and T1
The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.
-
Optimising Language Models for Downstream Tasks: A Post-Training Perspective
A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.
Discussion (0). Sign in to comment.