REVIEW 21 cited by
Token-level Direct Preference Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Fine-tuning pre-trained Large Language Models (LLMs) is essential to align them with human values and intentions. This process often utilizes methods like pairwise comparisons and KL divergence against a reference LLM, focusing on the evaluation of full answers generated by the models. However, the generation of these responses occurs in a token level, following a sequential, auto-regressive fashion. In this paper, we introduce Token-level Direct Preference Optimization (TDPO), a novel approach to align LLMs with human preferences by optimizing policy at the token level. Unlike previous methods, which face challenges in divergence efficiency, TDPO incorporates forward KL divergence constraints for each token, improving alignment and diversity. Utilizing the Bradley-Terry model for a token-based reward system, TDPO enhances the regulation of KL divergence, while preserving simplicity without the need for explicit reward modeling. Experimental results across various text tasks demonstrate TDPO's superior performance in balancing alignment with generation diversity. Notably, fine-tuning with TDPO strikes a better balance than DPO in the controlled sentiment generation and single-turn dialogue datasets, and significantly improves the quality of generated responses compared to both DPO and PPO-based RLHF methods. Our code is open-sourced at https://github.com/Vance0124/Token-level-Direct-Preference-Optimization.
Forward citations
Cited by 21 Pith papers
-
Token-Level Credit Assignment Optimization for Generative Document Retrieval
TCA, a token-level credit assignment reinforcement learning framework, improves R@1 and MRR@10 over sequence-level RL baselines in generative document retrieval on MS MARCO and Natural Questions.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift
A token-level correctness classifier trained with LoRA and then merged into the model boosts out-of-distribution factuality in summarization and translation.
-
rePIRL: Learn PRM with Inverse RL for LLM Reasoning
rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.
-
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
DiPO is a distribution-level unlearning method that constructs preference distributions from the model's own high-confidence logits and achieves state-of-the-art forget quality on TOFU while preserving utility.
-
FocusDPO: Dynamic Preference Optimization for Multi-Subject Personalized Image Generation via Adaptive Focus
FocusDPO adds dynamic spatial weighting to preference-based fine-tuning, improving subject fidelity and reducing attribute leakage in multi-subject personalized image generation.
-
Learning Safety Constraints for Large Language Models
A polytope learned in LLM representation space can detect unsafe regions and steer outputs back to safety at inference time, reducing jailbreak success across several models.
-
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
PCPO selects preference pairs by combining correct-answer status with token-level probability consistency, then trains with a weighted DPO+NLL loss, yielding small and partly inconsistent gains over outcome-only metho...
-
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
SGDPO modifies DPO with a subsequence-based pilot term and reports up to 9.19% relative MT-Bench gains, though the claimed gradient mechanism is not fully derived for the implemented loss.
-
Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Policy-labeled Preference Learning models preferences with regret and behavior-policy labels, adds contrastive KL regularization, and reports improved RLHF performance on MetaWorld offline and online control tasks.
-
VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
A hierarchical DPO training method and dataset reduce hallucination in video LLMs by aligning preferences at video, clip, object, and token levels.
-
PIPA: Preference Alignment as Prior-Informed Statistical Estimation
A unified maximum-likelihood framework with prior constraints that recovers DPO and KTO as special cases and yields new PIPA-M/PIPA-N losses with 3-10% gains on GSM8K and MATH.
-
SDPO: Segment-Level Direct Preference Optimization for Social Agents
SDPO trains social agents by applying DPO-style preference optimization to equal-length key segments around an erroneous turn, and reports state-of-the-art SOTOPIA scores.
-
T-REG: Preference Optimization with Token-Level Reward Regularization
T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...
-
Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability
A contrastive token-scoring method identifies 'critical tokens' in incorrect reasoning traces and penalizes them in DPO, yielding small but consistent accuracy gains on math benchmarks.
-
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
The paper generalizes Bradley-Terry reward modeling to ordinal feedback labels and proves that, under a marginal unbiasedness assumption, finer-grained labels reduce Rademacher complexity and can improve reward learning.
-
AI Agent Behavioral Science
AI agents should be studied as behavioral entities shaped by context and interaction, not only as trained models.
-
A Survey on Progress in LLM Alignment from the Perspective of Reward Design
This paper organizes the LLM alignment literature into a reward-design-centered taxonomy and claims the field's evolution runs from rule-based to learned rewards and from RL-based to RL-free optimization.
-
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
Training 2D-DPO with an expected loss over uniform segment-score perturbations yields higher win rates under score noise than a clean-trained 2D-DPO baseline, though the comparison is confounded.
-
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Cal-DPO modifies DPO by adding squared losses that anchor the policy's implicit rewards of chosen and rejected responses to plus or minus 1/(2*beta), reporting gains on reasoning, summarization, and dialogue benchmarks.
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
Discussion (0). Continue with ORCID to comment.