SLiC-HF: Sequence Likelihood Calibration with Human Feedback

1. Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function

   cs.LG 2025-06 conditional novelty 7.0 of 10

   ZSPO provably converges to a stationary policy using only the sign of preference feedback, without knowing the link function between preferences and rewards.

2. Test-Time Scaling via Error Localization

   cs.LG 2026-07 conditional novelty 6.0 of 10

   TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

3. Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

   eess.SP 2026-07 conditional novelty 6.0 of 10

   Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.

4. Multi-Turn On-Policy Distillation with Prefix Replay

   cs.LG 2026-07 conditional novelty 6.0 of 10

   ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

5. Unbiased Alignment for Large Language Models with Noisy Preferences

   cs.LG 2026-07 conditional novelty 6.0 of 10

   URM and UDPO losses mathematically correct Bradley–Terry preference flip noise so LLMs can be aligned unbiasedly from noisy preference data without clean supervision.

6. Adaptive Margin RLHF via Preference over Preferences

   cs.LG 2025-09 conditional novelty 6.0 of 10

   Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.

7. Phi-Ground Tech Report: Advancing Perception in GUI Grounding

   cs.CV 2025-07 conditional novelty 6.0 of 10

   Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...

8. DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)

   cs.LG 2025-07 conditional novelty 6.0 of 10

   DPO's connection to human choice theory generalizes: any monotone loss can pair with any choice model in a new KLST* class, so loss convexity is not normatively required.

9. Improving Consistency in Vehicle Trajectory Prediction Through Preference Optimization

   cs.LG 2025-07 conditional novelty 6.0 of 10

   Adapting SimPO preference optimization for multi-agent trajectory prediction lowers collision rates on Argoverse 2, Interaction, and Waymo Open Motion Dataset with minimal accuracy loss.

10. ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.

11. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

12. Debiasing Online Preference Learning via Preference Feature Preservation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PFP preserves the distribution of human preference features during online preference learning by mapping each new instruction to a feature set, converting it to system prompts, and training with DPO.

13. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

14. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

15. CALM: Co-evolution of Algorithms and Language Model for Automatic Heuristic Design

    cs.NE 2025-05 conditional novelty 6.0 of 10

    Co-evolution of an LLM's weights and heuristic search prompts via GRPO-based reinforcement learning yields better optimization heuristics than fixed-model LLM baselines, even with a locally hosted quantized 7B model.

16. SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SoLoPO decouples long-context preference optimization into short-context preference learning plus a reward-consistency term that transfers the behavior to long contexts.

17. ShiQ: Bringing back Bellman to LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    ShiQ is a Bellman-derived loss that makes LLM logits behave as Q-values, enabling off-policy token-level and multi-turn reinforcement-learning fine-tuning with the softmax over logits as the policy.

18. Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LPC adds a discrete latent code layer to DPO-family alignment objectives, improving average preference accuracy and downstream scores across three base models.

19. Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL

    cs.LG 2025-05 conditional novelty 6.0 of 10

    GVM-RAFT allocates samples per prompt according to acceptance rate and gradient norm, reducing gradient variance and speeding up RAFT-style chain-of-thought training.

20. From Evidence to Belief: A Bayesian Epistemology Approach to Language Models

    cs.AI 2025-04 conditional novelty 6.0 of 10

    Language models follow Bayesian confirmation with golden evidence but fail to follow Bayesian disconfirmation and irrelevance assumptions, and they are biased toward golden evidence.

21. Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Preference examples vary in difficulty; overly difficult examples degrade DPO alignment, and filtering them out improves AlpacaEval 2 win rates by 9-16 percentage points.

22. Design Considerations in Offline Preference-based RL

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A unified theory of offline RLHF shows that loss curvature and data coverage control suboptimality, explaining why squared-loss IPO is more stable than logistic-loss DPO.

23. Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms

    cs.LG 2025-02 reject novelty 6.0 of 10

    The authors show that DPO, IPO, DRO, PPO and SAC can be viewed through one loss-construction framework, but their claim that DPO's target distribution differs from the standard DPO target rests on an incomplete argument.

24. On Almost Surely Safe Alignment of Large Language Models at Inference-Time

    cs.LG 2025-02 conditional novelty 6.0 of 10

    An inference-time beam-search method with a safety-state tracker and latent critic enforces a user-supplied safety cost model, with an almost-sure guarantee only relative to that model.

25. BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

    cs.LG 2025-01 reject novelty 6.0 of 10

    BRiTE is an EM-style algorithm that uses RL to sample high-likelihood reasoning chains and fine-tunes the LLM on them, with a theoretical 1/T convergence rate and mixed empirical gains.

26. CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A confidence-reward score for selecting preference pairs improves DPO-based machine translation fine-tuning over reward-only selection methods on ALMA-7B and NLLB-1.3B.

27. Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Applying test-time verifiers, DPO preference alignment, and a new adaptive reward model (PARM) to autoregressive image generators improves GenEval score from 53% to 77%.

28. PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A pairwise chain-of-thought judge combined with a knockout tournament outperforms score-based reward models for best-of-N sampling in math reasoning.

29. Online Preference Alignment for Language Models via Count-based Exploration

    cs.LG 2025-01 conditional novelty 6.0 of 10

    COPO adds a pseudo-count exploration bonus to DPO-style preference optimization and reports improved online RLHF results on Zephyr and Llama-3.

30. DIVE: Diversified Iterative Self-Improvement

    cs.CL 2025-01 conditional novelty 6.0 of 10

    DIVE combines global sample pooling with diversity-aware data selection to counter output diversity collapse in iterative preference learning for LLMs.

31. Efficient Long Context Language Model Retrieval with Compression

    cs.IR 2024-12 conditional novelty 6.0 of 10

    CoLoR, a preference-optimized passage compressor, cuts LCLM retrieval context by 1.91x while improving average retrieval accuracy by 6% over nine benchmarks.

32. Understanding the Logic of Direct Preference Alignment through Logic

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Direct preference alignment losses can be expressed as logical programs over model predictions, yielding an organized landscape of billions of definable losses and a route to new variants.

33. VideoDPO: Omni-Preference Alignment for Video Diffusion Generation

    cs.CV 2024-12 reject novelty 6.0 of 10

    VideoDPO shows that DPO-style training on automatically selected best and worst video pairs improves overall VBench scores on three open text-to-video models, with some sub-metrics degrading and weak gains on external...

34. Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A hybrid online-plus-offline preference optimization algorithm, HPO, provably needs fewer samples than pure online or offline RLHF in linear MDP settings.

35. Weighted-Reward Preference Optimization for Implicit Model Fusion

    cs.CL 2024-12 conditional novelty 6.0 of 10

    WRPO tunes an 8B chat model by combining its own preferred responses (on-policy) with high-reward responses from ten heterogeneous source LLMs (off-policy) using an annealed weight, beating prior fusion and preference...

36. T-REG: Preference Optimization with Token-Level Reward Regularization

    cs.CL 2024-12 conditional novelty 6.0 of 10

    T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...

37. Time-Reversal Provides Unsupervised Feedback to LLMs

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Time-reversed language models, which score queries given responses, improve best-of-N reranking, citation attribution, retrieval, and jailbreak filtering over forward perplexity baselines.

38. AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward

    cs.CV 2024-11 conditional novelty 6.0 of 10

    AToM uses GPT-4Vision-generated preference scores to fine-tune MotionGPT with IPO and LoRA, improving event-level alignment for integrity, temporal order, and frequency in text-to-motion generation.

39. Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

    cs.LG 2024-11 conditional novelty 6.0 of 10

    The paper generalizes Bradley-Terry reward modeling to ordinal feedback labels and proves that, under a marginal unbiasedness assumption, finer-grained labels reduce Rademacher complexity and can improve reward learning.

40. Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Shell-LCC models the high-quality data manifold as an isotropic shell to derive cost-free reward signals that improve realism and high-frequency details in text-to-video generation.

41. Normalized Rewards for Preference Optimization

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...

42. Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A modified DPO loss with a hinge margin improves small LLM alignment on AlpacaEval by about 2 points over the APO-zero baseline.

43. Cultivating Helpful, Personalized, and Creative AI Tutors: A Framework for Pedagogical Alignment using Reinforcement Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    EduAlign trains a three-dimensional reward model (HPC-RM) and uses GRPO to fine-tune Qwen2.5-72B, reporting improved helpfulness, personalization, and creativity on its own and public benchmarks.

44. Value-Free Policy Optimization via Reward Partitioning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    RPO regresses the policy log-ratio against rewards centered by an empirical partition function estimated from same-prompt responses, removing the value network and reporting improved alignment, diversity, and lower toxicity.

45. Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

    cs.LG 2025-06 conditional novelty 5.0 of 10

    SFT is cast as a special case of implicit reward learning, and a slower SFT learning rate improves downstream DPO/SimPO results by up to 25% relative win rate.

46. On Monotonicity in AI Alignment

    math.ST 2025-06 conditional novelty 5.0 of 10

    Comparison-based preference learning losses of the studied form are locally pairwise monotone at stable minima, while individual-score and probability monotonicity require much stronger conditions that likely fail for...

47. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

48. Direct Advantage Regression: Aligning LLMs with Online AI Reward

    cs.AI 2025-04 conditional novelty 5.0 of 10

    DAR aligns LLMs through advantage-weighted supervised fine-tuning on online AI scalar rewards with dual KL regularization, and reports win-rate gains over online RLHF and online preference methods.

49. LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

    cs.CL 2025-02 conditional novelty 5.0 of 10

    LARPO, an iterative preference optimization method that adapts information retrieval techniques such as listwise ranking losses, hard negatives, and candidate lists, is claimed to substantially improve LLM alignment o...

50. A Checks-and-Balances Framework for Context-Aware Ethical AI Alignment

    cs.CL 2025-01 reject novelty 5.0 of 10

    A checks-and-balances alignment framework uses separate AI agents for knowledge, guardrails, and adversarial review, and an emotion-based classifier that beats zero-shot GPT-4 by 11.3 points on love-letter valence labeling.

51. Harnessing Preference Optimisation in Protein LMs for Hit Maturation in Cell Therapy

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A preference-fine-tuned protein language model ranks single and double mutants of CAR binding domains better than the pretrained model, finding several mutants that outperform their parent.

52. DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs

    cs.SE 2024-11 conditional novelty 5.0 of 10

    DSTC builds preference pairs by selecting the hardest self-generated test that the best self-generated code passes, then fine-tunes code LMs with DPO or KTO to improve pass@1 accuracy.

53. BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

    cs.CL 2024-11 conditional novelty 5.0 of 10

    BPO balances knowledge breadth and depth in preference data by compressing prompts and dynamically augmenting the number of response pairs per prompt using gradient-based clustering, achieving stronger alignment with ...

54. RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution

    cs.CL 2024-11 conditional novelty 5.0 of 10

    RED assigns token-level rewards by taking the difference of a sequence reward model's scores on adjacent prefixes, improving RLHF training without additional reward-model training.

55. Alignment-Aware Decoding

    cs.LG 2025-09 conditional novelty 4.0 of 10

    AAD decodes by greedily maximizing the per-token DPO/SFT log-likelihood ratio under a min-p plausibility filter and reports higher oracle-reward win rates than greedy, best-of-N, and emulated fine-tuning baselines.

56. BPO: Revisiting Preference Modeling in Direct Preference Optimization

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Replacing DPO's relative reward margin with min(r_w, -α r_l) improves math benchmark accuracy by 5-10 points in this paper, though the exact gain figure is misreported and α is tuned on test data.

57. On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

    cs.LG 2025-05 reject novelty 4.0 of 10

    Symmetric losses preserve action rankings under symmetric label noise, and the paper's claim that they also handle asymmetric noise is invalid.

58. InfoPO: On Mutual Information Maximization for Large Language Model Alignment

    cs.LG 2025-05 reject novelty 4.0 of 10

    InfoPO's mutual information derivation fails: its rejected-response regularizer is constant in expectation, leaving plain supervised fine-tuning on chosen responses.

59. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

    cs.LG 2024-12 reject novelty 4.0 of 10

    Cal-DPO modifies DPO by adding squared losses that anchor the policy's implicit rewards of chosen and rejected responses to plus or minus 1/(2*beta), reporting gains on reasoning, summarization, and dialogue benchmarks.

60. SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Higher learning-rate-to-batch-size ratios improve reasoning-task performance in small language models, and the resulting SmolTulu-1.7B reports top sub-2B scores on IFEval and GSM8K.

See all 64 Pith citations