Pith. sign in

REVIEW 35 cited by

RRHF: Rank Responses to Align Language Models with Human Feedback without tears

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.05302 v3 pith:S4YJM6DB submitted 2023-04-11 cs.CL

Zheng Yuan , Hongyi Yuan , Chuanqi Tan , Wei Wang , Songfang Huang , Fei Huang This is my paper · ORCID
classification cs.CL
keywords rrhfhumanmodelmodelsresponseslanguagealignpreferences
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ProSec: Fortifying Code LLMs with Proactive Security Alignment

    cs.CR 2024-11 conditional novelty 8.0 of 10

    ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.

  2. LPOI: Listwise Preference Optimization for Vision Language Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    LPOI reduces VLM hallucination by training the model to prefer the original image over progressively masked versions of the same image, using a listwise ranking loss built from pairwise preference data.

  3. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  4. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  5. Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

    eess.SP 2026-07 conditional novelty 6.0 of 10

    Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.

  6. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  7. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  8. RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    RLearner-LLM's Hybrid-DPO fuses DeBERTa NLI and LLM verifier scores to deliver up to 6x higher NLI entailment than standard SFT while preserving answer coverage across academic domains.

  9. Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

    cs.LG 2025-10 conditional novelty 6.0 of 10

    MAHALO aligns LLMs to multiple objectives in one model via per-objective action heads and PRM-guided decoding, improving math, value, and tutoring metrics jointly.

  10. Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security

    cs.CR 2025-07 conditional novelty 6.0 of 10

    An iterative attacker-defender reinforcement learning method that makes a multimodal LLM refuse more jailbreak prompts without over-refusing ordinary queries.

  11. Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A knowledge-graph-guided preference optimization framework that fine-tunes protein language models to generate fewer sequences similar to known harmful proteins.

  12. Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A flipped distillation method lets a decoder-only LLM learn text-matching similarity from a smaller encoder teacher through LoRA and a margin-aware contrastive loss, improving matching accuracy and online FAQ retrieval.

  13. From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.

  14. IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages

    cs.CL 2025-06 conditional novelty 6.0 of 10

    IndoSafety, a culturally grounded safety benchmark for five Indonesian language varieties, shows unsafe response rates up to 40% in regional models and demonstrates that safety tuning on formal Indonesian transfers to...

  15. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

  16. SimulPL: Aligning Human Preferences in Simultaneous Machine Translation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    SimulPL adds latency-aware preference optimization to simultaneous machine translation and reports better human-aligned quality at low latency on three language pairs.

  17. Understanding the Logic of Direct Preference Alignment through Logic

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Direct preference alignment losses can be expressed as logical programs over model predictions, yielding an organized landscape of billions of definable losses and a route to new variants.

  18. Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    PoFT is a new supervised fine-tuning loss that makes the target model outrank aligned reference LLMs on the same data, improving robustness and slightly raising benchmark scores.

  19. Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A diffusion-model alignment method estimates the terminal image distribution from intermediate steps, deriving a credit assignment that emphasizes middle denoising steps.

  20. SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

    cs.CV 2024-11 conditional novelty 6.0 of 10

    SymDPO replaces answer text in multimodal demonstrations with meaningless symbols during preference training, forcing models to use image context and improving in-context learning performance on five benchmarks.

  21. Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A variational Bayesian inverse RL objective, AVA, models token-level rewards and outperforms reward-difference baselines in LLM alignment on standard benchmarks.

  22. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

  23. Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    A jailbreak defense that reasons about hidden manipulations in attack prompts, trained with supervised fine-tuning plus entropy-guided reinforcement learning, generalizes to attacks never seen in training.

  24. Activation Reward Models for Few-Shot Model Alignment

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...

  25. Value-Free Policy Optimization via Reward Partitioning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    RPO regresses the policy log-ratio against rewards centered by an empirical partition function estimated from same-prompt responses, removing the value network and reporting improved alignment, diversity, and lower toxicity.

  26. AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models

    cs.LG 2025-06 reject novelty 5.0 of 10

    AMoPO uses the model's own token probabilities to define Gaussian-sampled weights, combining per-dimension SimPO-style losses for reference-free multi-objective alignment.

  27. Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

    cs.CL 2025-05 reject novelty 5.0 of 10

    MARA aligns LLMs with human preferences by training a 4M-parameter MLP to accept or reject candidate tokens, avoiding full-model fine-tuning, with measured gains based on the same reward models used in training.

  28. Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Training-free amplification of selected last-layer activations, combined with 'wait' token insertion, elicits long chain-of-thought reasoning in base LLMs and improves accuracy on math and science benchmarks.

  29. DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization

    cs.LG 2025-02 conditional novelty 5.0 of 10

    SPO fine-tunes an LLM to rewrite drug molecules into analogs that score higher on docking, drug-likeness, solubility, and synthesizability, beating baselines on two protein targets.

  30. LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

    cs.CL 2025-02 conditional novelty 5.0 of 10

    LARPO, an iterative preference optimization method that adapts information retrieval techniques such as listwise ranking losses, hard negatives, and candidate lists, is claimed to substantially improve LLM alignment o...

  31. AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

    q-bio.BM 2025-06 reject novelty 4.0 of 10

    DPO with contrastive sequence-annotation alignment improves GO term prediction by 2 to 4 percent relative F1-Max over supervised fine-tuning alone.

  32. Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences

    cs.CV 2025-06 conditional novelty 4.0 of 10

    SmPO-Diffusion improves diffusion-model preference alignment with reward-model soft labels and ReNoise inversion, reporting higher human-preference scores and up to 26x lower training cost than Diffusion-KTO.

  33. PKG-DPO: Optimizing Domain-Specific AI systems with Physics Knowledge Graphs and Direct Preference Optimization

    cs.AI 2025-08 reject novelty 3.0 of 10

    A physics knowledge graph plus Direct Preference Optimization reduces physically invalid AI recommendations in welding by 17% over knowledge-graph-only DPO, but the evidence is weakened by circular evaluation and no p...

  34. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

  35. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools