Pith. sign in

REVIEW 27 cited by

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12845 v1 pith:QYZC66WY submitted 2024-06-18 cs.LG cs.CL

Haoxiang Wang , Wei Xiong , Tengyang Xie , Han Zhao , Tong Zhang This is my paper · ORCID
classification cs.LGcs.CL
keywords humanrewardmodelpreferencestrainedarmormperformanceabsolute-rating
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

  2. OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    OpsLLM's pipeline (HITL data curation, SFT, GRPO RL with a domain process reward model) improves LLM accuracy on software-operations QA and RCA, especially on in-distribution root-cause-analysis tasks.

  3. Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

    cs.LG 2025-10 conditional novelty 6.0 of 10

    MAHALO aligns LLMs to multiple objectives in one model via per-objective action heads and PRM-guided decoding, improving math, value, and tutoring metrics jointly.

  4. Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICON2 uses representation-space steering to generate preference data from the model itself, improving alignment benchmarks and cutting cost.

  5. ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models

    cs.CL 2025-08 conditional novelty 6.0 of 10

    ReaLM trains small language models to learn from both right and wrong reasoning chains, then fades the chains out so the model reasons independently, improving benchmark accuracy.

  6. BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

    cs.CL 2025-07 conditional novelty 6.0 of 10

    BMMR provides a 110k-question bilingual, multimodal, college-level dataset across 300 subjects where state-of-the-art models score at most about 50%.

  7. BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.

  8. Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A small aligned model drafts the start of an LLM response, then a large base model takes over via a confidence-based switch, improving preference alignment without fine-tuning the large model.

  9. Accelerating RLHF Training with Reward Variance Increase

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A new reward reshaping method provably increases reward variance for GRPO-based RLHF training, with an O(n log n) global optimization algorithm and preliminary speedups in experiments.

  10. Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Fair-PP contributes a synthetic persona-anchored preference dataset for social equity and a reweighted DPO/SFT alignment method that outperforms baselines on LLM-similarity tests.

  11. MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MedGUIDE tests whether LLMs follow structured NCCN cancer-care decision trees and finds that even medical LLMs often lag general models on this task.

  12. R.I.P.: Better Models by Survival of the Fittest Prompts

    cs.CL 2025-01 conditional novelty 6.0 of 10

    RIP filters preference-optimization training data by keeping prompts whose rejected responses are high quality and whose chosen/rejected reward gap is small, yielding consistent benchmark gains over unfiltered data.

  13. Data-adaptive Safety Rules for Training Reward Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Selecting the five safety rules with largest response discrepancy maximizes mutual information under stated assumptions, and a reward model trained with this adaptive labeling achieves top RewardBench safety.

  14. InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

    cs.CV 2025-01 conditional novelty 6.0 of 10

    IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.

  15. AlphaPO: Reward Shape Matters for LLM Alignment

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AlphaPO replaces SimPO's log reward with a length-normalized alpha-divergence reward; a slightly positive alpha improves AlpacaEval 2 length-controlled win rates by 7-10 percent on two instruct models.

  16. Boosting LLM via Learning from Data Iteratively and Selectively

    cs.CL 2024-12 conditional novelty 6.0 of 10

    IterIT iteratively re-scores instruction samples during fine-tuning and greedily selects a small, diverse, high-complexity subset each epoch.

  17. T-REG: Preference Optimization with Token-Level Reward Regularization

    cs.CL 2024-12 conditional novelty 6.0 of 10

    T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...

  18. Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

    cs.LG 2024-11 conditional novelty 6.0 of 10

    The paper generalizes Bradley-Terry reward modeling to ordinal feedback labels and proves that, under a marginal unbiasedness assumption, finer-grained labels reduce Rademacher complexity and can improve reward learning.

  19. Adaptive Decoding via Latent Preference Optimization

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Adaptive Decoding learns to pick a discrete sampling temperature at token or sequence level via a DPO-style loss over latent temperature choices, and beats fixed temperatures on average across three task families.

  20. MOSLIM:Align with diverse preferences in prompts through reward classification

    cs.CL 2025-05 reject novelty 5.0 of 10

    A prompt-controlled multi-objective alignment method using a multi-head classification reward model and a z-score reward mapping, claimed to work with off-the-shelf models.

  21. MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

    cs.CR 2025-05 conditional novelty 5.0 of 10

    MTSA pairs a thought-guided red-team attacker with future-reward multi-turn reinforcement learning to make LLMs more robust against multi-round jailbreaks.

  22. CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A small CoT-based synthesizer model trained on candidate-response analysis improves LLM reasoning accuracy, including cases where all sampled candidate answers are incorrect.

  23. Interpreting Language Reward Models via Contrastive Explanations

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Reward model preferences can be explained by generating counterfactual and semifactual answer variations along 15 hand-picked evaluation attributes and measuring which attribute changes flip the model's preference.

  24. T-POP: Test-Time Personalization with Online Preference Feedback

    cs.LG 2025-09 conditional novelty 4.0 of 10

    T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

  25. DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A reward model that multiplies stepwise correctness and potential scores improves best-of-N verification accuracy for math reasoning.

  26. Atla Selene Mini: A General Purpose Evaluation Model

    cs.CL 2025-01 conditional novelty 4.0 of 10

    The paper presents Selene Mini, an 8B open-weights judge model that reports state-of-the-art average scores across 11 LLM evaluation benchmarks, with gains on medical and financial expert agreement.

  27. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

Pith tools