Pith. sign in

REVIEW 10 cited by

HelpSteer2-Preference: Complementing Ratings with Preferences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.01257 v2 pith:QN3ABX2I submitted 2024-10-02 cs.LG cs.AIcs.CL

Zhilin Wang , Alexander Bukharin , Olivier Delalleau , Daniel Egert , Gerald Shen , Jiaqi Zeng , Oleksii Kuchaiev , Yi Dong This is my paper
classification cs.LGcs.AIcs.CL
keywords datamodelsrewardbradley-terryregressionadequatelyapproachhttps
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024. We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new -- 1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Online self-play between attacker and defender roles of a single LLM improves safety robustness and attack diversity across Llama and Qwen models.

  2. Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

    cs.CR 2026-07 conditional novelty 6.0 of 10

    An evidence-grounded hybrid of 4B and 8B open-weight models reaches 35.30% on malware-report QA, slightly beating the best ungrounded frontier model (34.77%) while trailing a frontier model given the same evidence (38.22%).

  3. Adaptive Margin RLHF via Preference over Preferences

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.

  4. Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs

    cs.SE 2025-08 conditional novelty 6.0 of 10

    Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.

  5. Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LLaVA-Reward extracts reward scores from the hidden states of a multimodal LLM with a skip-connection cross-attention head, and reports state-of-the-art text-to-image evaluation across alignment, fidelity, and safety.

  6. Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A small aligned model drafts the start of an LLM response, then a large base model takes over via a confidence-based switch, improving preference alignment without fine-tuning the large model.

  7. OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 2.5M-example code reasoning dataset with critique traces enables Qwen2.5-based models to surpass prior open-weight distilled models on LiveCodeBench via test-time self-critique selection.

  8. From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuning

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Controlling the variety of mid-frequency response tokens during SFT dataset construction correlates more strongly with downstream model performance than instruction-level diversity strategies.

  9. HEAL: A Hypothesis-Based Preference-Aware Analysis Framework

    cs.CL 2025-08 conditional novelty 4.0 of 10

    HEAL evaluates preference optimization by measuring ranking accuracy and strength correlation between model likelihoods and proxy reward scores over multi-response hypothesis spaces.

  10. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools