Pith. sign in

REVIEW 11 cited by

HelpSteer2: Open-source dataset for training top-performing reward models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.08673 v1 pith:6WX6LW3L submitted 2024-06-12 cs.CL cs.AIcs.LG

Zhilin Wang , Yi Dong , Olivier Delalleau , Jiaqi Zeng , Gerald Shen , Daniel Egert , Jimmy J. Zhang , Makesh Narsimhan Sreedhar
show 1 more author
Oleksii Kuchaiev
This is my paper
classification cs.CLcs.AIcs.LG
keywords helpsteer2modelsrewardpreferencedatasetsllmsdatasetmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usage imposed by model providers. To improve upon both generated responses and attribute labeling quality, we release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0). Using a powerful internal base model trained on HelpSteer2, we are able to achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming currently listed open and proprietary models, as of June 12th, 2024. Notably, HelpSteer2 consists of only ten thousand response pairs, an order of magnitude fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly efficient for training reward models. Our extensive experiments demonstrate that reward models trained with HelpSteer2 are effective in aligning LLMs. In particular, we propose SteerLM 2.0, a model alignment approach that can effectively make use of the rich multi-attribute score predicted by our reward models. HelpSteer2 is available at https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at https://github.com/NVIDIA/NeMo-Aligner

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Step-Level Preference Learning for Generative Agents in Social Simulations

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Step-level human preference data collected via SimPref, then SFT+DPO, improves long-horizon social-simulation behavior of open-weight LLM agents on held-out events.

  2. TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A temporal-aware hierarchical reinforcement learning method improves a 7B LLM's social reasoning enough to rival DeepSeek-R1 and OpenAI-O3 on in-domain theory-of-mind benchmarks.

  3. Improving Large Vision and Language Models by Learning from a Panel of Peers

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.

  4. Second-Order Bounds for [0,1]-Valued Regression via Betting Loss

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A betting loss for bounded regression yields a variance-adaptive generalization bound, improving on first-order bounds that scale with the worst-case proxy f(1-f).

  5. Tiny Reward Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    TinyRM shows that 400M-parameter bidirectional masked language models, tuned with FLAN-style prompting, DoRA, and layer freezing, outperform a 70B reward model on RewardBench reasoning and come close on safety.

  6. When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Hedged sampling, checklist-based one-pass selection (CHOPS), and cross-lingual MBR (X-MBR) improve multilingual LLM output quality when scaling from one to five samples.

  7. PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PersonaFeedback provides a human-labeled benchmark showing current LLMs, including strong reasoners, score only about 65-70 percent on hard personalization choices, and explicit persona information helps more than retrieval.

  8. Discriminative Policy Optimization for Token-Level Reward Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Q-RM learns token-level rewards from preference data via a discriminative policy and improves RL alignment on math, reading, and instruction-following benchmarks.

  9. Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    On-policy multi-evaluator consensus filtering retains ~3.5% of candidates and, with KTO on 1.9k labels, improves Mistral-7B preference metrics over stronger data-heavy baselines, with mixed transfer to Llama.

  10. OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 2.5M-example code reasoning dataset with critique traces enables Qwen2.5-based models to surpass prior open-weight distilled models on LiveCodeBench via test-time self-critique selection.

  11. SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    SFT-GO retrains LLMs by focusing on the worst-performing group of important or unimportant tokens, yielding modest average benchmark improvements over standard supervised fine-tuning.

Pith tools