Pith. sign in

REVIEW 2 cited by

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.17031 v1 pith:BMTMCSSA submitted 2024-03-24 cs.LG

Shengyi Huang , Michael Noukhovitch , Arian Hosseini , Kashif Rasul , Weixun Wang , Lewis Tunstall This is my paper
classification cs.LG
keywords detailsrlhffeedbackimplementationmodelmodelsopenaisummarization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field (\url{https://github.com/vwxyzjn/summarize_from_feedback_details}).

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning a Pessimistic Reward Model in RLHF

    cs.LG 2025-05 reject novelty 6.0 of 10

    Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.

  2. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

Pith tools