Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.
The rejection sampling process is effectively a policy as it takes a prompt as input and stochastically outputs a response
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Learning a Pessimistic Reward Model in RLHF
Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.