REVIEW 12 cited by
RL with KL penalties is better viewed as Bayesian inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Reinforcement learning (RL) is frequently employed in fine-tuning large language models (LMs), such as GPT-3, to penalize them for undesirable features of generated sequences, such as offensiveness, social bias, harmfulness or falsehood. The RL formulation involves treating the LM as a policy and updating it to maximise the expected value of a reward function which captures human preferences, such as non-offensiveness. In this paper, we analyze challenges associated with treating a language model as an RL policy and show how avoiding those challenges requires moving beyond the RL paradigm. We start by observing that the standard RL approach is flawed as an objective for fine-tuning LMs because it leads to distribution collapse: turning the LM into a degenerate distribution. Then, we analyze KL-regularised RL, a widely used recipe for fine-tuning LMs, which additionally constrains the fine-tuned LM to stay close to its original distribution in terms of Kullback-Leibler (KL) divergence. We show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function. We argue that this Bayesian inference view of KL-regularised RL is more insightful than the typically employed RL perspective. The Bayesian inference view explains how KL-regularised RL avoids the distribution collapse problem and offers a first-principles derivation for its objective. While this objective happens to be equivalent to RL (with a particular choice of parametric reward), there exist other objectives for fine-tuning LMs which are no longer equivalent to RL. That observation leads to a more general point: RL is not an adequate formal framework for problems such as fine-tuning language models. These problems are best viewed as Bayesian inference: approximating a pre-defined target distribution.
Forward citations
Cited by 12 Pith papers
-
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
CALM uses bilevel optimization to tune per-vocabulary temperature-like logit adjustments during LLM fine-tuning, and reports improved out-of-domain calibration for aligned language models.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis
Smoothed Best-of-N has finite-sample KL and regret bounds under imperfect reward models, and tuning its temperature can make its regret bound beat hard Best-of-N in the overoptimization regime.
-
Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Concept Preference Optimization, a DPO-based loss for concept bottleneck models, improves task accuracy and noise robustness over binary cross-entropy.
-
A General Framework for Inference-time Scaling and Steering of Diffusion Models
FK steering uses interacting particle systems with intermediate reward potentials to steer diffusion models toward high-reward samples at inference time.
-
InfAlign: Inference-aware language model alignment
Reward calibration plus a procedure-specific reward transformation lets RLHF optimize inference-time (best-of-N / worst-of-N) win rates better than standard RLHF, IPO, BoND, and BoNBoN.
-
Time-Reversal Provides Unsupervised Feedback to LLMs
Time-reversed language models, which score queries given responses, improve best-of-N reranking, citation attribution, retrieval, and jailbreak filtering over forward perplexity baselines.
-
Discovering Algorithms with Computational Language Processing
A machine learning framework called CLP discovers, improves, and tailors algorithms by chaining computational tokens with MCTS and RL, with strong results on the Quadratic Assignment Problem and quantum search.
-
Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training
PRIOR reweights the next-token prediction loss in vision-language pretraining by 1 minus the probability assigned by a text-only reference LLM, and reports consistent benchmark improvements over standard NTP.
-
CoDe: Blockwise Control for Denoising Diffusion Models
CoDe applies blockwise best-of-N sampling during diffusion denoising, with Tweedie-based reward estimates, to align generated images to differentiable or non-differentiable rewards.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
A Survey of Reinforcement Learning For Economics
Reinforcement learning is presented as a natural, sample-based extension of dynamic programming for economic models.
Discussion (0). Continue with ORCID to comment.