REVIEW 6 cited by
Value Augmented Sampling for Language Model Alignment and Personalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Aligning Large Language Models (LLMs) to cater to different human preferences, learning new skills, and unlearning harmful behavior is an important problem. Search-based methods, such as Best-of-N or Monte-Carlo Tree Search, are performant, but impractical for LLM adaptation due to their high inference cost. On the other hand, using Reinforcement Learning (RL) for adaptation is computationally efficient, but performs worse due to the optimization challenges in co-training the value function and the policy. We present a new framework for reward optimization, Value Augmented Sampling (VAS), that can maximize different reward functions using data sampled from only the initial, frozen LLM. VAS solves for the optimal reward-maximizing policy without co-training the policy and the value function, making the optimization stable, outperforming established baselines, such as PPO and DPO, on standard benchmarks, and achieving comparable results to Best-of-128 with lower inference cost. Unlike existing RL methods that require changing the weights of the LLM, VAS does not require access to the weights of the pre-trained LLM. Thus, it can even adapt LLMs (e.g., ChatGPT), which are available only as APIs. In addition, our algorithm unlocks the new capability of composing several rewards and controlling the extent of each one during deployment time, paving the road ahead for the future of aligned, personalized LLMs.
Forward citations
Cited by 6 Pith papers
-
Selective Safety Steering via Value-Filtered Decoding
Value-filtered decoding steers LLM outputs for safety at decoding time using a value criterion with an explicit bound on false interventions controlled by one threshold hyperparameter.
-
Controlling Multimodal LLMs via Reward-guided Decoding
MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHA...
-
PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization
PICACO optimizes a meta-instruction by maximizing total correlation between intended values and LLM responses, and reports consistent though modest gains across five value sets and three target models.
-
From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment
SP-PRM trains process reward models from outcome reward models via score and preference consistency, improving reward-guided search on dialogue, summarization, and reasoning tasks.
-
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Dualizing Safe RLHF yields a one-dimensional convex calibration of λ that defines a drop-in safety-aware reward for Best-of-N and token-level inference-time decoders.
-
Test-time reward-guided alignment of language models by importance sampling on pre-logit space
AISP is a training-free decode-time alignment method: Gaussian-perturb LLM pre-logits, score sampled responses with a reward model, and iteratively shift the perturbation mean by reward-weighted importance sampling.
Discussion (0). Sign in to comment.