REVIEW 2 cited by
RL-finetuning LLMs from on- and off-policy data with a single algorithm
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms.
Forward citations
Cited by 2 Pith papers
-
Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
ReMix cuts reinforcement finetuning rollout volume by 30x to 450x on math reasoning by mixing historical and on-policy data with a convex KL constraint and a mid-training switch to on-policy updates.
-
On a few pitfalls in KL divergence gradient estimation for RL
Differentiating KL estimates as losses gives biased or reversed KL gradients; the paper derives and tests unbiased sequence-level estimators.
Discussion (0). Sign in to comment.