A learned latent-space surrogate reward enables stable fine-tuning of one/two-step diffusion models with arbitrary, non-differentiable reward signals, outperforming policy-gradient baselines.
Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In recent years, fully differentiable rigid body physics simulators have been developed, which can be used to simulate a wide range of robotic systems. In the context of reinforcement learning for control, these simulators theoretically allow algorithms to be applied directly to analytic gradients of the reward function. However, to date, these gradients have proved extremely challenging to use, and are outclassed by algorithms using no gradient information at all. In this work we present a novel algorithm, cross entropy analytic policy gradients, that is able to leverage these gradients to outperform state of art deep reinforcement learning on a set of challenging nonlinear control problems.
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward
A learned latent-space surrogate reward enables stable fine-tuning of one/two-step diffusion models with arbitrary, non-differentiable reward signals, outperforming policy-gradient baselines.