REVIEW 5 cited by
Hyperparameter Selection for Offline Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. However, existing hyperparameter selection methods for offline RL break the offline assumption by evaluating policies corresponding to each hyperparameter setting in the environment. This online execution is often infeasible and hence undermines the main aim of offline RL. Therefore, in this work, we focus on \textit{offline hyperparameter selection}, i.e. methods for choosing the best policy from a set of many policies trained using different hyperparameters, given only logged data. Through large-scale empirical evaluation we show that: 1) offline RL algorithms are not robust to hyperparameter choices, 2) factors such as the offline RL algorithm and method for estimating Q values can have a big impact on hyperparameter selection, and 3) when we control those factors carefully, we can reliably rank policies across hyperparameter choices, and therefore choose policies which are close to the best policy in the set. Overall, our results present an optimistic view that offline hyperparameter selection is within reach, even in challenging tasks with pixel observations, high dimensional action spaces, and long horizon.
Forward citations
Cited by 5 Pith papers
-
Active Offline-to-Online Reinforcement Learning
UCB selection driven by local linear forecasts of fine-tuning returns allocates limited online interactions more effectively than single-policy or equal-budget O2O-RL baselines across continuous-control tasks.
-
Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations
RLVR training teaches a 30B LLM to strategically explore a multi-buyer market and extract 70% of available surplus, outperforming frontier models up to 1T parameters in concurrent negotiation.
-
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
ADWM learns a latent diffusion world model with per-transition independent denoising and policy-conditioned guidance to enable accurate offline evaluation of LLM agent policies.
-
Accelerating Detailed Routing Convergence through Offline Reinforcement Learning
A conservative Q-learning model that selects per-iteration routing cost weights speeds up OpenROAD detailed routing on ISPD19 benchmarks by 1.56x on average without worsening DRVs.
-
Learning to Evaluate Autonomous Behaviour in Human-Robot Interaction
A neural behavior classifier trained on teleoperated joint trajectories is proposed and tested as an offline meta-evaluator for imitation learning policies in human-robot interaction.
Discussion (0). Sign in to comment.