DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
arXiv preprint arXiv:2007.09055 , year =
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. However, existing hyperparameter selection methods for offline RL break the offline assumption by evaluating policies corresponding to each hyperparameter setting in the environment. This online execution is often infeasible and hence undermines the main aim of offline RL. Therefore, in this work, we focus on \textit{offline hyperparameter selection}, i.e. methods for choosing the best policy from a set of many policies trained using different hyperparameters, given only logged data. Through large-scale empirical evaluation we show that: 1) offline RL algorithms are not robust to hyperparameter choices, 2) factors such as the offline RL algorithm and method for estimating Q values can have a big impact on hyperparameter selection, and 3) when we control those factors carefully, we can reliably rank policies across hyperparameter choices, and therefore choose policies which are close to the best policy in the set. Overall, our results present an optimistic view that offline hyperparameter selection is within reach, even in challenging tasks with pixel observations, high dimensional action spaces, and long horizon.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
A projected gradient descent algorithm for noisy inductive matrix completion achieves linear convergence and stable recovery at sample complexity governed by side-information dimension, extending to inexact side-information with optimal error degradation.
RLVR training teaches a 30B LLM to strategically explore a multi-buyer market and extract 70% of available surplus, outperforming frontier models up to 1T parameters in concurrent negotiation.
ADWM is an offline evaluation method that uses a policy-guided latent diffusion world model to rank LLM agent policies from pre-collected trajectories, reporting positive Spearman correlations on four benchmarks.
An adaptive UCB-based policy selection and fine-tuning strategy improves performance over standard O2O-RL baselines under interaction budgets.
A comprehensive benchmark study of offline imitation learning methods on multi-stage robot manipulation tasks identifies key sensitivities to algorithm design, data quality, and stopping criteria while releasing all datasets and code.
Develops Bishop-style constructive apparatus for geometric sets, integration, extremum theorems, selectors, differential inclusions, Markov chains, and densities in systems and control.
citing papers explorer
-
$\text{DT}^2$: Decision-Targeted Digital Twins
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
-
Sample-efficient inductive matrix completion with noise and inexact side-information
A projected gradient descent algorithm for noisy inductive matrix completion achieves linear convergence and stable recovery at sample complexity governed by side-information dimension, extending to inexact side-information with optimal error degradation.
-
Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations
RLVR training teaches a 30B LLM to strategically explore a multi-buyer market and extract 70% of available surplus, outperforming frontier models up to 1T parameters in concurrent negotiation.
-
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
ADWM is an offline evaluation method that uses a policy-guided latent diffusion world model to rank LLM agent policies from pre-collected trajectories, reporting positive Spearman correlations on four benchmarks.
-
Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
An adaptive UCB-based policy selection and fine-tuning strategy improves performance over standard O2O-RL baselines under interaction budgets.
-
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
A comprehensive benchmark study of offline imitation learning methods on multi-stage robot manipulation tasks identifies key sensitivities to algorithm design, data quality, and stopping criteria while releasing all datasets and code.
-
Some Essential Constructive Foundations for Systems and Control
Develops Bishop-style constructive apparatus for geometric sets, integration, extremum theorems, selectors, differential inclusions, Markov chains, and densities in systems and control.