Using imperfect counterfactual annotations only in the reward model part of a doubly robust estimator is the theoretically and empirically safest way to incorporate them.
Q($\lambda$) with Off-Policy Corrections
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD($\lambda$). We illustrate this theoretical relationship empirically on a continuous-state control task.
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation
Using imperfect counterfactual annotations only in the reward model part of a doubly robust estimator is the theoretically and empirically safest way to incorporate them.