RECORD
Off-Policy Interval Estimation with Lipschitz Value Iteration
Machine Learning · Submitted 29 Oct 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. When applied to high-stakes scenarios such as medical diagnosis or financial decision-making, it is crucial to provide provably correct upper and lower bounds of the expected reward, not just a classical single point estimate, to the end-users, as executing a poor policy can be very costly. In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting. The idea is to search for the maximum and minimum values of the expected reward among all the Lipschitz Q-functions that are consistent with the observations, which amounts to solving a constrained optimization problem on a Lipschitz function space. We go on to introduce a Lipschitz value iteration method to monotonically tighten the interval, which is simple yet efficient and provably convergent. We demonstrate the practical efficiency of our method on a range of benchmarks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. | background 1.00 | 0.05 | 0.03 | asserted 1.00 |
| s1 | When applied to high-stakes scenarios such as medical diagnosis or financial decision-making, it is crucial to provide provably correct upper and lower bounds of the expected reward, not just a classical single point estimate, to the end-users, as executing a poor policy can be very costly. | background 0.75 | 0.05 | 0.06 | asserted 1.00 |
| s2 | In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting. | contribution 1.00 | 0.93 | 0.52 | asserted 1.00 |
| s3 | The idea is to search for the maximum and minimum values of the expected reward among all the Lipschitz Q-functions that are consistent with the observations, which amounts to solving a constrained optimization problem on a Lipschitz function space. | method 0.97 | 0.77 | 0.10 | asserted 1.00 |
| s4 | We go on to introduce a Lipschitz value iteration method to monotonically tighten the interval, which is simple yet efficient and provably convergent. | method 0.86 | 0.95 | 0.22 | asserted 1.00 |
| s5 | We demonstrate the practical efficiency of our method on a range of benchmarks. | result 0.80 | 0.94 | 0.15 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. When applied to high-stakes scenarios such as medical diagnosis or financial decision-making, it is crucial to provide provably correct upper and lower bounds of the expected reward, not just a classical single point estimate, to the end-users, as executing a poor policy can be very costly. In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting. The idea is to search for the maximum and minimum values of the expected reward among all the Lipschitz Q-functions that are consistent with the observations, which amounts to solving a constrained optimization problem on a Lipschitz function space. We go on to introduce a Lipschitz value iteration method to monotonically tighten the interval, which is simple yet efficient and provably convergent. We demonstrate the practical efficiency of our method on a range of benchmarks.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 1 sentence without a confident label
Cites as background 4 papers
-
There are typically two types of OPE methods, importance sampling (IS) based methods [ref] and value function or model based methods [ref] Another line of work combines these two methods and yields a doubly-robust estimator for off-policy evaluation [ref] .
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning · Introduction · 1.00
-
Another related set of works are PAC-RL [ref] , which mainly focus on the regret bound or sample complexity of the Q-learning exploration.
Policy Certificates: Towards Accountable Reinforcement Learning · Introduction · 0.93
-
Another related set of works are PAC-RL [ref] , which mainly focus on the regret bound or sample complexity of the Q-learning exploration.
Efficient Model-free Reinforcement Learning in Metric Spaces · Introduction · 0.93
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.