CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

Aishwarya Mandyam; Barbara E. Engelhardt; Jenna Wiens; Jiayu Yao; Shengpu Tang

REVIEW 2 major objections 6 minor 21 references

CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that imperfect counterfactual annotations should enter a doubly robust estimator only through its reward model: that estimator stays unbiased under biased annotations, and it performs best when the reward model is…

desk verdict The bias analysis and practical recommendation are solid and new; the variance theorem has a real algebra error and the sepsis experiments run outside the theory's coverage assumption. read the letter →

arxiv 2412.08052 v2 pith:IUQZOT57 submitted 2024-12-11 cs.LG stat.ML

Aishwarya Mandyam , Shengpu Tang , Jiayu Yao , Jenna Wiens , Barbara E. Engelhardt This is my paper · ORCID

classification cs.LGstat.ML

keywords off-policyevaluationcontextualbanditscounterfactualannotationsdoublyrobustestimationimportancesamplingdirectmethodannotationbiashealthcaredecision-making

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy evaluation estimates how a new decision policy would perform before it is deployed, using only data logged under a different behavior policy, and its accuracy is limited by the coverage of that data. Prior work augmented the logged data with expert counterfactual annotations, predicted rewards for actions that were not taken, but those annotations are often imperfect and can make estimates worse than using none at all. This paper asks where inside a doubly robust estimator such annotations should be placed, and proves an asymmetry: when annotations are biased, the estimator that uses them only to fit the reward model (DM+-IS) remains unbiased ($E[\hat{V}^{\mathrm{DM}+\text{-IS}}] = v(\pi_e)$), while estimators that also feed annotations into the importance-sampling term inherit an explicit bias. The paper argues this is the realistic operating regime, since reward models are typically misspecified and annotations imperfect, and in that regime DM+-IS has the lowest estimation error across three simulated environments including a sepsis treatment setting. If the paper is right, practitioners get a simple default for high-stakes deployments: when annotation and model quality are uncertain, use imperfect annotations only in the reward model.

What carries the argument

The central mechanism is the placement of the counterfactual-annotated dataset $D_+$ inside the doubly robust decomposition $\hat{V} = (1/N)\sum_i [\hat{R}(s_i,\pi_e) + \rho_{s_i}(a_i)(r_i - \hat{R}(s_i,a_i))]$. The three estimators are the three nontrivial placements: DM+-IS fits the reward model $\hat{R}_+$ on $D_+$ and keeps the IS correction on factual rewards; DM-IS+ keeps the standard reward model and pushes annotations into the IS term through the augmented behavior policy $\pi_b^+(a|s) = \bar{W}(a|s,a)\pi_b(a|s) + \sum_{\tilde{a}\neq a} \bar{W}(a|s,\tilde{a})\pi_b(\tilde{a}|s)$, with per-sample weights that sum to one; DM+-IS+ does both. The augmented policy and weights preserve the factual context distribution when annotations are appended, and the naive alternative that skips them distorts the context distribution and is arbitrarily biased (Appendix D). The load-bearing identity is the telescoping in the proof of Proposition 1: $E_{a\sim\pi_b}[\rho_s(a)(r - \hat{R}_+(s,a))] = E_{a\sim\pi_e}[\bar{R}(s,a) - \hat{R}_+(s,a)]$, which exactly cancels the $E[\hat{R}_+(s,\pi_e)]$ term and leaves $v(\pi_e)$ with no dependence on the annotation bias $\epsilon_G$.

What would settle it

Run the DM+-IS estimator in a two-dataset regime where Assumptions 2 and 4 provably hold, with annotations biased by a known amount $\epsilon_G$; Proposition 1 predicts the estimate's bias is exactly zero for every $\epsilon_G$, so a statistically significant non-zero bias would refute it. A second practical check is to repeat the experiment in the single-dataset regime (reward model and OPE estimate on the same behavior data): if a systematic bias appears that grows with $\epsilon_G$, the two-dataset independence assumption is load-bearing and the guarantee does not transfer to that regime.

Watch

Extended reading notes

Core claim

Under Assumption 2 (annotations with expectation $\bar{R}(s,a) + \epsilon_G(s,a)$) and common support between behavior and target policies, Proposition 1 establishes that $E[\hat{V}^{\mathrm{DM}+\text{-IS}}] = v(\pi_e)$: the estimator that fits its reward model on the annotation-augmented dataset and applies standard importance sampling to factual rewards is unbiased, no matter how biased the annotations are. Theorem 2 shows the two estimators that route annotations through the augmented importance-sampling term, DM-IS+ and DM+-IS+, both carry the same bias term, $E_{s\sim d_0, a\sim\pi_e}[(1 - \bar{W}(a|s,a)\pi_b(a|s)/\pi_b^+(a|s))\epsilon_G(s,a)]$, which is the bias term that already affected the earlier IS+ estimator. The mechanism is a cancellation: the reward model's annotation-induced bias is exactly offset by the inverse-propensity correction $\rho(r_i - \hat{R}_+(s_i,a_i))$ evaluated on factual rewards, which remain unbiased. Empirically, across a two-context bandit, a Heartsteps step-count environment, and a sepsis treatment simulator with misspecified reward models, DM+-IS achieves the lowest RMSE across all tested magnitudes of annotation bias, and the paper concludes that any further use of imperfect annotations beyond the reward model can worsen the estimate. A lookup-table guide and a sensitivity analysis in the sepsis setting support the recommendation that DM+-IS is the safe default when annotation and reward-model quality are unknown.

Load-bearing premise

The unbiasedness proof assumes the reward model $\hat{R}_+$ is fitted on a behavior dataset separate from the one used to compute the OPE estimate; if a practitioner fits and evaluates on the same data, the cancellation that removes the annotation bias can break, and the paper does not analyze that single-dataset regime.

Editorial extensions

If this is right

When counterfactual annotations are available, they should be used to fit the reward model only; feeding them into the importance-sampling term adds a bias proportional to the annotation bias.
If annotation quality and reward-model misspecification are unknown, the empirical comparison supports DM+-IS as the default estimator, with sepsis-setting estimates within a small margin of the best method across all annotation qualities.
The naive strategy of plugging the annotated dataset directly into a standard doubly robust estimator is unsafe because it silently changes the context distribution and can be arbitrarily biased.
With perfect annotations and equal sample weights, IS+, DM-IS+, and DM+-IS+ coincide, so the choice among those three matters only when annotations are imperfect or weights are unequal.
Under a misspecified reward model, pure DM and DM+ degrade while all doubly robust variants keep the protection of their IS term, and DM+-IS gets that protection without the annotation-bias cost of the IS+ variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A general design rule suggested by Proposition 1, though not stated in the paper: in a doubly robust estimator, imperfect auxiliary data should enter through the model term that the unbiased factual correction can cancel, never through the reweighted term that inherits the auxiliary data's bias.
The two-dataset assumption implies a practical requirement the paper leaves implicit: practitioners need two separate behavior datasets or sample splitting to realize the unbiasedness guarantee, since the single-dataset regime is not analyzed.
A testable extension the paper does not run: with a fixed annotation budget, the theory suggests annotations should be spent where they most improve the reward model (under-covered contexts and actions), because placement does not affect bias, only the quality of the reward model.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on CANDOR. The practical claim is worth taking seriously: under a misspecified reward model and biased counterfactual annotations, you should put the annotations in the DM part of a DR estimator, not the IS part. Proposition 1 and Theorem 2 are clean — the DR cancellation works, and the bias term for the IS+ variants (Eq. 4) matches what you'd expect. The three-estimator family is a genuine extension of the authors' own IS+ work, and the self-citation is not a problem; the unbiasedness result stands on the DR cancellation, not on IS+ being correct. The experiments are decent: three environments, well-specified vs misspecified reward models, annotation bias and variance swept, bootstrapped errors. The Figure 4 lookup table is a useful distillation for practitioners.

Now the soft spots, in order of importance.

First, Theorem 3 does not check out. In the appendix proof, going from sum_a pi_e(a)^2 (1/pi_b(a)) V[R_hat+] minus sum_a pi_e(a)^2 V[R_hat+] to the stated final term E_{a~pi_b}[(rho^2 - 1/pi_b)Delta] requires sum_a pi_e(a)^2 V = sum_a V, which fails in general. The variance decomposition as stated is not established. This does not sink the main recommendation, which rides on the bias results, but the theorem needs to be fixed or weakened.

Second, the sepsis experiments violate Assumption 4 for three of the six behavior policies. pi_b1 through pi_b3 put zero mass on actions that pi_e supports, so rho is undefined, the IS correction does not fire, and DM+-IS carries an unanalyzed bias of sum_{a:pi_b=0} pi_e(a)(E[R_hat+(a)] - R_bar(a)). The paper averages these runs into the across-all-datasets claim without noting that the regime falls outside the theorem. I checked the appendix; the stress-test concern holds. This is the exact regime — insufficient coverage — that motivates counterfactual annotations, so the gap matters.

Third, smaller items: the abstract claims real-world EHR data, but all three environments are simulators; that sentence should be corrected. The theory assumes separate datasets for the OPE estimate and the reward model fit; the single-dataset regime is not analyzed, which is fine as a stated assumption but should be flagged as a practical gap. Code is promised on acceptance, not shipped.

None of this kills the paper. The bias analysis and practical guidance are solid, and I'd trust the empirical ranking more than the variance theorem. It deserves a serious referee; expect heavy revision, mainly on Theorem 3 and the coverage gap. I'd cite it for the DM+-IS recommendation, and I'd take it to a reading group to argue about the coverage issue.

Referee Report

2 major / 6 minor

Summary. The paper studies off-policy evaluation (OPE) in contextual bandits when counterfactual annotations are available but possibly imperfect. It proposes three doubly robust inspired estimators that incorporate annotations in different places: DM+-IS uses annotations only in the reward model, DM-IS+ uses them only in the importance sampling part, and DM+-IS+ uses them in both. The main theoretical result is Proposition 1, which states that under biased annotations and common support, DM+-IS is unbiased, while Theorem 2 shows that DM-IS+ and DM+-IS+ carry an annotation-bias term. Theorem 3 gives a variance decomposition for DM+-IS. Experiments on a two-context bandit, Heartsteps, and a Sepsis treatment simulator compare the estimators under misspecified reward models and imperfect annotations, leading the authors to recommend DM+-IS as the most robust choice. The appendix contains proofs, additional experiments, and a discussion of weighted versus unweighted reward models.

Significance. If the main claims hold, the paper provides clear practical guidance: when counterfactual annotations are imperfect, they should be used only in the direct-method component of a doubly robust estimator. Proposition 1 is a clean and useful result, and the paper is a rare OPE contribution that both proves a property and ships code and detailed empirical comparisons. However, the proof of Theorem 3 contains an algebraic error that invalidates the stated variance formula, and the Sepsis experiments violate Assumption 4 on which Proposition 1 depends. These issues affect the formal and empirical support for the 'most robust across all datasets' claim, so the manuscript needs revision before the recommendations can be accepted as stated.

major comments (2)

[Appendix H.1.2 (Theorem 3)] The displayed derivation does not establish the stated variance formula. After the unbiased-DM substitution, the derivation reaches an expression of the form E_s[ Σ_a πe(a|s)^2 (1/πb(a|s) - 1) V_{D̂R}[R̂+(s,a)] ]. The next displayed line rewrites this as E_s E_{a∼πb}[(ρs(a)^2 - 1/πb(a|s)) Δ_{R̂+}(s,a)], which equals Σ_a (πe(a|s)^2/πb(a|s) - 1) Δ_{R̂+}(s,a), not the previous expression; the factor πe(a|s)^2 on the second term has been replaced by 1. The valid simplification would keep E_s[ E_{a∼πb}[ρs(a)^2 Δ_{R̂+}(s,a)] - Σ_a πe(a|s)^2 Δ_{R̂+}(s,a) ]. Since Theorem 3 is the variance result listed in Table 3 and is invoked in Section 6.1 to interpret the RMSE behavior of DM+-IS, the theorem must be corrected or explicitly qualified. The same passage also contains an apparent algebraic slip in the biased-DM case, where a term 2vπe(s)^2 R̃(s,πe) appears instead of 2vπe(s)R̃(s,πe).
[Appendix B.3 and Section 6.2] The Sepsis experiments include behavior policies that violate Assumption 4, which is required by Proposition 1. For example, πb1 = [0.1,0.1,0.4,0.3,0.1,0,0,0] and πb2 = [0.1,0.1,0.4,0.2,0.1,0.1,0,0] place zero mass on actions that the target policy πe = [0.3,0.2,0,0,0.2,0.1,0.1,0.1] supports. The proof of Proposition 1 uses the identity E_{a∼πb}[ρ(a)f(a)] = E_{a∼πe}[f(a)], which fails when πb does not cover πe. For unsupported actions, the estimator carries an additional, unmodeled bias of order Σ_{a:πb(a|s)=0} πe(a|s)(R̄(s,a) - E[R̂+(s,a)]), which is not analyzed. Nevertheless, Figure 3c averages RMSE over these behavior policies and Section 6.2 concludes that DM+-IS is most robust 'across all datasets.' Please either restrict the Sepsis evaluation to behavior policies satisfying Assumption 4 or extend the theoretical analysis to partial coverage and report the corresponding bias term.

minor comments (6)

[Section 4, Theorem 3 display] The displayed variance formula has a bracket mismatch and an unclosed expectation expression; please correct the typesetting so that the formula is parseable.
[Appendix H.1.1, Propositions 12 and 13] Propositions 12 and 13 are stated under Assumptions 1 and 5, but their proofs use the standard IS ratio and rely on Assumption 4 (common support without annotations). Please align the stated assumptions with the proofs.
[Appendix C and Acknowledgements] The reported total compute time differs between the acknowledgements (approximately 30 hours) and Appendix C (approximately 100 hours); please reconcile these numbers.
[Appendix I, Corollary 18] The corollary title lists DM+-IS, but the statement and proof concern DM-IS+ and DM+-IS+; please correct the title.
[Appendix B.1] Please clarify whether the dataset used to fit R̂ and R̂+ is independent of the dataset used to form the OPE estimate, as required by Section 4; the phrase 'the same dataset size and distribution is used to learn a reward function estimate' is ambiguous.
[Section 7] In the paragraph on unknown reward model and annotation quality, 'annotation equality' should read 'annotation quality.'

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: DM+-IS unbiasedness is a standard DR cancellation; self-citations to prior IS+ work are not load-bearing.

full rationale

The paper's central theoretical claim (Proposition 1) is not circular. The proof in Appendix H.1.1 shows that E[V_hat^{DM+-IS}] = v(pi_e) by a standard doubly robust cancellation: under Assumption 4, the IS correction term E_{a~pi_b}[rho(s,a)(Rbar(s,a) - E[R_hat+(s,a)])] is converted to E_{a~pi_e}[Rbar(s,a) - E[R_hat+(s,a)]], which cancels the DM term E_{a~pi_e}[E[R_hat+(s,a)]]. This cancellation holds for any reward model estimate, including one whose expectation is biased by imperfect annotations, because the reward model is fitted on an independent dataset, as explicitly assumed in Section 4. Thus the unbiasedness result follows from the estimator's algebraic structure, not from assuming the conclusion. Theorem 2's bias expression for DM-IS+ and DM+-IS+ is derived in Appendix H.2.1; the observation that the bias term is identical to that of IS+ is a comparison, not a load-bearing citation. The self-citations to Tang and Wiens supply the IS+ estimator and the augmented behavior policy pi_b+ construction, but the paper re-derives the needed identities and the recommended estimator, DM+-IS, does not rely on IS+ being correct. There are no fitted parameters renamed as predictions, no imported uniqueness theorems, and no ansatz smuggled in solely by citation. The Sepsis support violation is an internal-validity concern about the empirical claim, not a circularity of the derivation.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

No free parameters are fitted to data; the only hand-chosen design quantity is the annotation weight distribution. All assumptions are standard domain assumptions for OPE with an augmented dataset, and no new physical or formal entities are introduced.

free parameters (1)

Annotation weights w_i,a = equal weights (1/|A|) in all experiments
IS+, DM-IS+, and DM+-IS+ require user-defined weights to preserve the context distribution in the augmented dataset. The theory allows general weight distributions, but every experiment uses equal weights, so robustness to weight choice is untested.

assumptions (6)

domain assumption Behavior policy pi_b and target policy pi_e are known, so propensity ratios rho are known.
Stated in Section 2.1; all unbiasedness proofs rely on known rho.
domain assumption The reward-model training dataset D_Rhat is independent of the OPE evaluation dataset D0.
Section 4 assumes two behavior datasets; without this independence the DR cancellation in DM+-IS is not assured.
domain assumption Common support: pi_e(a|s) > 0 implies pi_b(a|s) > 0 (Assumption 4) or pi_b_plus(a|s) > 0 (Assumption 5).
Used in Propositions 1, 12, 14, 16 and Theorems 2 and 3.
domain assumption Annotations are drawn from a distribution G(s,a) with expectation R_bar + eps_G (Assumption 2) and variance sigma_R^2 + Delta_G (Assumption 3).
This is the model of annotation imperfection used in all theory; real expert bias may be correlated with the factual sample rather than a fixed per-context-action offset.
domain assumption Misspecified reward model: for the function class F, min_f ||f - R*||_D > 0 (Assumption 6).
Defines the misspecification setting that drives the experimental design; no proof of this bound is needed.
domain assumption Finite discrete context and action spaces with rewards of finite variance.
The bandit definition in Section 2 and all variance derivations assume finite sums and finite moments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation." pith.science (2026). https://pith.science/paper/IUQZOT57

@misc{pith2026241208052,
  author       = {Pith},
  title        = {Pith review of: CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUQZOT57}},
  note         = {Machine review of arXiv:2412.08052}
}

read the original abstract

Off-policy evaluation (OPE) is critical for applying contextual bandit algorithms to high-stakes decision-making settings such as healthcare, where new treatment policies must be evaluated prior to deployment. Unfortunately, OPE techniques are inherently limited by the breadth of the available data, which may not be sufficient to evaluate the performance of a new policy. Recent work attempts to improve dataset coverage by adding expert-annotated counterfactual samples. However, such annotations are often imperfect and can lead to worse estimator performance than using no annotations at all. To better leverage imperfect annotations, we propose a family of OPE estimators grounded in the doubly robust (DR) framework, which combines importance sampling (IS) with a reward model (direct method, DM) for better statistical guarantees. We study three ways of incorporating counterfactual annotations. Under mild assumptions, we prove that using annotations within just the DM component yields the most desirable theoretical results. Experiments on multiple healthcare tasks, including real-world electronic health records (EHR) data, show that this strategy is most robust under misspecified reward models and inaccurate annotations. By addressing the challenges posed by imperfect annotations, this work broadens the applicability of OPE methods and facilitates safer deployment of decision-making policies in healthcare.

Figures

Figures reproduced from arXiv: 2412.08052 by the authors.

**Figure 2.** Heatmap of mean RMSE with a well-specified reward model in the 2-context bandit setting (lower RMSE is represented lighter): The bias of counterfactual annotations has a larger impact on RMSE than the variance. The x, y-axis represents the variance (∆G) and the bias (ϵG) of the annotations, respectively. The RMSE hardly varies across the x-axis but increases proportionally to the magnitude of the annotation bias. Th… view at source ↗

**Figure 3.** Heatmaps of mean RMSE with a misspecified reward model and imperfect annotations.(lower RMSE is represented lighter). The x-axis represents annotation bias, ϵG. Across all datasets, DM+ -IS performs either better than all baselines, or comparably to the best-performing baseline. Among all methods that use counterfactual annotations, DM+ -IS is most robust to biased annotations and a misspecified reward model. In com… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Lookup table capturing the practical considerations when choosing an OPE estimator. The most critical factors include whether the reward model is misspecified or not and the quality of the annotations. If the reward model and annotation equality is known a priori, the …

**Figure 5.** Figure 5: Exploring the consequences of always using DM+ -IS in the sepsis environment. In the vast majority of cases, the reward model or annotation quality is unknown. Regardless, we choose DM+ -IS to learn an OPE estimate and compare it to the best performing OPE method. We f…

**Figure 6.** Figure 6: Heatmap of mean and standard deviation of RMSE with a well-specified reward model (lower mean/standard deviation is represented lighter): The x, y-axis represents the variance (∆G) and the bias (ϵG) of the annotations, respectively. In a well-specified reward setting, …

**Figure 7.** Figure 7: Heatmap of mean and standard deviation of RMSE with a misspecified reward model (lower mean/standard deviation is represented lighter): The x, y-axis represents the variance (∆G) and the bias (ϵG) of the annotations, respectively. In a misspecified reward setting, DM a…

**Figure 8.** Figure 8: Here we compare the consequences of choosing the other two proposed estimators ( [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages

[1]

Beygelzimer and J

A. Beygelzimer and J. Langford. The offset tree for learning with partial labels. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, page 129–138, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584959. doi:10.1145/1557019.1557040. URL https://doi.org/10.1145/1557019.1557040

arXiv 2009
[2]

Dudik, J

M. Dudik, J. Langford, and L. Li. Doubly robust policy evaluation and learning, 2011

work page 2011
[3]

Dudík, D

M. Dudík, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. Statistical Science, 29 0 (4), Nov. 2014. ISSN 0883-4237. doi:10.1214/14-sts500. URL http://dx.doi.org/10.1214/14-STS500

doi:10.1214/14-sts500 2014
[4]

Farajtabar, Y

M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation, 2018

work page 2018
[5]

Q($\lambda$) with Off-Policy Corrections

A. Harutyunyan, M. G. Bellemare, T. Stepleton, and R. Munos. Q( ) with off-policy corrections, 2016. URL https://arxiv.org/abs/1602.04951

work page Pith review arXiv 2016
[6]

D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47 0 (260): 0 663--685, 1952. ISSN 01621459. URL http://www.jstor.org/stable/2280784

arXiv 1952
[7]

Jiang and L

N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning, 2016

work page 2016
[8]

A. S. Lan and R. Baraniuk. A contextual bandits framework for personalized learning action selection. In Educational Data Mining, 2016. URL https://api.semanticscholar.org/CorpusID:15394680

work page 2016

Show all 21 references

[9]

H. M. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints, 2019. URL https://arxiv.org/abs/1903.08738

2019 arXiv
[10]

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, WWW ’10. ACM, Apr. 2010. doi:10.1145/1772690.1772758. URL http://dx.doi.org/10.1145/...

2010
[11]

V. Li. Pre/dicta takes a radically different approach to predictive analytics than others, 05 2023

2023
[12]

Liu and S

S. Liu and S. Zhang. Efficient policy evaluation with offline data informed behavior policy design, 2024. URL https://arxiv.org/abs/2301.13734

2024 arXiv
[13]

Mandyam, M

A. Mandyam, M. Jörke, W. Denton, B. E. Engelhardt, and E. Brunskill. Adaptive interventions with user-defined goals for health behavior change, 2024

2024
[14]

Oberst and D

M. Oberst and D. Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models, 2019. URL https://arxiv.org/abs/1905.05824

2019 arXiv
[15]

Precup, R

D. Precup, R. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, 06 2000

2000
[16]

R. S. Sutton and A. G. Barto. Reinforcement learning: an introduction. Adaptive computation and machine learning series. The MIT Press, second edition edition, 2018. ISBN 9780262039246

2018
[17]

Tang and J

S. Tang and J. Wiens. Counterfactual-augmented importance sampling for semi-offline policy evaluation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=dsH244r9fA

2023
[18]

P. S. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning, 2016

2016
[19]

van Seijen, H

H. van Seijen, H. V. Hasselt, S. Whiteson, and M. A. Wiering. A theoretical and empirical analysis of expected sarsa. 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 177--184, 2009. URL https://api.semanticscholar.org/CorpusID:6230754

2009
[20]

Voloshin, H

C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of off-policy policy evaluation for reinforcement learning, 2021

2021
[21]

J. Yao, E. Brunskill, W. Pan, S. Murphy, and F. Doshi-Velez. Power constrained bandits. In Proceedings of the 6th Machine Learning for Healthcare Conference, pages 209--259, 2021

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Beygelzimer and J

A. Beygelzimer and J. Langford. The offset tree for learning with partial labels. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, page 129–138, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584959. doi:10.1145/1557019.1557040. URL https://doi.org/10.1145/1557019.1557040

arXiv 2009

[2] [2]

Dudik, J

M. Dudik, J. Langford, and L. Li. Doubly robust policy evaluation and learning, 2011

work page 2011

[3] [3]

Dudík, D

M. Dudík, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. Statistical Science, 29 0 (4), Nov. 2014. ISSN 0883-4237. doi:10.1214/14-sts500. URL http://dx.doi.org/10.1214/14-STS500

doi:10.1214/14-sts500 2014

[4] [4]

Farajtabar, Y

M. Farajtabar, Y. Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation, 2018

work page 2018

[5] [5]

Q($\lambda$) with Off-Policy Corrections

A. Harutyunyan, M. G. Bellemare, T. Stepleton, and R. Munos. Q( ) with off-policy corrections, 2016. URL https://arxiv.org/abs/1602.04951

work page Pith review arXiv 2016

[6] [6]

D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47 0 (260): 0 663--685, 1952. ISSN 01621459. URL http://www.jstor.org/stable/2280784

arXiv 1952

[7] [7]

Jiang and L

N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning, 2016

work page 2016

[8] [8]

A. S. Lan and R. Baraniuk. A contextual bandits framework for personalized learning action selection. In Educational Data Mining, 2016. URL https://api.semanticscholar.org/CorpusID:15394680

work page 2016