A General Framework for Off-Policy Learning with Partially-Observed Reward

Kosuke Kawakami; Masahiro Asami; Rikiya Takehi; Yuta Saito

REVIEW 3 major objections 4 minor 31 references

A General Framework for Off-Policy Learning with Partially-Observed Reward

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HyPeR learns policies from partially observed target rewards by combining them with densely observed secondary rewards in a policy-gradient estimator that remains unbiased while reducing variance.

desk verdict Solid hybrid DR estimator for partial rewards, but the proof has a fixable slip and the paper overclaims applicability when observation depends on actions. read the letter →

arxiv 2506.14439 v1 pith:5FHTK76A submitted 2025-06-17 cs.LG

Rikiya Takehi , Masahiro Asami , Kosuke Kawakami , Yuta Saito This is my paper · ORCID

classification cs.LG

keywords off-policylearningcontextualbanditspolicygradientpartiallyobservedrewardssecondarydoublyrobustestimationvariancereductionmissingatrandom

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy learning in contextual bandits normally assumes every logged action comes with its reward, but in practice target rewards are often missing, delayed, or censored. This paper studies the setting where a densely observed secondary reward—clicks, dwell time, short-term medical indicators—is available for every interaction, and asks how to maximize the expected target reward without discarding the missing-reward data or trusting the secondary reward as a biased surrogate. Its answer is HyPeR, a policy-gradient estimator that keeps the unbiasedness of inverse-propensity and doubly robust estimation while adding a correction built from secondary rewards. The paper proves HyPeR is unbiased under the usual full-support condition and proves its variance is lower than reward-only doubly robust estimation whenever the secondary reward improves prediction of the target reward; it also shows that deliberately tuning the mixture weight away from the objective weight improves finite-sample policy value. If correct, this gives a practical recipe for learning from sparse conversion or rating signals in recommendation, advertising, and clinical settings where dense proxies are available.

What carries the argument

The load-bearing object is the three-term target-reward gradient estimator in Eq. (10), combined with the mixture rule Eq. (11). The first term is a regression baseline that uses every context through the action-marginal expectation of $\hat{q}$; the second term is an importance-weighted adjustment from $\hat{q}(x,a)$ to $\hat{q}(x,a,s)$ that is available for every logged tuple because the secondary reward $s$ is always observed; the third term corrects the residual $r_i-\hat{q}(x_i,a_i,s_i)$ and is weighted by $o_i/p(o_i|x_i)$. In expectation the second and third terms telescope: the model terms cancel and the residual leaves the true mean reward, so the estimator recovers $\nabla_\theta V^r(\pi_\theta)$ without needing the target reward on more than the observed fraction of tuples. Mixing this with a doubly robust secondary-reward gradient makes $\gamma$ an explicit bias-variance dial: lower $\gamma$ trusts the target objective, higher $\gamma$ leans on the denser secondary signal.

What would settle it

Simulate a bandit with a large sample in which the observation probability is $p(o=1 \mid x,a)$ and grows with the expected target reward, compute the true policy gradient from full-information data, and compare it with the gradient returned by Eq. (10) using correctly estimated reward models; if the gap between the two does not shrink as $n$ grows, the missing-at-random assumption is violated in a way that matters.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the dilemma between a sparse, accurate target reward and a dense, possibly misaligned secondary reward need not be resolved by choosing one; a single unbiased gradient estimator can use both. The target-reward part of the estimator is the three-term expression in Eq. (10): a doubly robust baseline using the action-marginal expectation of $\hat{q}$, an importance-weighted correction that moves from $\hat{q}(x,a)$ to $\hat{q}(x,a,s)$, and an observation-weighted residual correction $(o_i/p(o_i|x_i)) w(x_i,a_i)(r_i-\hat{q}(x_i,a_i,s_i)) g_\theta(x_i,a_i)$. The full HyPeR gradient is the mixture $(1-\gamma)\hat{\nabla}_\theta V^r + \gamma \hat{\nabla}_\theta V^s$. Theorem 1 shows the target-gradient estimator is unbiased against $\nabla_\theta V^r(\pi_\theta)$ under full support; Theorem 2 shows it has lower variance than r-DR exactly when $\hat{q}(x,a,s)$ predicts the target reward better than $\hat{q}(x,a)$. The paper further claims that using a weight $\gamma \neq \beta$ in the mixture can improve the combined policy value in finite samples, and that a bootstrap-based validation procedure can select $\gamma$ from logged data alone.

Load-bearing premise

The entire unbiasedness argument rests on the assumption that, conditioned on the context $x$, whether the target reward is observed is independent of the action taken and of the reward value; if missingness is driven by the action itself, the guarantee collapses.

Editorial extensions

If this is right

Systems that currently discard logged interactions with missing target rewards can instead feed every tuple into the gradient estimate, making learning feasible at low target-reward observation rates.
Platforms with delayed conversions or censored outcomes can train immediately on clicks, dwell time, or short-term medical indicators while keeping an unbiased target-reward gradient.
For a combined objective, the mixture weight $\gamma$ should be tuned on validation data rather than fixed to the objective weight $\beta$, because the tuned weight can beat the unbiased choice in finite samples.
The variance-reduction theorem makes the benefit of HyPeR proportional to how informative the secondary rewards are: the better $s$ predicts $r$, the larger the gap between HyPeR and reward-only doubly robust estimation.
The method still uses importance weighting, so the logging policy must keep full support over actions; the variance reduction does not remove the need for exploration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct but untested extension is to carry the same residual-correction structure into off-policy evaluation and offline reinforcement learning, where partially observed rewards inflate variance in value estimates just as they do in policy gradients.
In e-commerce, conversion observation is often triggered by clicks—an action-dependent event—so the paper's missing-at-random assumption will frequently fail; a practical deployment should either condition on the click event or quantify worst-case bias as a function of dependence between $o$ and $a$.
One testable generalization is a context-dependent mixture weight $\gamma(x)$, leaning on secondary rewards more in contexts where the target reward is rarely observed and less where it is dense.
The bootstrap re-sampling trick for weight selection is a general recipe for hybrid estimators: when training on a reduced dataset, re-sample the training set to full size so that variance, not sample-size artifact, drives the chosen weight.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my take on HyPeR. The paper is a reasonable incremental contribution to OPL: it formulates the partial-reward setting with secondary rewards, and proposes a DR-style estimator that plugs in a reward model conditioned on secondary rewards. The unbiasedness claim is correct under the paper's DGP (o independent of a, s, r given x), and the variance-reduction theorem is a straightforward comparison of two error terms. The gamma tuning is just a validation-based hyperparameter selection, not a deep idea.

What the paper does well: clean problem statement, extensive synthetic and KuaiRec experiments, and the estimator is a natural extension that practitioners could use. The authors give credit to related work, including Aminian et al. and Saito et al.

Where it gets soft: - The proof of Theorem 1 in Appendix B.1 incorrectly replaces E_s[qhat(x,a,s)] with qhat(x,a,f(x,a)). That's not true for a nonlinear qhat. The theorem does hold if you keep the expectation, so the proof needs fixing, but the result stands. - More importantly, the DGP assumes p(o|x) independent of action and secondary rewards. The motivating e-commerce example—conversion observed after a click—violates this. When observation depends on a or s, the estimator is biased and the paper gives no bound or correction. All experiments use constant p(o|x), so they never test the method under the failure mode that the motivation suggests. - In KuaiRec, two secondary rewards are deterministic thresholds of the target reward, which makes the 'misalignment' claim weaker than the abstract suggests. Still, the other two dimensions (upload time, video length) are genuinely independent. - No code released, which limits reproducibility.

Overall: the core idea is valid and useful in its narrow scope, but the paper overclaims applicability to realistic partial-observation settings. A referee should ask for a corrected proof, discussion of bias under action-dependent observation (or a modified DGP), and ideally a synthetic experiment where p(o|x,a,s) varies. This deserves a serious peer review; I would not desk-reject. Would I cite it? Possibly, as a reference for hybrid DR estimation with secondary rewards, but with caveats. Reading group: maybe.

Referee Report

3 major / 4 minor

Summary. The paper studies off-policy learning (OPL) in contextual bandits where the target reward is only partially observed and where a fully observed secondary reward is available. It proposes HyPeR, a policy-gradient estimator that combines a doubly robust target-reward term using a secondary-reward-conditional q-function with a doubly robust secondary-reward term, mixed by a tunable weight gamma. The paper proves unbiasedness of the target-gradient estimator (Theorem 1), gives a variance-reduction comparison with r-DR (Theorem 2), and proposes a validation-based procedure for tuning gamma, possibly to a value different from the objective weight beta. Experiments on synthetic data and on KuaiRec report that HyPeR outperforms r-IPS/r-DR and s-IPS/s-DR baselines on combined, target, and secondary policy values.

Significance. The setting is practically important, and the estimator design is sensible: secondary rewards are used to reduce variance while preserving unbiasedness under the stated data-generating process. The variance decomposition in Theorem 2 is a useful formal result, and the gamma-tuning procedure is evaluated on a validation split, so the main empirical comparison is not circular. The KuaiRec experiment uses a fully observed logged dataset, which is a strength. However, the printed proof of Theorem 1 contains an invalid expectation step, and the paper's motivating examples violate the missingness assumption p(o|x) on which the unbiasedness result rests. Because every experiment sets p(o|x) constant, the empirical study never exercises that load-bearing assumption. These issues need to be addressed before the claimed generality of the framework is established.

major comments (3)

[Appendix B.1 (Theorem 1 proof)] The printed proof of Theorem 1 contains an invalid step: it replaces the expectation over p(s|x,a) of q_hat(x,a,s) with q_hat(x,a,f(x,a)), and similarly replaces q(x,a,s) with q(x,a,f(x,a)), where f(x,a) = E[s|x,a]. This is justified only if q_hat and q are affine in s, which is not assumed. The theorem statement is nevertheless correct under the stated DGP: taking expectations in Eq. (10) gives E_{π_θ(a|x)p(s|x,a)}[q_hat(x,a,s) g] + E_{π_θ(a|x)p(s|x,a)}[(q(x,a,s) - q_hat(x,a,s)) g] = E_{π_θ(a|x)}[q(x,a) g] after cancellation. The proof should be rewritten accordingly; as printed, it does not establish the theorem.
[Section 2.3, Eq. (6); Section 1] The DGP in Eq. (6) assumes p(o|x), so the observation indicator is independent of the action and the secondary reward given the context. The introductory e-commerce example, however, explicitly says conversions are observed only after a click, and clicks are action-dependent secondary rewards. Under the true process p(o|x,a,s), the inner expectation of the third term in Eq. (10) is [p(o=1|x,a,s)/p(o=1|x)] times (q(x,a,s) - q_hat(x,a,s)), not (q(x,a,s) - q_hat(x,a,s)); hence Eq. (10) is biased for ∇_θ V_r and Eq. (11) inherits the bias. Full support does not help, and the gamma-tuning procedure in Section 4.1 optimizes a biased estimate. This is load-bearing because the paper motivates HyPeR with exactly this click-conversion scenario. The authors should either restrict the claims to observation mechanisms satisfying p(o|x,a,s)=p(o|x), or generalize the estimator to use a model of p(o|x,a,s) and quantify the resulting bias.
[Section 5; Section 6] The experiments never depart from the modeling assumption that p(o|x) is constant: the synthetic default is p(o|x)=0.2 for all x (Section 5), and the KuaiRec setup likewise sets p(o|x)=0.2 for all x (Section 6). Thus every simulation has o independent of (a,s) by construction, and the empirical success cannot be read as support for the method in the click-dependent missingness scenarios of the introduction. A minimal addition would be a synthetic condition in which p(o=1|x,a) depends on the action or on a secondary reward, with the reported target policy value and the bias of HyPeR compared against ground truth.

minor comments (4)

[Section 5.1] The formula for the relative policy value appears inverted; as printed, a random policy gives a 0/0 expression and the optimal policy gives 0 rather than 1. It should be (V(π_θ) - V(π_unif)) / (V(π*) - V(π_unif)).
[Appendix B.2, Eq. (19)] The left-hand side should be n(V_D[∇_θ \hat V_r-DR(π_θ;D)] - V_D[∇_θ \hat V_r(π_θ;D)]); the second V_D is missing in the printed expression.
[Section 4.1] Equation (12) restricts γ to [0,1) while Eq. (14) optimizes over [0,1]; the boundaries should be made consistent, especially because the Table 3 results include γ=1.0 as a reported value.
[Section 4.1] The bootstrapped training set D'_tr has the same cardinality as D but contains duplicates, so its gradient-estimation variance is not identical to that of an independent sample of size n; a sentence clarifying that this is a heuristic correction rather than an exact variance match would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the HyPeR derivation is self-contained and the empirical comparisons are against external benchmarks.

full rationale

I examined the estimator definitions in Eqs. (4)-(11), the unbiasedness proof in Appendix B.1, the variance decomposition in Appendix B.2, and the gamma-tuning procedure in Section 4.1. The unbiasedness of Eq. (10) is established by a direct expectation calculation under the stated data-generating process in Eq. (6); it does not reduce to a fitted parameter or to a self-citation. The only substantive condition is p(o|x) being independent of (a,s,r), and if that condition fails the estimator can be biased, but that is a correctness limitation for the motivating e-commerce example, not a circular derivation. Theorem 2 is an algebraic variance decomposition whose sign depends on whether qhat(x,a,s) estimates q(x,a,s) better than qhat(x,a); this is an empirical condition, not an equivalence by construction. The strategic gamma tuning in Section 4.1 is performed on a validation split, and the resulting policies are evaluated on held-out data, so the 'incorrect gamma can improve performance' claim is not forced by the tuning objective itself. Self-citations, including Saito et al. (2024a) and Saito and Nomura (2024), appear as background or related work and do not carry the load-bearing argument. Both synthetic experiments and the KuaiRec real-world evaluation compare against independent baselines, so the empirical claims are externally testable. No circular step meets the evidentiary bar required by the rubric.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method's claims rest on standard bandit assumptions (full support), a missing-at-random condition on the observation indicator, and an implicit modeling assumption that the secondary rewards improve the conditional reward prediction. No fundamentally new entities are introduced.

free parameters (3)

gamma (mixture weight in HyPeR) = tuned on validation, e.g., 0.39 to 1.0 across experiments
The weight gamma in Eq. (11) is not set by theory; the paper proposes data-driven tuning on a validation split (Eq. 14). The central claim that strategic gamma tuning improves results depends on this parameter.
qhat(x,a) and qhat(x,a,s) regression models = not specified
Nuisance models estimated from logged data; Theorem 2's variance reduction holds only if qhat(x,a,s) has smaller squared error. Their functional form and training details are not given.
validation split for gamma tuning = not reported
The bi-level optimization in Eq. (14) requires splitting the data; the split ratio is unstated and affects the quality of tuned gamma.

assumptions (4)

domain assumption Full support: the logging policy pi0(a|x)>0 for all contexts and actions (Condition 1).
Required for unbiased IPS/DR-style estimators including HyPeR (Theorem 1).
domain assumption The observation indicator o is independent of (a,s,r) given x, with 0<p(o|x)<1.
Stated in the DGP Eq. (6) and used in the proof of Theorem 1; if observation depends on action or secondary reward, the estimator is biased.
domain assumption Secondary rewards are fully observed for every logged tuple.
The whole formulation assumes s is always available; in practice this is plausible but not guaranteed.
ad hoc to paper The auxiliary reward model qhat(x,a,s) is more accurate than qhat(x,a) for predicting q(x,a,s).
Theorem 2 states variance reduction only when this inequality holds; it is an assumption about the data and modeling, not a proven property of the algorithm.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A General Framework for Off-Policy Learning with Partially-Observed Reward." pith.science (2026). https://pith.science/paper/5FHTK76A

@misc{pith2026250614439,
  author       = {Pith},
  title        = {Pith review of: A General Framework for Off-Policy Learning with Partially-Observed Reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FHTK76A}},
  note         = {Machine review of arXiv:2506.14439}
}

read the original abstract

Off-policy learning (OPL) in contextual bandits aims to learn a decision-making policy that maximizes the target rewards by using only historical interaction data collected under previously developed policies. Unfortunately, when rewards are only partially observed, the effectiveness of OPL degrades severely. Well-known examples of such partial rewards include explicit ratings in content recommendations, conversion signals on e-commerce platforms that are partial due to delay, and the issue of censoring in medical problems. One possible solution to deal with such partial rewards is to use secondary rewards, such as dwelling time, clicks, and medical indicators, which are more densely observed. However, relying solely on such secondary rewards can also lead to poor policy learning since they may not align with the target reward. Thus, this work studies a new and general problem of OPL where the goal is to learn a policy that maximizes the expected target reward by leveraging densely observed secondary rewards as supplemental data. We then propose a new method called Hybrid Policy Optimization for Partially-Observed Reward (HyPeR), which effectively uses the secondary rewards in addition to the partially-observed target reward to achieve effective OPL despite the challenging scenario. We also discuss a case where we aim to optimize not only the expected target reward but also the expected secondary rewards to some extent; counter-intuitively, we will show that leveraging the two objectives is in fact advantageous also for the optimization of only the target reward. Along with statistical analysis of our proposed methods, empirical evaluations on both synthetic and real-world data show that HyPeR outperforms existing methods in various scenarios.

Figures

Figures reproduced from arXiv: 2506.14439 by the authors.

**Figure 2.** Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Comparing the combined policy values of OPL methods under (left) varying target reward observation probabilities (p(o|x)), (center) varying data sizes (n), and (right) varying true weights β in the combined policy value on the KuaiRec dataset. dimension is video length…

**Figure 6.** Figure 6: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

**Figure 7.** Figure 7: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

**Figure 13.** Figure 13: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

**Figure 14.** Figure 14: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

**Figure 15.** Figure 15: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

**Figure 16.** Figure 16: Comparing the combined, target, and secondary policy values of OPL methods with [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 18 canonical work pages

[5]

These are regression-based methods (also known as Direct Methods), which use regression estimates to predict expected target rewards based solely on logging data

Specifically, we present results for additional baseline methods, including what we refer to asr-DMands-DM. These are regression-based methods (also known as Direct Methods), which use regression estimates to predict expected target rewards based solely on logging data. The policy is then derived by choosing the action with the highest predicted reward in...

work page 2025
[9]

URLhttp://dx.doi.org/ 10.1145/3240323.3240406

doi: 10.1145/3240323.3240406. URLhttp://dx.doi.org/ 10.1145/3240323.3240406. Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination.arXiv preprint arXiv:2202.07234,

arXiv
[10]

URLhttps://doi.org/10.1145/1645953.1645965

1145/1645953.1645965. URLhttps://doi.org/10.1145/1645953.1645965. W. S. Barnett. Effectiveness of early educational intervention.Science, 333(6045):975–978,

arXiv
[11]

Jadidinejad, Craig Macdonald, and Iadh Ounis

Amir H. Jadidinejad, Craig Macdonald, and Iadh Ounis. Unifying explicit and implicit feedback for rating prediction and ranking recommendation tasks. InProceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’19, pp. 149–156, New York, NY , USA,

work page 2019
[12]

ISBN 9781450368810

Association for Computing Machinery. ISBN 9781450368810. doi: 10.1145/ 3341981.3344225. URLhttps://doi.org/10.1145/3341981.3344225. Janus Christian Jakobsen, Christian Gluud, Jørn Wetterslev, and Per Winkel. When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts.BMC Medic...

work page arXiv
[14]

Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction

URLhttps://arxiv.org/abs/1907.06558. Tor Lattimore and Csaba Szepesv´ari.Bandit Algorithms. Cambridge University Press,

work page Pith review arXiv 1907
[16]

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H. Chi. Off-policy learning in two-stage recommender systems. InProceedings of The Web Con- ference 2020, WWW ’20, pp. 463–473, New York, NY , USA,

work page 2020
[18]

Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, No ´emie Elhadad, Daniel Grant, Matthew H

URLhttps://proceedings.neurips.cc/paper_files/paper/2021/ file/4476b929e30dd0c4e8bdbcc82c6ba23a-Paper.pdf. Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, No ´emie Elhadad, Daniel Grant, Matthew H. Mohebbi, Sean Mooney, Christine Suver, John Wilbanks, Lara Mangravite, Patrick J. Heagerty, Pat Are ´an, and Larsson Omberg. Indicators of r...

work page 2021

Show all 31 references

[19]

doi: 10.1038/s41746-020-0224-8

ISSN 2398-6352. doi: 10.1038/s41746-020-0224-8. URLhttps://doi.org/10. 1038/s41746-020-0224-8. Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis.Proceedings of the AAAI Conference on Artificial Intelligence, 33 (...

doi
[20]

URLhttps://ojs.aaai

doi: 10.1609/aaai.v33i01.33014798. URLhttps://ojs.aaai. org/index.php/AAAI/article/view/4407. Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data ...

doi
[21]

12 Published as a conference paper at ICLR 2025 Yuta Saito

URLhttps://arxiv.org/ abs/2310.15433. 12 Published as a conference paper at ICLR 2025 Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. InProceedings of the 14th ACM Conference on Recommender Systems, pp. 92–100,

2025 arXiv
[22]

Hyperparameter optimization can even be harmful in off-policy learning and how to deal with it.arXiv preprint arXiv:2404.15084,

Yuta Saito and Masahiro Nomura. Hyperparameter optimization can even be harmful in off-policy learning and how to deal with it.arXiv preprint arXiv:2404.15084,

arXiv
[23]

Long-term off-policy evaluation and learning

Yuta Saito, Himan Abdollahpouri, Jesse Anderton, Ben Carterette, and Mounia Lalmas. Long-term off-policy evaluation and learning. InProceedings of the ACM on Web Conference 2024, pp. 3432–3443, 2024a. Yuta Saito, Jihan Yao, and Thorsten Joachims. Potec: Off-policy learning for...

2024 arXiv
[24]

Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dud´ık

URLhttps://arxiv.org/ abs/1811.02672. Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dud´ık. Doubly robust off-policy evaluation with shrinkage, 2020a. Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. InI...

arXiv
[25]

A review of off-policy evaluation in rein- forcement learning.arXiv preprint arXiv:2212.06355,

Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning.arXiv preprint arXiv:2212.06355,

arXiv
[26]

ISBN 9781450359016

Association for Computing Machinery. ISBN 9781450359016. doi: 10.1145/3240323.3240369. URLhttps://doi.org/10.1145/3240323.3240369. Nan Wang, Hongning Wang, Maryam Karimzadehgan, Branislav Kveton, and Craig Boutilier. Imo3: Interactive multi-objective off-policy optimization. J...
[27]

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi

URLhttps://arxiv.org/ abs/2304.07920. Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.),Proceedings of the 36th International Conference on M...

arXiv
[28]

However, their primary focus is on data-adaptive estimator selection for OPE, rather than on effectively tuning hyperparameters for OPL

is relevant in this context. However, their primary focus is on data-adaptive estimator selection for OPE, rather than on effectively tuning hyperparameters for OPL. Our work is also related to the multi-objective optimization problems (Wang et al., 2022a; Alizadeh et al., 202...

2010
[30]

Data Size Meanγ ∗ Meanˆγ∗ 500 0.7205 0.5458 1000 0.6612 0.5890 2000 0.5678 0.6392 4000 0.4462 0.5309 8000 0.3356 0.4794 16000 0.2878 0.4306 (c) Different Beta Values (β) (c.f. Figure

2000
[31]

However, we observe that HyPeR methods consistently outperform all other methods across all evaluated policy values

As the action set size increases, the performance of all methods degrades due to an increase in variance in gradient estimation. However, we observe that HyPeR methods consistently outperform all other methods across all evaluated policy values. Below, Tables 3, 3, 3, and 3 pr...

2025
[2010]

ISBN 9781450300995

Association for Computing Machinery. ISBN 9781450300995. doi: 10.1145/1871437.1871643. URLhttps://doi.org/10.1145/1871437.1871643. Ben London and Ted Sandler. Bayesian counterfactual risk minimization. InInternational Confer- ence on Machine Learning, pp. 4125–4133. PMLR,
[2011]

URLhttps://www.science.org/doi/abs/10.1126/ science.1204534

doi: 10.1126/science.1204534. URLhttps://www.science.org/doi/abs/10.1126/ science.1204534. Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top-k off- policy correction for a reinforce recommender system,

doi
[2014]

doi: 10.1214/ 14-sts500

ISSN 0883-4237. doi: 10.1214/ 14-sts500. URLhttp://dx.doi.org/10.1214/14-STS500. Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation,

doi
[2017]

doi: 10.1186/s12874-017-0442-1

ISSN 1471-2288. doi: 10.1186/s12874-017-0442-1. 11 Published as a conference paper at ICLR 2025 Olivier Jeunen and Bart Goethals. Pessimistic reward models for off-policy learning in recom- mendation. InProceedings of the 15th ACM Conference on Recommender Systems, RecSys ’21,...

2025 doi
[2018]

Germano Gabbianelli, Gergely Neu, and Matteo Papini

URLhttps://arxiv.org/abs/1802.03493. Germano Gabbianelli, Gergely Neu, and Matteo Papini. Importance-weighted offline learning done right. InInternational Conference on Algorithmic Learning Theory, pp. 614–634. PMLR,

arXiv
[2019]

Azin Ashkan and Charles L.A

doi: 10.1109/ACCESS.2019.2938659. Azin Ashkan and Charles L.A. Clarke. Characterizing commercial intent. InProceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM ’09, pp. 67–76, New York, NY , USA,

2019
[2020]

ISBN 9781450370233

Association for Com- puting Machinery. ISBN 9781450370233. doi: 10.1145/3366423.3380130. URLhttps: //doi.org/10.1145/3366423.3380130. Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estim...
[2021]

ISBN 9781450384582

Association for Computing Machinery. ISBN 9781450384582. doi: 10.1145/3460231.3474247. URLhttps://doi.org/10.1145/ 3460231.3474247. Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit feedback. InInternational Conference on Learning Rep...
[2022]

URL https://doi.org/10.1145/3511808.3557220

doi: 10.1145/3511808.3557220. URL https://doi.org/10.1145/3511808.3557220. Guy Hadash, Oren Sar Shalom, and Rita Osadchy. Rank and rate: multi-task learning for recom- mender systems. InProceedings of the 12th ACM Conference on Recommender Systems, RecSys ’18. ACM, September
[2023]

arXiv preprint arXiv:2310.18617

URLhttp:// arxiv.org/abs/2310.18617. arXiv preprint arXiv:2310.18617. Gholamali Aminian, Armin Behnamnia, Roberto Vega, Laura Toni, Chengchun Shi, Hamid R. Ra- biee, Omar Rivasplata, and Miguel R. D. Rodrigues. Semi-supervised batch learning from logged data,

arXiv
[2024]

Javier Arroyo, Francesco Corea, Guillermo Jimenez-Diaz, and Juan A

URLhttps://arxiv.org/abs/2209.07148. Javier Arroyo, Francesco Corea, Guillermo Jimenez-Diaz, and Juan A. Recio-Garcia. Assessment of machine learning performance for decision support in venture capital investments.IEEE Access, 7:124233–124243,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [5]

These are regression-based methods (also known as Direct Methods), which use regression estimates to predict expected target rewards based solely on logging data

Specifically, we present results for additional baseline methods, including what we refer to asr-DMands-DM. These are regression-based methods (also known as Direct Methods), which use regression estimates to predict expected target rewards based solely on logging data. The policy is then derived by choosing the action with the highest predicted reward in...

work page 2025

[2] [9]

URLhttp://dx.doi.org/ 10.1145/3240323.3240406

doi: 10.1145/3240323.3240406. URLhttp://dx.doi.org/ 10.1145/3240323.3240406. Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination.arXiv preprint arXiv:2202.07234,

arXiv

[3] [10]

URLhttps://doi.org/10.1145/1645953.1645965

1145/1645953.1645965. URLhttps://doi.org/10.1145/1645953.1645965. W. S. Barnett. Effectiveness of early educational intervention.Science, 333(6045):975–978,

arXiv

[4] [11]

Jadidinejad, Craig Macdonald, and Iadh Ounis

Amir H. Jadidinejad, Craig Macdonald, and Iadh Ounis. Unifying explicit and implicit feedback for rating prediction and ranking recommendation tasks. InProceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’19, pp. 149–156, New York, NY , USA,

work page 2019

[5] [12]

ISBN 9781450368810

Association for Computing Machinery. ISBN 9781450368810. doi: 10.1145/ 3341981.3344225. URLhttps://doi.org/10.1145/3341981.3344225. Janus Christian Jakobsen, Christian Gluud, Jørn Wetterslev, and Per Winkel. When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts.BMC Medic...

work page arXiv

[6] [14]

Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction

URLhttps://arxiv.org/abs/1907.06558. Tor Lattimore and Csaba Szepesv´ari.Bandit Algorithms. Cambridge University Press,

work page Pith review arXiv 1907

[7] [16]

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H. Chi. Off-policy learning in two-stage recommender systems. InProceedings of The Web Con- ference 2020, WWW ’20, pp. 463–473, New York, NY , USA,

work page 2020

[8] [18]

Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, No ´emie Elhadad, Daniel Grant, Matthew H

URLhttps://proceedings.neurips.cc/paper_files/paper/2021/ file/4476b929e30dd0c4e8bdbcc82c6ba23a-Paper.pdf. Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, No ´emie Elhadad, Daniel Grant, Matthew H. Mohebbi, Sean Mooney, Christine Suver, John Wilbanks, Lara Mangravite, Patrick J. Heagerty, Pat Are ´an, and Larsson Omberg. Indicators of r...

work page 2021