Pith. sign in

REVIEW 3 major objections 6 minor 57 references

Off-Policy Evaluation and Learning for the Future under Non-Stationarity

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OPFV estimates a policy's future value by reweighting historical samples through repeated time features, with its bias pinned to the regressor's residual-difference error.

desk verdict A genuinely new estimator for future policy value with a clean idea, but the main theorem silently assumes stationary context and does not cover the paper's advertised non-stationary-context claim. read the letter →

arxiv 2506.20417 v1 pith:6IP6QUO7 submitted 2025-06-25 cs.LG cs.AI

Tatsuhiro Shimizu , Kazuki Kawamura , Takanori Muroi , Yusuke Narita , Kei Tateno , Takuma Udagawa , Yuta Saito This is my paper · ORCID
classification cs.LGcs.AI
keywords off-policyevaluationfuturepolicyvaluenon-stationaritycontextualbanditstime-seriesfeaturesimportanceweightinggradientrecommendationsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on a counterfactual question that standard off-policy evaluation cannot answer: what would a recommendation policy be worth in an upcoming month or season, when the logged data were collected by an older policy in the past and contain no samples from the future? It proposes the OPFV estimator, which decomposes the reward into a time-feature effect that recurs across weeks, seasons, or holidays, estimated by reweighting historical samples whose calendar feature matches the target time, plus a residual effect handled by a fitted regressor. The central theoretical result gives an exact bias formula: bias equals the expected, time-feature-weighted error of the regressor in reproducing the reward difference between the past and target timestamps, so the bias shrinks when the time feature is fine or the regressor captures the residual well. The same construction yields OPFV-PG, a policy-gradient learner for optimizing a future policy from historical data alone. In synthetic non-stationary bandit experiments and on the KuaiRec recommender log, the proposed methods track the time-varying future value and find higher-value future policies than IPS, DR, and Prognosticator baselines.

What carries the argument

The load-bearing object is the time feature function $\phi(t)$, a partition of time such as day of week or season, together with the decomposition $q(x,t,a)=g(x,\phi(t),a)+h(x,t,a)$. The new importance weight $\mathbb{I}\{\phi(t)=\phi(t')\}/p(\phi(t'))$ lets the estimator non-parametrically capture the time-feature effect $g$ from past samples that share the target feature, while the regression $\hat{f}$ handles the residual effect $h$; the bias identity then ties estimator accuracy to the residual-difference error evaluated only on same-feature time pairs.

What would settle it

Fix a synthetic bandit as in the paper's Eq. (9) with the same time-feature structure, but make the context mean drift smoothly within each season, and compare OPFV's empirical bias for a future time $t'$ to the right-hand side of Theorem 3.3 evaluated with the true regressor error; if the difference grows with the amplitude of the context drift, the stationary-context premise is load-bearing and the theorem's stated generality over $p(x,t)$ needs qualification.

Watch

Extended reading notes

Core claim

The central claim is that the future value $V_{t'}(\pi_e)$ can be estimated with low bias by the OPFV estimator, defined as averaging, over historical samples, the target-time importance weight $I_{\phi(t_i,t')}/p(\phi(t')) \, \pi_e(a_i|x_i,t')/\pi_0(a_i|x_i,t_i)$ on the residual $r_i - \hat{f}$ plus the regressor's target-time expectation $\mathbb{E}_{\pi_e(\cdot|x_i,t')}[\hat{f}(x_i,t',\cdot)]$. The paper proves that the bias of this estimator is $\mathbb{E}\left[\frac{I_{\phi(t,t')}}{p(\phi(t'))}\left(\Delta q(x,t,t',a) - \Delta \hat{f}(x,t,t',a)\right)\right]$, a quantity controlled by the coarseness of the time feature $\phi$ and by how accurately the regressor $\hat{f}$ preserves pairwise reward differences between same-feature timestamps; under pairwise correctness of the regressor, the bias vanishes. The paper further claims that the gradient of OPFV has the same bias structure multiplied by the policy score function, so policy-gradient learning for a future target time inherits the low-bias guarantee. Empirically, the paper claims that OPFV and OPFV-PG outperform IPS, DR, and the Prognosticator in both evaluation and learning across synthetic and real-world non-stationary setups.

Load-bearing premise

The theorem's low-bias guarantee silently assumes the context distribution is stationary in time: the proof invokes 'stationary context', so if $p(x|t)$ drifts within a repeated time feature, the original OPFV carries an additional bias that the formula does not quantify.

Editorial extensions

If this is right

  • For any target time $t'$ with positive time-feature density $p(\phi(t'))>0$, OPFV provides a low-bias estimate of $V_{t'}(\pi_e)$ whenever the true reward contains a recurring time-feature component; IPS and DR carry an unresolvable stationarity bias in the same setting.
  • The bias-variance tradeoff is controlled by the granularity of $\phi$, so practitioners can tune the time feature on historical data alone by minimizing an estimated squared-bias-plus-variance criterion.
  • OPFV-PG enables offline learning of a policy aimed at a specific future period, and its estimated gradient is low-bias when the regressor preserves pairwise reward differences on same-feature timestamps.
  • When the time feature carries no information, OPFV performs comparably to existing estimators rather than worse, so adopting it under non-stationarity does not sacrifice the stationary baseline.
  • The extended estimator with separate context and reward time features can also handle non-stationary context distributions, provided the unmodeled part of the context distribution is conditionally stationary across same-feature times.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit in the proof is a sensitivity warning: the derivation of the bias formula uses stationary context, so with drifting $p(x|t)$ the original estimator has an unquantified bias term that a practitioner should address by modeling context shift or by using the context-conditional extension.
  • The same mechanism applies to any repeated discrete feature besides calendar time, such as user cohort, device type, or store location, suggesting a general repeated-feature-effect estimator beyond time series.
  • Because the bias depends only on pairwise residual differences, a regressor trained on pairwise differences within same-feature groups, as outlined in the paper's regression-optimization section, should beat a globally fitted regressor; this is a testable design choice the paper proposes but does not fully ablate.
  • The real-world evaluation estimates test-set policy value with OPE estimators on holdout data; a direct online A/B test of OPFV-PG against the logging policy would be a stronger test of the claimed future-value gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper formulates future off-policy evaluation (F-OPE) and learning (F-OPL): given historical logged bandit data collected over [0,T], estimate or optimize the value of an evaluation policy at a future target time t'>T under non-stationary reward and possibly context distributions. The proposed OPFV estimator reweights logged samples whose time feature phi(t) matches phi(t') with weight I_phi(t,t')/p(phi(t')), applies the ordinary importance ratio, and adds a regression correction based on fhat(x,t',a). The authors derive a bias formula (Theorem 3.3), a variance formula (Proposition 3.4), a data-driven procedure for choosing phi by an estimated MSE proxy, and a policy-gradient extension OPFV-PG. Experiments on synthetic data and the KuaiRec recommendation dataset compare OPFV/OPFV-PG against IPS, DR, Prognosticator, and variants.

Significance. The core idea is novel and potentially useful: it is the first to exploit periodic time-series features through a new importance weight for future OPE, and the bias/variance decomposition is a clear conceptual advance over Prognosticator's two-stage regression. The paper also releases code, reports results over multiple seeds, and includes an appendix extension for non-stationary contexts. However, the central theoretical guarantee as stated in the main text is not valid for non-stationary context distributions, and the real-world demonstration uses the unextended estimator. If the theory is scoped correctly and the experiments are aligned with the estimator whose properties are proved, the contribution would be solid; in its current form the load-bearing claim overreaches.

major comments (3)
  1. [Section 3.2 / Theorem 3.3; Appendix H.2] Theorem 3.3 states a bias formula under only Conditions 3.1 and 3.2, but the proof in Appendix H.2 twice replaces E_{p(x,t)} with E_{p(x)p(t)} and E_{p(x|t')} with E_{p(x)} using the annotation '∵stationary context'. Section 3 explicitly allows a general joint density p(x,t), so the theorem as stated is not supported. For a general p(x,t), the right-hand side of Eq. (5) omits the term E_{p(x,t)pi_e(a|x,t')}[ I_phi(t,t')/p(phi(t')) (q(x,t',a)-fhat(x,t',a)) ] + E_{p(x,t)}E_{pi_e(a|x,t')}[fhat(x,t',a)] - E_{p(x|t')pi_e(a|x,t')}[q(x,t',a)], which is generically nonzero when p(x|t) varies with t and q depends on x. This is load-bearing because the abstract and Introduction advertise non-stationary context and reward, and the Appendix C extension (Eq. (11)) exists precisely because the original estimator is not bias-controlled under context shift. The theorem should either be stated with a stationary-context assumption or proved under the assumptions that justify it.
  2. [Section 3.2 / Proposition 3.4; Appendix H.3] Proposition 3.4 is also stated under only Conditions 3.1 and 3.2, but the proof in Appendix H.3 invokes Condition C.3 (which is not introduced in the main text) and the stationary-context replacement more than once. Consequently the variance formula in Eq. (6) has unstated assumptions. This matters for Section 3.3, where the data-driven selection of phi uses the estimated variance; if Eq. (6) is not valid under the stated conditions, the MSE proxy in Eq. (7) lacks the advertised theoretical grounding.
  3. [Section 3.4, Appendix F, and Section 4 real-world experiment] The same stationary-context gap transfers to OPFV-PG: Theorem F.2 is stated under Conditions F.1 and 3.2, and Proposition F.4 under F.1, 3.2, C.3, but their proofs in H.6 and H.7 use '∵stationary context' to factor p(x,t). Since the KuaiRec F-OPL experiment in Section 4 (Table 1) uses the original OPFV-PG (Eq. (8)) and not the extended estimator of Appendix C (Eq. (11)), the real-world result is not covered by the stated theory unless the authors either justify the stationary-context assumption for that dataset or run the extended estimator. The paper should make this limitation explicit and align the claims with the estimator actually analyzed.
minor comments (6)
  1. [Appendix C.2] The heading 'Unibasedness' should be 'Unbiasedness'.
  2. [Section 4] The word 'non-staitonarity' appears in the text and in the caption of Figure 8; it should be 'non-stationarity'.
  3. [Section 3.1, Eq. (3)] Because g and h are unrestricted, the decomposition exists for any q and the two components are not identified. The text should state explicitly that the 'time feature effect' is a property of the estimator's weighting, not an identified component of the reward function, to avoid giving the impression that the method estimates an identified g.
  4. [Table 2] Table 2 and several appendix tables use '%' and '"' characters where check/cross symbols are intended; these do not render reliably and should be replaced with explicit symbols.
  5. [Appendix H.5] In the proof of Proposition F.3, the line containing E_{pi_zeta(a'|x,t')}[fhat(x,t,a's_zeta(x,t',a'))] has misplaced parentheses; it should be E_{pi_zeta(a'|x,t')}[fhat(x,t',a') s_zeta(x,t',a')].
  6. [Section 4, real-world experiment] In the KuaiRec comparison (Table 1 and Appendix J.4), the 10-run standard deviations of the reported policy values overlap between OPFV-PG and Prognosticator; reporting paired differences or confidence intervals across seeds would make the 'substantially outperform' claim more robust.

Circularity Check

2 steps flagged · score 2.0 of 10

No fitted-value or self-citation circularity; the low-bias theorem is, however, proved only under a hidden stationary-context substitution, and the phi-selection bias proxy is self-referential.

  1. other [Appendix H.2, proof of Theorem 3.3; same substitution in H.1, H.3, H.5-H.7]
    "=E_{p(x)}p(t)π_0(a|x,t)(...) − E_{p(x)}π_e(a|x,t′)[q(x,t′,a)] ∵stationary context ... =E_{p(x|t′)}π_e(a|x,t′)[q(x,t′,a)] ∵stationary context i.e., p(x)=p(x|t′)"

    The theorem states OPFV's bias is Eq. (5) under only Conditions 3.1 and 3.2, in a formulation where p(x,t) is a general joint density and the abstract/intro advertise non-stationary context and reward distributions. The proof obtains the claimed formula by replacing E_{p(x,t)} with E_{p(x)p(t)} and E_{p(x|t')} with E_{p(x)}, annotated 'stationary context'. Those substitutions are exactly the assumption p(x|t)=p(x). Without them, Eq. (5) is missing an extra term E_{p(x,t)π_e(a|x,t')}[q(x,t',a)] − E_{p(x|t')π_e(a|x,t')}[q(x,t',a)], which is generically nonzero under context drift. Thus the central low-bias guarantee is not derived from the advertised non-stationary inputs; it is proved for a stationary-context special case.

  2. other [Section 3.3, Eq. (7)]
    "dBias(ϕ) := V̂^{OPFV}_{t′}(π_e;D,ϕ) − V̂^{OPFV}_{t′}(π_e;D,ϕ_∞) is an estimate of the bias of OPFV"

    This step is self-referential: the estimator being tuned is used to estimate its own bias, and the quantity is a centered difference, so its expectation is Bias(ϕ) − Bias(ϕ_∞), not Bias(ϕ). Any offset common to all candidate features—including the target value V_{t'}(π_e)—cancels, so the objective in Eq. (7) cannot identify the absolute bias it claims to estimate. This weakens the stated justification of the phi-tuning procedure. However, the final synthetic and KuaiRec evaluations are against held-out future values, so this self-referential selection step does not by itself force the headline empirical results.

full rationale

OPFV is not fitted to the future values it claims to evaluate: Theorem 3.3 is an honest expectation calculation, and the synthetic and KuaiRec experiments evaluate on genuinely future periods, so the headline prediction is not statistically forced. The self-citations (e.g., Saito et al. [40] for reward decomposition) are acknowledged inspirations and are not used as an external uniqueness theorem; no load-bearing claim reduces to an author's prior result. I therefore do not find a fitted-input or self-citation circularity at the 6+ level. Two points keep the score above zero. First, the proof of Theorem 3.3 (and the other main theorems) silently substitutes 'stationary context' twice, so the claimed bias formula is derived only for p(x|t)=p(x) even though the problem formulation advertises non-stationary context and reward; the Appendix C estimator and Condition C.1 are what would be needed, but the main theorem and real-world experiment do not use them. Second, Eq. (7) defines dBias as a difference of two OPFV estimates, which is a self-referential proxy for absolute bias because it cannot see the common target-value offset; this weakens the phi-tuning argument but does not force the held-out evaluation results. These are correctness/identification gaps and a minor self-referential selection step, not a derivation that equals its inputs by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on i.i.d. sampling, common support, recurrence of the chosen time feature, stationary context in the main theorem, and a strong pairwise-correctness condition on the regressor for unbiasedness. The time-feature function and the reward regressor are the effective degrees of freedom; without them, the method reduces to existing DR or regression approaches.

free parameters (2)
  • Time feature function phi = Data-driven selection; day_of_week in KuaiRec; 8-season phi_true in synthetic
    Granularity controls the bias-variance tradeoff (Theorem 3.3, Proposition 3.4). It is tuned by Eq (7) via estimated bias against the finest phi_inf, so reported performance depends on this categorical hyperparameter.
  • Reward regressor fhat(x,t,a) = Trained model (random forest in synthetic; neural network in real-world)
    Appears in the residual term and in the oracle term fhat(x,t',a) at the future time. The bias formula (Eq 5) vanishes only if fhat captures pairwise differences of the residual effect, and there is no historical data at t' to validate this extrapolation.
assumptions (5)
  • domain assumption Condition 3.1 (Common Support): pi0(a|x,t)>0 whenever pi_e(a|x,t')>0 for all x, t in [0,T], a.
    Standard overlap needed for importance weighting; if actions from the evaluation policy were never logged, IPS-style weights blow up. Stated in Section 3.2.
  • domain assumption Condition 3.2 (Common Time Feature Support): p(phi(t'))>0, i.e., some historical timestamp shares the target time's feature.
    Without it, the time-feature weight has zero support and OPFV degenerates to regression on the residual. The paper argues phi can be chosen to satisfy it, but this limits applicability to recurring calendar or time structures.
  • ad hoc to paper Stationary context p(x|t)=p(x) in the main theory.
    Used implicitly in the proofs of Theorem 3.3 and Proposition 3.4 ('∵stationary context'); not stated in the theorem statements. The paper's problem formulation allows p(x,t) to vary, and real-world data likely violates this.
  • ad hoc to paper Condition C.3 (Conditional Pairwise Correctness): fhat(x,t,a)-fhat(x,t',a)=q(x,t,a)-q(x,t',a) for all t with phi(t)=phi(t').
    Required for unbiasedness of OPFV (Proposition C.4) and for the variance simplification in Proposition 3.4. It requires the regressor to extrapolate the residual effect to the future target time, which cannot be verified from historical data.
  • domain assumption Condition C.1 (Conditional Stationarity for Context) for the extended OPFV estimator.
    Needed for bias Theorem C.2 when the context distribution is non-stationary; requires the non-feature part of the context p2(x|t) to equal p2(x|t') on timestamps sharing phi_x. This is strong and untestable from D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Policy Evaluation and Learning for the Future under Non-Stationarity." pith.science (2026). https://pith.science/paper/6IP6QUO7

@misc{pith2026250620417,
  author       = {Pith},
  title        = {Pith review of: Off-Policy Evaluation and Learning for the Future under Non-Stationarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6IP6QUO7}},
  note         = {Machine review of arXiv:2506.20417}
}
read the original abstract

We study the novel problem of future off-policy evaluation (F-OPE) and learning (F-OPL) for estimating and optimizing the future value of policies in non-stationary environments, where distributions vary over time. In e-commerce recommendations, for instance, our goal is often to estimate and optimize the policy value for the upcoming month using data collected by an old policy in the previous month. A critical challenge is that data related to the future environment is not observed in the historical data. Existing methods assume stationarity or depend on restrictive reward-modeling assumptions, leading to significant bias. To address these limitations, we propose a novel estimator named \textit{\textbf{O}ff-\textbf{P}olicy Estimator for the \textbf{F}uture \textbf{V}alue (\textbf{\textit{OPFV}})}, designed for accurately estimating policy values at any future time point. The key feature of OPFV is its ability to leverage the useful structure within time-series data. While future data might not be present in the historical log, we can leverage, for example, seasonal, weekly, or holiday effects that are consistent in both the historical and future data. Our estimator is the first to exploit these time-related structures via a new type of importance weighting, enabling effective F-OPE. Theoretical analysis identifies the conditions under which OPFV becomes low-bias. In addition, we extend our estimator to develop a new policy-gradient method to proactively learn a good future policy using only historical data. Empirical results show that our methods substantially outperform existing methods in estimating and optimizing the future policy value under non-stationarity for various experimental setups.

Figures

Figures reproduced from arXiv: 2506.20417 by the authors.

Figure 1
Figure 1. The key concept of Future OPE (F-OPE) and its comparison with conventional OPE, which operates under the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The bias-variance tradeoff of OPFV is controlled [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparing (i) the true future policy value [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Comparison of the MSE, squared bias, and variance of estimators with varying cardinalities of time features [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the MSE, squared bias, and variance of estimators in F-OPE with varying logged data sizes [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the policy values (higher the better) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparing the true time feature 𝜙true (= 𝜙8) and the time feature functions 𝜙 ∈ Φ used in the optimization procedure in Section 3.3 for the OPFV estimator where Φ := {𝜙2𝑖 : ∀𝑖 = 1, 2, · · · , 8} for any year 𝑔(𝑥, 𝜙 (𝑡), 𝑎) = 𝜈 ⊤ 𝑥 𝑠𝑔1 (𝑥) + 𝜈 ⊤ 𝜙 (𝑡) one_hot𝜙 (t) + one…
Figure 8
Figure 8. Figure 8: Comparing (i) the true value 𝑉𝑡 (𝜋𝑒 ) and the estimated policy values 𝑉ˆ 𝑡 (𝜋𝑒 ; D) with varying target time range 𝑡 ′ and (ii) the MSE of estimators with varying 𝜆 values that controls the informativeness of the time feature [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]
Figure 9
Figure 9. Figure 9: Comparison of the MSE, squared bias, and variance of estimators in an F-OPE problem with varying sample sizes [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Comparison of the policy values achieved by F-OPL methods with (left) varying target times [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: Comparison of the test policy values of the Regression-based approach, IPS and DR Policy Gradient, Prognosticator, [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Comparison of the true value and estimated values of IPS, DR, Prognosticator, OPFV (true [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]
Figure 13
Figure 13. Figure 13: Comparison of the MSE, bias, and variance of IPS, DR, Prognosticator, OPFV (true [PITH_FULL_IMAGE:figures/full_fig_p034_13.png]
Figure 14
Figure 14. Figure 14: Comparison of the MSE, bias, and variance of IPS, DR, Prognosticator, OPFV (true [PITH_FULL_IMAGE:figures/full_fig_p034_14.png]
Figure 15
Figure 15. Figure 15: Comparison of the MSE, bias, and variance of IPS, DR, Prognosticator, OPFV ( [PITH_FULL_IMAGE:figures/full_fig_p035_15.png]
Figure 16
Figure 16. Figure 16: Comparing the true value 𝑉𝑡 (𝜋) of completely random policy 𝜋 with varying dates in Kuai Rec training dataset [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]
Figure 17
Figure 17. Figure 17: Comparing the true value 𝑉𝑡 (𝜋) of completely random policy 𝜋 with varying dates in Kuai Rec test dataset {Sun, Mon, Tue, Wed, Thu, Fri, Sat}) as we briefly described in Section 4. For OPFV-PG (w/ tuned 𝜙), we employ the following set Φ := n 𝜙day_of_week, 𝜙weekday_wee…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 32 canonical work pages

  1. [1]

    A Agarwal, N Jiang, and M Kakade, S. 2020. Reinforcement Learning: Theory and algorithms.MIT(2020)

    work page 2020

  2. [2]

    Alina Beygelzimer and John Langford. 2009. The Offset Tree for Learning with Partial Labels. InProceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 129–138

    work page 2009

  3. [3]

    Yash Chandak, Shiv Shankar, Nathaniel Bastian, Bruno da Silva, Emma Brunskill, and Philip S Thomas. 2022. Off-policy evaluation for action-dependent non- stationary environments.Advances in Neural Information Processing Systems35 (2022), 9217–9232

    work page 2022

  4. [4]

    Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. 2020. Optimizing for the future in non-stationary mdps. InInternational Conference on Machine Learning. PMLR, 1414–1425

    work page 2020

  5. [5]

    Matej Cief, Michal Kompan, and Branislav Kveton. 2024. Cross-Validated Off- Policy Evaluation.arXiv preprint arXiv:2405.15332(2024)

    work page Pith review arXiv 2024

  6. [6]

    Omar Darwiche Domingues, Pierre Ménard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. 2021. A kernel-based approach to non-stationary reinforcement learning in metric spaces. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 3538–3546

    work page 2021

  7. [7]

    Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly Robust Policy Evaluation and Optimization.Statist. Sci.29, 4 (2014), 485–511

    2014

  8. [8]

    Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly robust policy evaluation and learning. InProceedings of the 28th International Conference on International Conference on Machine Learning. 1097–1104

    work page 2011

Show all 57 references
  1. [9]

    Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. 2018. More Robust Doubly Robust Off-Policy Evaluation. InProceedings of the 35th Interna- tional Conference on Machine Learning, Vol. 80. PMLR, 1447–1456

    2018

  2. [10]

    Nicolò Felicioni, Michael Benigni, and Maurizio Ferrari Dacrema. 2024. AutoOPE: Automated Off-Policy Estimator Selection.arXiv preprint arXiv:2406.18022(2024)

    2024 arXiv

  3. [11]

    Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022. KuaiRec: A fully-observed dataset and insights for evaluating recommender systems. InProceedings of the 31st ACM International Conference on Information ...

    2022

  4. [12]

    Joey Hong, Branislav Kveton, Manzil Zaheer, Yinlam Chow, and Amr Ahmed

  5. [13]

    Daniel G Horvitz and Donovan J Thompson. 1952. A generalization of sampling without replacement from a finite universe.Journal of the American statistical Association47, 260 (1952), 663–685

    1952

  6. [14]

    Rolf Jagerman, Ilya Markov, and Maarten de Rijke. 2019. When people change their mind: Off-policy evaluation in non-stationary recommendation environ- ments. InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 447–455

    2019

  7. [15]

    Olivier Jeunen and Bart Goethals. 2021. Pessimistic reward models for off- policy learning in recommendation. InProceedings of the 15th ACM Conference on Recommender Systems. 63–74

    2021

  8. [16]

    Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. 2022. Doubly robust distributionally robust off-policy evaluation and learning. InInternational Conference on Machine Learning. PMLR, 10598–10632

    2022

  9. [17]

    Nathan Kallus, Yuta Saito, and Masatoshi Uehara. 2021. Optimal off-policy evaluation from multiple logging policies. InInternational Conference on Machine Learning. PMLR, 5247–5256

    2021

  10. [18]

    Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. 2024. Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation. InThe Twelfth International Conference on Learning Representations

    2024

  11. [19]

    Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. 2024. Off-policy evaluation of slate bandit policies via optimizing abstraction. InProceedings of the ACM on Web Conference 2024. 3150–3161

    2024

  12. [20]

    Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. 2022. Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model. InProceedings of the 15th International Conference on Web Search and Data Mining

    2022

  13. [21]

    Haruka Kiyohara, Masatoshi Uehara, Yusuke Narita, Nobuyuki Shimizu, Yasuo Yamamoto, and Yuta Saito. 2023. Off-Policy Evaluation of Ranking Policies under Diverse User Behavior. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1154–1163

    2023

  14. [22]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conser- vative Q-Learning for Offline Reinforcement Learning. InAdvances in Neural Information Processing Systems, Vol. 33. 1179–1191

    2020

  15. [23]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline Rein- forcement Learning: Tutorial, Review, and Perspectives on Open Problems.arXiv preprint arXiv:2005.01643(2020)

    2020 arXiv

  16. [24]

    Mengbing Li, Chengchun Shi, Zhenke Wu, and Piotr Fryzlewicz. 2022. Testing stationarity and change point detection in reinforcement learning.arXiv preprint arXiv:2203.01707(2022)

    2022 arXiv

  17. [25]

    Dawen Liang and Nikos Vlassis. 2022. Local Policy Improvement for Recom- mender Systems.arXiv preprint arXiv:2212.11431(2022)

    2022 arXiv

  18. [26]

    Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. 2018. Breaking the curse of horizon: infinite-horizon off-policy estimation. InProceedings of the 32nd International Conference on Neural Information Processing Systems. 5361–5371

    2018

  19. [27]

    Vincent Liu, Yash Chandak, Philip Thomas, and Martha White. 2023. Asymptoti- cally Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonsta- tionary Environments. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 5474–5492

    2023

  20. [28]

    Yifei Ma, Yu-Xiang Wang, and Balakrishnan Narayanaswamy. 2019. Imitation- regularized offline learning. InThe 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2956–2965

    2019

  21. [29]

    Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. 2021. Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning. Advances in Neural Information Processing Systems34 (2021)

    2021

  22. [30]

    Tong Mu, Yash Chandak, Tatsunori B Hashimoto, and Emma Brunskill. 2022. Factored DRO: Factored Distributionally Robust Policies for Contextual Bandits. Advances in Neural Information Processing Systems35 (2022), 8318–8331

    2022

  23. [31]

    Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. 2020. Reinforce- ment learning algorithm for non-stationary environments.Applied Intelligence 50, 11 (2020), 3590–3606

    2020

  24. [32]

    Sutton, and Satinder P

    Doina Precup, Richard S. Sutton, and Satinder P. Singh. 2000. Eligibility Traces for Off-Policy Policy Evaluation. InProceedings of the 17th International Conference on Machine Learning. 759–766

    2000

  25. [33]

    inverse probability

    James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea Rotnitzky. 2007. Comment: Performance of double-robust estimators when" inverse probability" weights are highly variable.Statist. Sci.22, 4 (2007), 544–559

    2007

  26. [34]

    Noveen Sachdeva, Yi Su, and Thorsten Joachims. 2020. Off-Policy Bandits with Deficient Support. InProceedings of the 26th ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining. 965–975

    2020

  27. [35]

    Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley

  28. [36]

    Yuta Saito, Himan Abdollahpouri, Jesse Anderton, Ben Carterette, and Mounia Lalmas. 2024. Long-term Off-Policy Evaluation and Learning. InProceedings of the ACM on Web Conference 2024. 3432–3443

    2024

  29. [37]

    Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. 2021. Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    2021

  30. [38]

    Yuta Saito and Thorsten Joachims. 2021. Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent Advances. InProceedings of the 15th ACM Conference on Recommender Systems. 828–830

    2021

  31. [39]

    Yuta Saito and Thorsten Joachims. 2022. Off-Policy Evaluation for Large Action Spaces via Embeddings. InInternational Conference on Machine Learning. PMLR, 19089–19122

    2022

  32. [40]

    Yuta Saito, Ren Qingyang, and Thorsten Joachims. 2023. Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. InInternational Conference on Machine Learning. PMLR, 29734–29759

    2023

  33. [41]

    Yuta Saito, Jihan Yao, and Thorsten Joachims. 2024. POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition.arXiv preprint arXiv:2402.06151(2024)

    2024 arXiv

  34. [42]

    Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. 2020. Distributionally robust policy evaluation and learning in offline contextual bandits. InInternational Conference on Machine Learning. PMLR, 8884–8894

    2020

  35. [43]

    Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. 2020. Doubly Robust Off-Policy Evaluation with Shrinkage. InProceedings of the 37th International Conference on Machine Learning, Vol. 119. PMLR, 9167–9176

    2020

  36. [44]

    Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. 2020. Adaptive estimator se- lection for off-policy evaluation. InInternational Conference on Machine Learning. PMLR, 9196–9205

    2020

  37. [45]

    Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. 2019. Cab: Continuous adaptive blending for policy evaluation and learning. InInternational Conference on Machine Learning, Vol. 84. 6005–6014

    2019

  38. [46]

    Adith Swaminathan and Thorsten Joachims. 2015. The Self-Normalized Estimator for Counterfactual Learning.Advances in Neural Information Processing Systems 28 (2015)

    2015

  39. [47]

    Philip Thomas and Emma Brunskill. 2016. Data-Efficient Off-Policy Policy Evalu- ation for Reinforcement Learning. InProceedings of the 33rd International Confer- ence on Machine Learning, Vol. 48. PMLR, 2139–2148

    2016

  40. [48]

    Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Du- rugkar, and Emma Brunskill. 2017. Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. InPro- ceedings of the AAAI Conference on Artificial Inte...

    2017

  41. [49]

    George Tucker and Jonathan Lee. 2021. Improved Estimator Selection for Off- Policy Evaluation.Workshop on Reinforcement Learning Theory at the 38th Inter- national Conference on Machine Learning(2021). KDD ’25, August 3–7, 2025, Toronto, ON, Canada Tatsuhiro Shimizu, et al

    2021

  42. [50]

    Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno

  43. [51]

    Masatoshi Uehara, Masahiro Kato, and Shota Yasui. 2020. Off-policy evaluation and learning for external validity under a covariate shift.Advances in Neural Information Processing Systems33 (2020), 49–61

    2020

  44. [52]

    Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudık. 2017. Optimal and adaptive off-policy evaluation in contextual bandits. InInternational Conference on Machine Learning. PMLR, 3589–3597

    2017

  45. [53]

    " Is it applicable to abrupt non-stationarity?

    Da Xu, Yuting Ye, Chuanwei Ruan, and Bo Yang. 2022. Towards robust off- policy learning for runtime uncertainty. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 10101–10109. Off-Policy Evaluation and Learning for the Future under Non-Stationarity KDD ...

    2022

  46. [57]

    and𝑝2(𝑥|𝑡)= Î 𝑑∈[𝑑𝑥]N(𝜇 2,𝜎 2 2). For the mean value𝜇1 of the component of the non-stationary context depending on the time feature𝜙𝑥 for the context, we use𝜇1 =𝛾⊤ one_hot𝜙x(t) where one_hot𝜙x(t) is the one hot embedding of the time feature for the context𝜙𝑥, and𝛾 is a paramet...

    2025

  47. [2021]

    InInternational Conference on Artificial Intelligence and Statistics

    Non-stationary off-policy optimization. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 2494–2502

  48. [2023]

    InProceed- ings of the AAAI Conference on Artificial Intelligence, Vol

    Policy-Adaptive Estimator Selection for Off-Policy Evaluation. InProceed- ings of the AAAI Conference on Artificial Intelligence, Vol. 36

  49. [2024]

    In Proceedings of the ACM on Web Conference 2024

    Off-policy evaluation for large action spaces via policy convolution. In Proceedings of the ACM on Web Conference 2024. 3576–3585

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.