Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards

Chihiro Maru; Hiroko Shoji; Tatsuki Takahashi

REVIEW 2 major objections 4 minor 9 references

Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dividing logged rewards by their observation probability restores unbiased off-policy evaluation when missing rewards depend on display position.

desk verdict A correct estimator for position-dependent missingness, but the MNAR label oversells it: the model is MAR, so the proof does not cover the headline claim. read the letter →

arxiv 2502.08993 v1 pith:HZ3C7A3F submitted 2025-02-13 stat.ML cs.LG

Tatsuki Takahashi , Chihiro Maru , Hiroko Shoji This is my paper · ORCID

classification stat.MLcs.LG

keywords off-policyevaluationmissingnotatrandompositionbiasinversepropensityscorerecommendersystemsmarginalizedimportancesamplingunbiasedlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a realistic failure mode in off-policy evaluation for recommender systems: the data used to estimate a new policy's value are logged under a known recommendation policy, but they also have position-dependent missing rewards, because items shown lower on a page are less often observed. The paper first proves that the standard marginalized inverse propensity score (MIPS) estimator, which corrects only for the logging policy, is biased under this kind of missing-not-at-random reward, with per-position bias equal to the expected reward times the probability that the position's reward is unobserved. It then proposes MIPS(w/ROIPS), which multiplies the usual logging-policy correction by a second inverse propensity score for reward observation, and proves this estimator is unbiased for the policy value whenever the no-direct-effect assumption holds. A practical consequence demonstrated in synthetic experiments is that an observation propensity estimated from the logged data alone can be plugged in and still reduce mean squared error, because the bias introduced by a heuristic correction is outweighed by a large variance reduction.

What carries the argument

The load-bearing object is the double propensity weight $$w(x_i,e_{a_i,k})/\$\theta$(o_k|x_i) = \frac{\pi(e_{a_k}|x)}{\pi_0(e_{a_k}|x)}\cdot\frac{1}{\$\theta$(o_k|x_i)},$$ applied only to logged positions with $o_{i,k}=1$. The first factor, the marginalized importance weight, removes the logging-policy bias; the second, the inverse reward-observation propensity, removes the position bias by cancelling the probability that the position is observed. The estimator's unbiasedness rests on the identity $\mathbb{E}[I\{o_k=1\}|x]=\theta(o_k|x)$ together with the generative model in which $o$ and $r$ are independent given $x$.

What would settle it

Generate logged data in which the observation probability is a function of the reward itself, for example $\theta(o_k=1|x,r_k)$ decreasing for low $r_k$, while keeping every other part of the model identical; if the resulting MIPS(w/ROIPS) estimate differs systematically from the true policy value across many replications, the conditional independence that carries the proof is violated.

Watch

Extended reading notes

Core claim

The central claim is a bias formula and its cure. Under Assumption 1 (no direct effect), meaning $E[r_k|x,a]=E[r_k|x,e_{a_k}]$, when logged rewards are observed through a position-dependent mechanism $\theta(o|x)$, the MIPS estimator has bias $$\mathrm{Bias}(\hat{V}^{(k)}_{\mathrm{MIPS}}(\pi;D)) = \mathbb{E}_{p(x)\pi(e_{a_k}|x)}[q_k(x,e_{a_k})(1-\$\theta$(o_k|x))].$$ The proposed estimator, $$\hat{V}^{(k)}_{\mathrm{MIPS(w/ROIPS)}}(\pi;D) = \frac{1}{n}\sum_{i:o_{i,k}=1} w(x_i,e_{a_i,k})\frac{r_{i,k}}{\$\theta$(o_k|x_i)},$$ with $w(x,e_{a_k})=\pi(e_{a_k}|x)/\pi_0(e_{a_k}|x)$, is then proved to satisfy $E_D[\hat{V}^{(k)}_{\mathrm{MIPS(w/ROIPS)}}]=V^{(k)}(\pi)$. The proof is a cancellation: the inverse observation propensity removes the factor $\theta(o_k|x)$ introduced by conditioning on $o_{i,k}=1$.

Load-bearing premise

The argument holds only if reward depends on the full ranking action just through its category embedding (the no-direct-effect assumption) and whether a reward is observed is conditionally independent of the reward value once the context is fixed.

Editorial extensions

If this is right

Plain MIPS systematically undervalues policies that place rewards in positions with low observation probabilities, because the expectation of the estimator is $V^{(k)}(\pi)$ minus the bias term.
With the true observation propensity, MIPS(w/ROIPS) is unbiased per position, so the sum over positions estimates the full policy value without position-bias error.
When the true propensity is unknown, a heuristic position-only estimate from logged data keeps the estimator nearly unbiased in experiments and lowers variance enough to reduce MSE by about 88.4% at the strongest position bias tested.
The bias theorem applies to all basic OPE/L estimators that implicitly assume full observation of rewards, so the missingness correction is not specific to MIPS.
Because the correction uses only logged observation indicators, it can be applied in real recommendation logs without additional data collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the observation propensity truly depends on the reward value itself, the conditional-independence premise fails and the proof no longer holds; a natural extension is to model $\theta(o_k|x,r_k)$ and quantify the residual bias.
The same double-propensity correction can be paired with a doubly robust estimator or control variate to reduce the variance that the extra inverse weight adds, especially when true propensities are used.
Plugging MIPS(w/ROIPS) into policy learning objectives would give an unbiased gradient estimator under the same assumptions, extending the correction from evaluation to learning.
Estimating $\theta$ with a context-dependent model rather than a position-only rate is a testable upgrade: it should reduce the heuristic estimator's bias at the cost of higher variance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper derives a clean bias formula for MIPS when rewards are only observed with probability θ(o|x), and proposes a corrected estimator (MIPS with ROIPS) that divides by that observation propensity. The math is right for the model it defines, and the heuristic version works in their synthetic experiments. But the paper sells this as handling missing-not-at-random rewards, and that is not what the model assumes. The generative model in §3.1 factorizes as p(x)π0(a|x)θ(o|x)p(r|x,a), which makes o conditionally independent of r given x. That is missing-at-random with respect to the observed context, not MNAR. Under true MNAR, where p(o|x,r) depends on the unobserved reward, the unbiasedness proof collapses: E[r/θ(o|x)·1{o=1}] becomes E[r·θ(o|x,r)/θ(o|x)], which is not generally E[r]. So the central claim as advertised is not established.

What is genuinely useful: the bias formula in Theorem 1 is a clear, correct statement of how position-dependent missingness biases MIPS, and the ROIPS correction is a sensible fusion of known URL and MIPS techniques. The proof of the proposition is a direct expectation calculation and holds under the stated assumptions. They also ship code, which is good practice. The synthetic experiments show the heuristic estimator can reduce MSE substantially when the true θ is unknown, which is the practically relevant scenario.

The soft spots, in proportion: (1) The MNAR terminology is the load-bearing issue. If the paper were repositioned as handling MAR position bias, the contribution is modest but honest. (2) The experiments report no error bars around the MSE, and the heuristic ROIPS is a context-free per-position average, which is crude; it works in their setup, but the claim that it balances bias and variance would be stronger with variance bands. (3) The 'no direct effect' assumption is strong and they do acknowledge it, though it is not the main problem.

Who should read this: people working on OPE for recommender logs with position-based missingness, especially those building on Saito and Joachims' MIPS. It is a useful footnote that deserves to exist.

Recommendation: send it to peer review, but ask the authors to fix the MNAR claim or the model. If the model stays MAR, the title and abstract need to change. The estimator itself is worth refereeing.

Referee Report

2 major / 4 minor

Summary. The paper studies off-policy evaluation (OPE) for recommender systems when logged data contain both logging-policy bias and display-position-induced missing rewards. It defines a position-wise policy value, reviews the marginalized IPS (MIPS) estimator, and introduces an observation indicator o into the data-generating process. The paper proves that MIPS is biased when rewards are not always observed, proposes a new estimator, MIPS(w/ROIPS), that additionally divides by the reward-observation propensity, and proves unbiasedness under a no-direct-effect assumption. Synthetic experiments compare the proposed estimator with several baselines as the strength of position bias varies, and the paper reports that the version using a heuristic observation propensity achieves the lowest mean squared error.

Significance. If the result holds for the advertised setting, the paper addresses a practically important gap: standard OPE estimators implicitly assume that all rewards are observed, which is unrealistic when display position governs visibility. The unbiasedness proof is a direct and correct calculation under the stated generative model, and the proposed estimator is a natural combination of MIPS and an observation-propensity correction. A strength is that the implementation is provided. However, the paper's headline claim of handling missing-not-at-random rewards is not supported by the assumptions actually used, because the generative model makes the observation indicator independent of the reward given the context. The experimental section also lacks error bars, so the empirical superiority is not substantiated. With an honest reframing to position-dependent missingness, the contribution is useful but incremental.

major comments (2)

[§3.1, Proposition proof and Eq. (5)] The generative model is written as (x, a, e_a, o, r) ~ p(x)π0(a|x)θ(o|x)p(r|x,a), which makes o and r conditionally independent given x (and a). This is a missing-at-random type mechanism with respect to the observed context, not a missing-not-at-random mechanism. The proof of the Proposition relies on the identity E[r_k 1{o_k=1} | x,a] = E[r_k|x,a]·P(o_k=1|x), which is valid only under that conditional independence. Under genuine MNAR, where p(o_k|x,r_k) differs from p(o_k|x), the proof fails: E[r_k 1{o_k=1}/θ(o_k|x)] = E[r_k θ(o_k|x,r_k)/θ(o_k|x)], which is not generally equal to E[r_k]. Thus the theorem and proposition do not establish unbiasedness for the MNAR setting advertised in the title and abstract; they establish unbiasedness for position-dependent missingness that depends only on the observed context. The authors should either revise the title, abstract, and terminology to describe position-dependent MAR missingness, or extend the estimator and analysis to observation mechanisms that depend on the unobserved reward.
[§4.3 and Figure 2] The main experimental conclusion that MIPS(w/heuristic ROIPS) achieves the lowest MSE is reported without error bars, confidence intervals, or significance tests across the S=100 logged-data replicates. Each plotted MSE is itself a random quantity estimated from a finite number of seeds, so the reader cannot determine whether the observed differences, such as the reported 88.4% improvement over MIPS at α=3.0, are reliable. The authors should report standard errors or interquartile ranges and, ideally, paired comparisons between estimators.

minor comments (4)

[§1] The sentence 'when both bias exits in the logged data' contains a grammatical error; it should be 'when both biases exist in the logged data'.
[§3.2] The unbiasedness Proposition applies to the estimator using the true observation propensity θ(o_k|x). The experiments, however, emphasize MIPS(w/heuristic ROIPS), which uses a context-free estimate from the logged data. The paper should state explicitly that the theoretical unbiasedness result does not apply to the heuristic version and that its performance is only empirically assessed.
[Notation throughout] The notation e_{a,k} and e_{a_k} are used inconsistently for the embedding of the action at position k; a single consistent notation would improve readability.
[§3.1, Theorem statement] The phrase 'although Assumption 1 holds when rewards are MNAR' is grammatically ambiguous. It should be rephrased to say that the bias arises even when Assumption 1 holds and rewards are missing due to display position.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the unbiasedness proof is a direct expectation calculation from explicitly stated assumptions, and no fitted quantity is relabeled as a prediction.

full rationale

The paper's central theoretical claim (Theorem and Proposition in Section 3) is derived by taking expectations under the explicitly stated generative model p(x)π0(a|x)θ(o|x)p(r|x,a) with Assumption 1. The MIPS(w/ROIPS) estimator in Eq.(5) is defined with the same θ(ok|x) that appears in the model, and the proof computes E_D[1/n Σ w r/θ] = E_{x,ea}[q · (Σ_{o':o'_k=1} θ(o'|x))/θ(ok|x)] = V^(k)(π). This is a standard importance-weighting calculation, not an assumption that the estimator is unbiased. The heuristic ROIPS used in experiments is a plug-in estimate of θ from logged data and is explicitly evaluated as an approximation rather than presented as a prediction from the theory. The paper cites prior work for the base MIPS framework, but the new estimator and its bias calculation are established within the paper. No self-citation chain, uniqueness import, or fitted-parameter-as-prediction step is load-bearing. The potential mismatch between the MAR-style generative model and the advertised MNAR setting is a correctness and scope concern, not circularity, because the derivation does not presuppose its conclusion.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The proof is transparent about its main no-direct-effect assumption, but it also depends on an observation model that is weaker than true MNAR, plus support and positivity conditions inherited or assumed without empirical verification.

assumptions (5)

domain assumption No direct effect (Assumption 1): E[r_k | x, a] = E[r_k | x, e_{a,k}].
Used in the theorem and proposition to replace ranking actions with embedding-dependent rewards. Inherited from Saito and Joachims and acknowledged as strong in the conclusion.
ad hoc to paper Observation process factorizes as theta(o|x) and is independent of reward and action given x.
Introduced in the data-generation model in Section 3.1. It makes the missingness mechanism MAR-style rather than outcome-dependent MNAR, and it is not established from real data.
domain assumption Common support for marginalized action embeddings, pi(e_a|x) > 0 implies pi0(e_a|x) > 0.
Required for the importance weight w = pi(e_a|x) / pi0(e_a|x) to be defined. Cited from prior work rather than proved here.
domain assumption Positivity of observation propensities, theta(o_k|x) > 0 whenever o_k = 1.
Required for the 1/theta(o_k|x) factor to be finite. Not checked in experiments or real data.
domain assumption Logged data are i.i.d. from p(x)pi0(a|x)theta(o|x)p(r|x,a).
Standard sampling assumption on which the expectation calculations rest.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards." pith.science (2026). https://pith.science/paper/HZ3C7A3F

@misc{pith2026250208993,
  author       = {Pith},
  title        = {Pith review of: Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZ3C7A3F}},
  note         = {Machine review of arXiv:2502.08993}
}

read the original abstract

Unbiased recommender learning (URL) and off-policy evaluation/learning (OPE/L) techniques are effective in addressing the data bias caused by display position and logging policies, thereby consistently improving the performance of recommendations. However, when both bias exits in the logged data, these estimators may suffer from significant bias. In this study, we first analyze the position bias of the OPE estimator when rewards are missing not at random. To mitigate both biases, we propose a novel estimator that leverages two probabilities of logging policies and reward observations as propensity scores. Our experiments demonstrate that the proposed estimator achieves superior performance compared to other estimators, even as the levels of bias in reward observations increases.

Figures

Figures reproduced from arXiv: 2502.08993 by the authors.

**Figure 2.** MSE, squared bias, and variance with varying levels of bias in reward observations. Note that, as shown in Eq.(4), the MSE (left) can be decomposed into the squared bias term (middle) and the variance term of the estimator (right). timated using the logged data. In practice, we cannot obtain the true observation probabilities. Specifically, we average observation ˆθ(ok|·) = 1 n Pn i=1 oi,k for each position, which… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 8 canonical work pages

[1]

Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evalu- ation

Yuta Saito et al. “Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evalu- ation”. In: arXiv preprint arXiv:2008.07146 (2020)

arXiv 2020
[2]

Top-k off-policy correction for a REINFORCE recommender system

Minmin Chen et al. “Top-k off-policy correction for a REINFORCE recommender system”. In: Proceed- ings of the Twelfth ACM International Conference on Web Search and Data Mining. 2019, pp. 456–464

work page 2019
[3]

Offline evaluation to make de- cisions about playlistrecommendation algorithms

Alois Gruson et al. “Offline evaluation to make de- cisions about playlistrecommendation algorithms”. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining . 2019, pp. 420–428

work page 2019
[4]

Bias and debias in recommender system: A survey and future directions

Jiawei Chen et al. “Bias and debias in recommender system: A survey and future directions”. In: ACM Transactions on Information Systems 41.3 (2023), pp. 1–39

work page 2023
[5]

Recommendations as treat- ments: Debiasing learning and evaluation

Tobias Schnabel et al. “Recommendations as treat- ments: Debiasing learning and evaluation”. In: in- ternational conference on machine learning. PMLR. 2016, pp. 1670–1679

work page 2016
[6]

Unbiased recommender learning from missing-not-at-random implicit feedback

Yuta Saito et al. “Unbiased recommender learning from missing-not-at-random implicit feedback”. In: Proceedings of the 13th International Conference on Web Search and Data Mining. 2020, pp. 501–509

work page 2020
[7]

Coun- terfactual risk minimization: Learning from logged bandit feedback

Adith Swaminathan and Thorsten Joachims. “Coun- terfactual risk minimization: Learning from logged bandit feedback”. In: International Conference on Machine Learning. PMLR. 2015, pp. 814–823

work page 2015
[8]

Off-Policy Eval- uation for Large Action Spaces via Embeddings

Yuta Saito and Thorsten Joachims. “Off-Policy Eval- uation for Large Action Spaces via Embeddings”. In: Proceedings of the 39th International Conference on Machine Learning. PMLR. 2022, pp. 19089–19122

work page 2022

Show all 9 references

[9]

Offline evaluation of ranking poli- cies with click models

Shuai Li et al. “Offline evaluation of ranking poli- cies with click models”. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining . 2018, pp. 1685– 1694. 4

2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evalu- ation

Yuta Saito et al. “Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evalu- ation”. In: arXiv preprint arXiv:2008.07146 (2020)

arXiv 2020

[2] [2]

Top-k off-policy correction for a REINFORCE recommender system

Minmin Chen et al. “Top-k off-policy correction for a REINFORCE recommender system”. In: Proceed- ings of the Twelfth ACM International Conference on Web Search and Data Mining. 2019, pp. 456–464

work page 2019

[3] [3]

Offline evaluation to make de- cisions about playlistrecommendation algorithms

Alois Gruson et al. “Offline evaluation to make de- cisions about playlistrecommendation algorithms”. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining . 2019, pp. 420–428

work page 2019

[4] [4]

Bias and debias in recommender system: A survey and future directions

Jiawei Chen et al. “Bias and debias in recommender system: A survey and future directions”. In: ACM Transactions on Information Systems 41.3 (2023), pp. 1–39

work page 2023

[5] [5]

Recommendations as treat- ments: Debiasing learning and evaluation

Tobias Schnabel et al. “Recommendations as treat- ments: Debiasing learning and evaluation”. In: in- ternational conference on machine learning. PMLR. 2016, pp. 1670–1679

work page 2016

[6] [6]

Unbiased recommender learning from missing-not-at-random implicit feedback

Yuta Saito et al. “Unbiased recommender learning from missing-not-at-random implicit feedback”. In: Proceedings of the 13th International Conference on Web Search and Data Mining. 2020, pp. 501–509

work page 2020

[7] [7]

Coun- terfactual risk minimization: Learning from logged bandit feedback

Adith Swaminathan and Thorsten Joachims. “Coun- terfactual risk minimization: Learning from logged bandit feedback”. In: International Conference on Machine Learning. PMLR. 2015, pp. 814–823

work page 2015

[8] [8]

Off-Policy Eval- uation for Large Action Spaces via Embeddings

Yuta Saito and Thorsten Joachims. “Off-Policy Eval- uation for Large Action Spaces via Embeddings”. In: Proceedings of the 39th International Conference on Machine Learning. PMLR. 2022, pp. 19089–19122

work page 2022