Pith. sign in

REVIEW 3 major objections 4 minor 16 references

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a policy's value in a target domain can be estimated and optimized without any exploration there, as long as logged data from sufficiently similar source domains are available and a regression model preserves cross-dom

desk verdict A genuinely new cross-domain OPE/L setup with a plausible estimator, but the central unbiasedness proof doesn't match Eq. (9) — the paper needs a corrected estimator before the theory can be trusted. read the letter →

arxiv 2607.22012 v1 pith:BGI7UAKK submitted 2026-07-24 cs.LG

Yuta Natsubori , Masataka Ushiku , Yuta Saito This is my paper · ORCID
classification cs.LG
keywords cross-domainoff-policyevaluationcontextualbanditslearningdeterministicloggingpolicynewactionscommonsupportmultipleimportanceweightingrewarddecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces cross-domain off-policy evaluation and learning (OPE/L), a setup in which a target domain with scarce, deterministic, or action-limited logged data is supplemented by logged data from source domains. Its central claim is that the proposed Cross-domain Off-Policy Evaluation (COPE) estimator can be unbiased even when the target domain's logging policy is completely deterministic and contains entirely new actions, as long as the target cluster of source domains provides support and a regression model preserves relative reward differences between domains (Corollary 3.1). The estimator splits the expected reward into a shared domain-cluster effect and a per-domain specific effect, handling the first by multiple importance weighting across the cluster and the second by target-domain regression. If the claim holds, OPE/L becomes usable in real settings — personalized medicine, recommendation, education, advertising — where standard estimators break. The paper's experiments on a recommendation dataset and synthetic data show COPE and its policy-gradient extension beating target-only and all-domain baselines, especially with few target samples.

What carries the argument

The engine is the COPE estimator (Eq. 9), built on the reward decomposition q^k(x,a) = g(x,a,phi(k)) + h(x,a,k). Its first term applies multiple importance weighting with an averaged cluster logging policy p_{phi(T)}(a|x) in the denominator, so that source-domain interactions count toward the target value without requiring target-domain exploration; its second term uses target-domain regression to correct the domain-specific component. The formal condition that makes the machinery unbiased is Conditional Pairwise Correctness (Condition 3.2): the regression model need not predict absolute rewards, only preserve the differences q^k - q^T between each source domain and the target within the clu

What would settle it

Take two domains in one cluster with deterministic logging in the target and a constant reward offset between source and target. Compute the expectation of Eq. (9) with a single regression model fit on target data only. If the estimator is implemented literally, Corollary 3.1's unbiasedness cannot hold, because the offset makes the CPC condition reduce to q_source = q_target; seeing nonzero bias in this simulation settles that the paper's stated condition does not match its estimator.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated as Corollary 3.1, is that off-policy evaluation can remain unbiased under 'common cluster support' even when the target-domain logging policy is deterministic or includes actions never seen in the target domain. COPE achieves this by decomposing the expected reward q^k(x,a) into a domain-cluster effect g(x,a,phi(k)) shared by similar domains and a domain-specific effect h(x,a,k), then estimating the cluster effect through multiple importance weighting over all logged data in the target cluster and the specific effect through a target-domain regression. Theorem 3.1 shows that the estimator's bias is governed by how far the cluster-averaged regression erro

Load-bearing premise

The load-bearing premise is Condition 3.2, Conditional Pairwise Correctness: the regression model must preserve the relative reward differences between each source domain and the target domain inside the cluster — and the estimator, as written, appears to use a single target-domain regression rather than the per-domain regressions that condition requires.

Editorial extensions

If this is right

  • Actions that the target domain never logged become evaluable as long as some source domain in the target cluster explored them.
  • Deterministic logging in the target domain stops being a barrier: the support condition applies at the cluster level, not the target level.
  • The cluster-size hyperparameter interpolates between single-domain DR and all-domain DR, giving a principled bias-variance knob.
  • The COPE policy-gradient extension turns the estimator into a learner, so new policies can be optimized to exploit new actions.
  • In the paper's evaluations, COPE's advantage grows as target data shrink, with a reported ~6.8x MSE improvement over the best baseline at one data point per action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension follows from Theorem 3.1: if relative reward differences across domains are easier to model than absolute rewards, then COPE should stay low-bias even when absolute reward predictions are poor; this could be checked by perturbing source rewards by constants and observing bias.
  • The decomposition suggests clustering domains by similarity of reward-difference structure rather than by empirical mean reward, which is the heuristic used in the paper; a learned clustering might make the method more robust.
  • The estimator's logic generalizes beyond bandits: any transfer setting where a cluster effect can be isolated from a domain-specific effect can use multiple importance weighting over the cluster, e.g., offline reinforcement learning across environments.
  • The paper's unbiasedness requires per-domain regressions consistent with the CPC condition; reconciling that requirement with the single-regression implementation is a concrete next step if the method is to be used safely in practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Cross-Domain Off-Policy Evaluation and Learning (OPE/L) for contextual bandits, a setting in which logged data from source domains are used to evaluate and learn a policy for a target domain whose logging policy may be deterministic, whose action space contains new actions, or whose target data are few-shot. The proposed estimator, COPE, is based on decomposing the per-domain expected reward into a domain-cluster effect and a domain-specific effect, applying multiple importance weighting over the target cluster for the former and reward regression for the latter. The paper states an unbiasedness result (Corollary 3.1) under a Common Cluster Support condition and a Conditional Pairwise Correctness (CPC) condition, derives a bias formula (Theorem 3.1), extends the estimator to policy gradient learning (COPE-PG), and reports experiments on KuaiRec and synthetic data.

Significance. If correct, the paper would make a meaningful contribution by broadening OPE/L to settings that violate common support in severe ways: deterministic logging policies, completely new actions, and very small target datasets. The formulation is new to the best of my knowledge, and the bias decomposition is a natural extension of the single-domain action-embedding idea from Saito et al. (2023). Strengths of the manuscript include an explicit bias formula, a milder support condition (Condition 3.1), an explicit interpretation of COPE as a spectrum between DR and DR-ALL, and an extensive empirical study on both real and synthetic data with sensitivity analyses. However, the central theoretical claim is undermined by a mismatch between the estimator as defined in Eq. (9), the regression model appearing in the proof of Theorem 3.1 and Corollary 3.1, and the regression model described in the experiments. This is a load-bearing issue rather than a presentation concern.

major comments (3)
  1. [§3.1, Eq. (9), Theorem 3.1, Appendix B.1] The estimator defined in Eq. (9) uses a single regression function \hat q_T in both the residual term evaluated on source-domain data and the target regression term. The bias formula in Theorem 3.1, however, is derived for per-domain regression errors \Delta_{q,\hat q}^k = q^k - \hat q^k, and the proof in Appendix B.1 implicitly switches to a per-domain \hat q^k without comment. Taking Eq. (9) literally and re-deriving its expectation gives E[\hat V_COPE] = E_{p_T(x)\pi(a|x)}[\sum_{k\in\phi(T)} \eta(k) q^k(x,a)], not E_{p_T(x)\pi(a|x)}[q^T(x,a)]. Unbiasedness would require \sum_k \eta(k) q^k = q^T, which is not implied by Condition 3.2 unless \hat q^k = \hat q_T, in which case Condition 3.2 collapses to q^k = q^T. Thus Corollary 3.1 does not apply to the estimator as written.
  2. [§4, paragraph on baselines] The experimental section says that Random Forest cross-fitting is used to obtain \hat q_T for DR and DM, and \hat q for DR-ALL, DM-ALL, and COPE. This describes a third regression setup, distinct from both Eq. (9) (which uses \hat q_T in the first term) and Theorem 3.1 (which requires per-domain \hat q^k). Therefore the empirical results do not validate the estimator whose unbiasedness was proved. The authors need to clarify which regression model COPE actually uses, and either adjust Eq. (9) and the proof to match the implementation or rerun the experiments with per-domain regressions.
  3. [§3.1, Eq. (10)] The quantity p_{\phi(T)}(a|x) defined in Eq. (10) is not a conditional probability distribution over a given x, because summing over a does not yield 1 in general. It is a weighted mixture of logging policies scaled by density ratios p_k(x)/p_T(x). The algebraic manipulations used in the proof of Theorem 3.1 are valid, but the notation is misleading. The paper should either define this object as an unnormalized importance-weighting denominator or introduce a normalized mixture density and state the resulting weight explicitly.
minor comments (4)
  1. [Figure 5] Typo in the caption: 'rigt' should be 'right'.
  2. [Figure 7 and Appendix D.1] In Figure 7, the caption reads 'with with varying size'; the duplicate 'with' should be removed.
  3. [§3.1, line after Eq. (8)] The decomposition q^k(x,a) = g(x,a,\phi(k)) + h(x,a,k) is introduced without specifying how g and h are identified or estimated. The paper relies on a heuristic clustering based on average rewards; a brief discussion of identifiability or a reference to the clustering literature would help.
  4. [Appendix B.1] In the proof of Theorem 3.1, the notation for \hat q changes between the estimator (\hat q_T) and the final expressions (\hat q^k). This should be made consistent, especially because the mismatch is the source of the major concern above.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; COPE's bias derivation is self-contained. The only self-citation (OffCEM) is motivational, not load-bearing. The Eq. (9) vs. Condition 3.2 mismatch is a correctness risk, not a circular step.

full rationale

The derivation chain for Theorem 3.1 is a direct computation of E[V_COPE] - V_T under Condition 3.1, using the definition of p_phi(T) in Eq. (10) and the residual definitions; it does not assume unbiasedness. Corollary 3.1 then substitutes Condition 3.2 into the bias expression, and the algebra is self-contained. The reward decomposition q^k = g + h in Eq. (8) is presented as a modeling decomposition; although the paper credits OffCEM (Saito et al., 2023) as inspiration, no theorem from that citation is used as a proof step, so the self-citation is not load-bearing. No fitted parameter is relabeled as a prediction: COPE is evaluated against held-out policy values, and all comparisons are external to the fitted regression/cluster. I also checked the flagged estimator/condition mismatch: Eq. (9) writes the source residual with \hat{q}_T and the target term with \hat{q}_T, while Condition 3.2 and Appendix B use per-domain \hat{q}^k; Section 4 further says COPE trains a single \hat{q} on all domains. If Eq. (9) is read literally, Condition 3.2 reduces to q^k = q^T, making the unbiasedness claim trivial. This is a serious internal inconsistency in the definition-proof chain, but it is a correctness flaw, not circular reasoning: the paper does not define an input in terms of the output, and the intended proof does not rely on the conclusion. The paper itself cautions in Section 3.1 that 'we do not expect Condition 3.2 to hold in practice,' so the unbiasedness statement is explicitly a theoretical benchmark. Overall, the central estimator derivation is independent of its inputs, and the circularity burden is low.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The method relies on the assumed reward decomposition (Eq. 8), common cluster support (Condition 3.1), and the CPC condition (Condition 3.2). The density ratio p_k/p_T must be known or well-estimated, and the clustering φ must reflect true shared structure. None of these is derived from first principles; they are structural assumptions about the data-generating process. No new physical/computational entities are invented, and the main tuned component is the cluster size |φ(T)| (a hyperparameter).

free parameters (1)
  • |φ(T)| (target cluster size) = 4 (default in experiments)
    Controls the bias-variance tradeoff of COPE; selected heuristically in experiments, with no principled selection rule provided.
assumptions (5)
  • ad hoc to paper Reward decomposition q^k(x,a) = g(x,a,φ(k)) + h(x,a,k) with g shared within cluster and h domain-specific (Eq. 8)
    This exact decomposition must hold for transfer to be beneficial; no empirical or theoretical justification beyond heuristic clustering.
  • domain assumption Common Cluster Support: π(a|x)>0 ⇒ p_{φ(T)}(a|x)>0 (Condition 3.1)
    Needed for the importance weights to be finite; source domains must explore every action the new policy may take.
  • ad hoc to paper Conditional Pairwise Correctness: q^k - q^T = qhat^k - qhat^T (Condition 3.2)
    The paper admits this is not expected to hold in practice; moreover it is inconsistently connected to the estimator, which uses qhat_T rather than per-domain qhat^k.
  • domain assumption Known or accurately estimated density ratio p_k(x)/p_T(x) (Eq. 10)
    The weights in p_{φ(T)} require this ratio; in experiments it is estimated via uLSIF, and estimation error is not accounted for in Theorem 3.1.
  • domain assumption Domain clustering φ(k) is given or reliably estimable from logged data
    The paper uses empirical average rewards as domain embeddings; any clustering error can break the shared-cluster-effect assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits." pith.science (2026). https://pith.science/paper/BGI7UAKK

@misc{pith2026260722012,
  author       = {Pith},
  title        = {Pith review of: Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGI7UAKK}},
  note         = {Machine review of arXiv:2607.22012}
}
read the original abstract

Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the target domain in which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations.

Figures

Figures reproduced from arXiv: 2607.22012 by the authors.

Figure 1
Figure 1. Comparison of Conventional OPE/L and Cross-Domain OPE/L (Ours). There have been several previous efforts to enable OPE (Felicioni et al., 2022) and OPL (Sachdeva et al., 2020) under the violation of common support (also known as support deficiency), but they cannot handle deterministic logging policies and completely new actions. It might also be useful to use some structure in the action or reward space to relax th… view at source ↗
Figure 2
Figure 2. MSE(left), Squared Bias(center), and Variance(right) with varying ratios of new actions in the target domain [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. MSE(left), Squared Bias(center), and Variance(right) with varying numbers of users whose logging policy is deterministic in the target domain [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: MSE(left), Squared Bias(center), and Variance(right) with varying (per-action) logged data sizes in the target domain. We sample an action a k based on the domain-specific logging policy π k 0 , which is defined as below. π k 0 (a | xu) := exp(β k · (q(xu, a) + ηu,a)) …
Figure 5
Figure 5. Figure 5: Comparison of the (left) the test policy values V T (π) (normalized by V (π0)), (center) the test policy values within new actions, (rigt) the normalized test policy values within new actions, under varying ratios of new actions in the target domain [PITH_FULL_IMAGE:f…
Figure 6
Figure 6. Figure 6: Comparison of the test policy values V T (π) (normalized by V (π0)) of the OPL methods under (left) varying numbers of users whose logging is deterministic, (center) varying (per-action) training data sizes in the target domain, (right) varying sizes of the target clus…
Figure 7
Figure 7. Figure 7: MSE(left), Squared Bias(center), and Variance(right) with with varying size of the target cluster. which of IPS and DM is better, because the comparison depends on many (potentially unknown) parameters such as the reward noise and logged data size. This is why there ex…
Figure 8
Figure 8. Figure 8: MSE(left), Squared Bias(center), and Variance(right) with varying ratios of new actions in the target domain [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: MSE(left), Squared Bias(center), and Variance(right) with varying percentages of sam￾ples with a deterministic logging [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: MSE(left), Squared Bias(center), and Variance(right) with varying logged data sizes in the target domain. target logged data is smaller than the percentile point, we apply a deterministic logging policy; oth￾erwise, we apply a probabilistic logging policy defined by E…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 linked inside Pith

  1. [1]

    In contrast, the evaluation policyπis defined as π(a|x) = (1−ϵ)·I{a= argmax a′∈A q(x, a′, e;λ)}+ ϵ |A| ,(17) whereϵ∈[0,1]controls the quality ofπand we setϵ= 0.2as default

    18 Published as a conference paper at ICLR 2025 Table 1: MSE in OPE for varying ratios of new actions in the target domain ratios of new actions 0.0 0.2 0.4 0.6 0.8 COPE(Ours) 1.1641 1.1647 0.8066 1.0663 1.0801 DR(Target Domain) 4.3283 4.2225 2.3985 2.9498 3.7964 DR(ALL Domain) 2.7582 2.2401 3.5354 3.3172 1.4530 IPS(Target Domain) 5.6442 4.9797 3.6800 4.8...

    arXiv 2025

  2. [8]

    Logarithmic smoothing for pes- simistic off-policy evaluation, selection and learning.arXiv preprint arXiv:2405.14335,

    Otmane Sakhi, Imad Aouali, Pierre Alquier, and Nicolas Chopin. Logarithmic smoothing for pes- simistic off-policy evaluation, selection and learning.arXiv preprint arXiv:2405.14335,

    arXiv

  3. [9]

    In the experiment, we calculate the percentages (20%, 40%, 60%, 80%) of a particular one-dimensional context in the target domain. Then, if the value of the context in the 20 Published as a conference paper at ICLR 2025 Figure 8: MSE(left), Squared Bias(center), and Variance(right) with varying ratios of new actions in the target domain. Figure 9: MSE(lef...

    2025

  4. [10]

    A review of off-policy evaluation in rein- forcement learning.arXiv preprint arXiv:2212.06355,

    Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning.arXiv preprint arXiv:2212.06355,

    arXiv

  5. [13]

    Our main motivation is to solve the prevalent problem of (completely) deterministic logging and new actions, issues that pessimistic techniques do not aim to address

    because they are not relevant to our context. Our main motivation is to solve the prevalent problem of (completely) deterministic logging and new actions, issues that pessimistic techniques do not aim to address. However, our proposed method could easily be combined with a pessimistic approach if one wants to do so. 14 Published as a conference paper at I...

    2025

  6. [14]

    cluster sizes, as seen in the OPL experiment. 17 Published as a conference paper at ICLR 2025 Moreover, we exclude¯q(u)from the definition of the user distributionp k(u), following an ob- servation that this might favor our methods. Instead, we define a linear function of the user featuref k(u) =β kxu, where the coefficient vectorβ k is sampled from a nor...

    2025

  7. [2001]

    Learning action embeddings for off-policy evaluation

    Matej Cief, Jacek Golebiowski, Philipp Schmidt, Ziawasch Abedjan, and Artur Bekasov. Learning action embeddings for off-policy evaluation. InEuropean Conference on Information Retrieval, pp. 108–122. Springer, 2024a. Matej Cief, Michal Kompan, and Branislav Kveton. Cross-validated off-policy evaluation.arXiv preprint arXiv:2405.15332, 2024b. Miroslav Dud´...

    arXiv

  8. [2011]

    Offline policy evaluation in large action spaces via outcome-oriented action grouping

    Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang, Jian Pei, and Peng Cui. Offline policy evaluation in large action spaces via outcome-oriented action grouping. InProceedings of the ACM Web Conference 2023, pp. 1220–1230. ACM,

    2023

Show all 16 references
  1. [2012]

    Counterfactual risk minimization: Learning from logged bandit feedback

    12 Published as a conference paper at ICLR 2025 Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. InProceedings of the 32nd International Conference on Machine Learn- ing, volume 37, pp. 814–823. PMLR, 2015a. Adith...

    2025

  2. [2017]

    Pengzhou Wu and Kenji Fukumizu.β-intact-vae: Identifying and estimating causal effects under limited overlap.arXiv preprint arXiv:2110.05225,

    arXiv

  3. [2019]

    To- wards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems

    11 Published as a conference paper at ICLR 2025 Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. To- wards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. ...

    2025

  4. [2020]

    Off-policy evaluation for large action spaces via policy convolution

    Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off-policy evaluation for large action spaces via policy convolution. InProceedings of the ACM on Web Conference 2024, pp. 3576–3585. ACM,

    2024

  5. [2021]

    Cross-fitting and fast remainder rates for semiparametric estimation.arXiv preprint arXiv:1801.09138,

    Whitney K Newey and James R Robins. Cross-fitting and fast remainder rates for semiparametric estimation.arXiv preprint arXiv:1801.09138,

    arXiv

  6. [2022]

    Autoope: Automated off-policy estimator selection.arXiv preprint arXiv:2406.18022,

    Nicol`o Felicioni, Michael Benigni, and Maurizio Ferrari Dacrema. Autoope: Automated off-policy estimator selection.arXiv preprint arXiv:2406.18022,

    arXiv

  7. [2023]

    Towards assessing and benchmarking risk-return tradeoff of off-policy evaluation

    Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. Towards assessing and benchmarking risk-return tradeoff of off-policy evaluation. InThe Twelfth International Conference on Learning Representations, 2024a. Haruka Kiyohara, Masahi...

    2024

  8. [2024]

    13 Published as a conference paper at ICLR 2025 A RELATEDWORK Off-policy evaluation (OPE) and learning (OPL) have gained particular attention in contextual ban- dit settings as they offer a safe and cost-efficient alternative to online A/B testing (Mehrotra et al., 2018; Gilot...

    2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.