Pith. sign in

REVIEW 3 major objections 4 minor 67 references

A/B-test variance can be cut for free whenever the two versions overlap, by reweighting the same logs with a meta-policy importance weight—and the paper proves the resulting estimator never does worse than difference-in-means.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:37 UTC pith:BENABQBN

load-bearing objection The variance-dominance theorem is real, but the DR extension is overclaimed and the 'free lunch' needs exact propensities. the 3 major comments →

arxiv 2607.14604 v1 pith:BENABQBN submitted 2026-07-16 cs.LG cs.IRstat.ME

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Olivier Jeunen This is my paper
classification cs.LG cs.IRstat.ME
keywords A/B testingoff-policy estimationvariance reductioncounterfactual estimationpolicy overlapaverage treatment effectimportance weightingrecommender systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that standard A/B tests waste statistical power whenever the treatment and control policies sometimes choose the same action: those agreement samples carry no information about the treatment effect, yet the classic difference-in-means estimator gives them full weight. The fix is to view random treatment assignment as a mixture policy and reweight each logged action by the divergence between the two policies divided by that mixture, a Δ-off-policy estimate of the average treatment effect. The paper proves a dominance theorem: the reweighted estimator's variance is never above difference-in-means, and is strictly lower whenever the policies share support and outcomes are noisy there, regardless of the traffic split. It also derives a variance-optimal traffic allocation, a model-training objective that targets disagreement regions, and a ranking-policy estimator. If correct, A/B tests comparing similar models can finish sooner, or report the same precision with less traffic, at negligible engineering cost.

Core claim

The paper's central claim is that any A/B test between two known stochastic policies can be re-analysed with the same logged data to yield an unbiased estimate of the average treatment effect whose variance is never worse—and usually strictly better—than the standard difference-in-means estimate. The mechanism replaces the assignment weights 1/p and −1/(1−p) with the policy-aware weight (π(a|x) − π'(a|x)) / (pπ(a|x) + (1−p)π'(a|x)), where p is the fraction of traffic assigned to the treatment. The variance gap decomposes pointwise as αγ/(p(1−p)π₀²) times the squared outcome residual, which is nonnegative whenever both policies place positive mass on the same action. Hence the estimator's var

What carries the argument

The central object is the meta-policy π₀(a|x) = pπ(a|x) + (1−p)π'(a|x), the mixture over the two deployed policies induced by random treatment assignment. Its companion importance weight wΔ = (π − π')/π₀ turns A/B-test data into an off-policy estimator of the treatment effect. The proof of dominance rests on a pointwise identity: the difference between the squared difference-in-means weight and wΔ² equals αγ/(p(1−p)π₀²), where α = π(a|x) and γ = π'(a|x). Because this quantity is nonnegative, the variance gap is the expectation of a nonnegative quantity times (Y − β*)², which is why the reweighted estimator dominates when policies overlap.

Load-bearing premise

The load-bearing premise is that the action probabilities π(a|x) and π'(a|x) for both deployed policies are known exactly and correctly specified—as the paper's own limitations section concedes, if these propensities are misspecified or numerically unstable, the reweighted estimator becomes biased and the variance-dominance guarantee no longer follows.

What would settle it

On logged data where all propensities are known, compute both weights, estimate the pointwise variance gap E[(wDiM² − wΔ²)(Y − β*)²] using cross-fitting, and compare the resulting ATE estimates. A negative gap in any region where π(a|x) > 0 and π'(a|x) > 0, or a material divergence between the reweighted estimate and the difference-in-means due to propensity misspecification, would refute the theorem's dominance claim in that setting.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can obtain tighter confidence intervals for the same sample size—or reach significance with less traffic—simply by reweighting existing A/B logs using known policy probabilities.
  • The variance-optimal treatment split is generally not 50/50; it depends on the divergence between policies and can be estimated from pre-experiment data, favouring the higher-entropy policy when supports are asymmetric.
  • The doubly robust extension inherits the dominance guarantee and adds further variance reduction from a reward model; the Δ-MRDR objective concentrates model capacity on disagreement regions, lowering ATE variance more than global mean-squared-error training.
  • For ranking policies, the exposure-based Δ-DCG estimator preserves the overlap benefit, and rank-specific optimal baselines prevent the variance explosion that unbaselined importance weighting suffers at high divergence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference, not the paper's claim: the variance-gap formula suggests a cheap pre-experiment diagnostic—compute E[αγ/π₀²] on logged data to predict how much power an overlap-aware analysis would add before committing traffic to a live test.
  • Inference, not the paper's claim: the dominance guarantee requires exactly known action probabilities, so for large-language-model or huge-action deployments where exact token-level probabilities are intractable, the practical route is approximate-propensity estimation or truncation methods whose bias would need explicit accounting.
  • Inference, not the paper's claim: because the estimator stays unbiased under time-varying logging policies with known propensities, the same reweighting logic could support sequential or adaptive experiments that update the traffic split mid-test without inflating false-positive rates—an idea the paper sketches but does not fully validate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a new protocol for A/B testing in which the random assignment of users to treatment or control is viewed as a meta-policy, and Δ-Off-Policy Estimation (Δ-OPE) is used to estimate the Average Treatment Effect (ATE) between two stochastic policies. The main theoretical claim is that the policy-aware Δβ*-IPS estimator, using weights wΔ=(π−π′)/(pπ+(1−p)π′), has variance no greater than the standard Difference-in-Means (DiM) estimator, with strict improvement when the policies have common support and the overlap region has nonzero residual variance. The paper also derives an optimal traffic allocation ratio, proposes a Δ-MRDR objective for training reward models, extends the framework to ranking policies via Δ-DCG, and validates the methods through synthetic simulations.

Significance. If the central variance-dominance theorem is correct, the paper offers a simple, theoretically grounded way to reduce the sample size required for A/B tests when the treatment and control policies are known stochastic objects. The proof of Theorem 3.1 is self-contained and the algebra appears sound under the stated assumptions (known propensities, i.i.d. sampling, fixed allocation). The paper also provides reproducible code and careful simulation studies that exhibit the predicted ordering. The main weaknesses are the incorrect generality claimed for Theorem 3.2, the unjustified variance-minimization derivation for Δ-MRDR, and the reliance on exactly known propensities in settings where the abstract claims practical applicability.

major comments (3)
  1. [§3.1, Theorem 3.2 (Eqs. 20–22)] The theorem is stated for any reward model f:X×A→R, but RADiM as defined in Eq. (3) uses f(x_i) only, not f(x_i,a_i). For an action-dependent f, RADiM is not unbiased for the ATE: its expectation is V(π)−V(π′) − (E_{π}[f(X,A)]−E_{π′}[f(X,A)]), so the proof's opening assumption that 'both estimators are unbiased' fails. The shared-control-variate argument also collapses because the residual used by RADiM is Y−f(X), while Δ-DR uses Y−f(X,A). The proof is only valid when f is independent of the action. This is load-bearing for the claimed Δ-DR dominance over RADiM and for the motivation of Δ-MRDR. Please either restrict Theorem 3.2 to context-only f, or define an action-dependent 'RADiM' variant that includes a bias-correction term (in which case the comparison becomes tautological).
  2. [§3.3, Eq. (24)] The derivation of the Δ-MRDR objective is incomplete. The variance of the Δ-DR estimator includes Var(τ̂(X)), where τ̂(x)=E_{a∼π}[f(x,a)]−E_{a∼π′}[f(x,a)], and a covariance term between τ̂(X) and wΔ(Y−f(X,A)). The text says these are 'neglected' because τ̂ is constant with respect to action assignment given the context, but τ̂ varies with X and depends on f, so its variance and covariance are not constant when optimizing f. Thus minimizing E[wΔ²(Y−f)²] is not shown to be equivalent to minimizing the estimator variance. The empirical results are suggestive, but the theoretical claim that Δ-MRDR 'directly minimises variance' is not established. A correct variance decomposition is needed to justify the WLS objective.
  3. [§4.5 and Abstract] The main dominance result, Theorem 3.1, requires exact knowledge of π(a|x) and π′(a|x). Section 4.5 concedes that in LLM and large-action settings these probabilities may be expensive or numerically unstable, and explicitly states that sensitivity to misspecification 'warrants further empirical investigation.' Since the abstract highlights LLM interfaces and recommender systems as target applications, the headline claim of a 'free variance reduction' is not established for the settings where propensities are not known exactly. A small error in π or π′ can create large importance-weight errors precisely in the low-support regions where the claimed variance reduction is largest. Please provide a misspecification analysis (e.g., an upper bound on bias and MSE as a function of propensity error) or substantially narrow the applicability claims to settings with known propensities.
minor comments (4)
  1. [Abstract] The phrasing 'variance scales with the divergence between policies rather than raw outcome variance' is imprecise. The variance in Theorem 3.1 still scales with E[(Y−β*)²|x,a], multiplied by the squared importance weight. It is the combination of divergence and outcome variance that matters. Please rephrase to avoid overstatement.
  2. [§2.2, Eq. (10)] The displayed formula for w_DiM is notationally confusing: the denominator appears to be π0(T=π) for both indicator terms, which is not correct for the second term. Rewrite using explicit case distinctions (1/p if T=π, −1/(1−p) if T=π′) to avoid ambiguity.
  3. [§3.4, Δ-DCG] The marginal importance weight wΔ(x,a_j) uses exposure propensities that combine ranking policy and position bias. The unbiasedness argument should explicitly cite the conditions under which the PBM exposure model is correct (e.g., independence across positions), as the estimator otherwise incurs bias that is not analyzed here.
  4. [Figure 2] The y-axis labels overlap ('Variance' and '/Theoretical Variance'), making the figure hard to read. Please separate the two curves or use a two-axis layout.

Circularity Check

0 steps flagged

No significant circularity; Theorem 3.1 is proved from the estimator definitions by direct algebra.

full rationale

The central variance-dominance claim is self-contained. Theorem 3.1 starts from the definitions of DiM (Eq. 2) and Δβ*-IPS (Eqs. 5–7 with wΔ from Eq. 12) and computes the per-sample squared-weight gap as Δw² = αγ/[p(1−p)π0²] ≥ 0 (Eqs. 14–19), yielding Var(DiM) − Var(Δβ*-IPS) = E[Δw²(Y−β*)²] ≥ 0. This is pure algebra from the stated propensities; no parameter is fitted to data to obtain the dominance inequality, and β* is an oracle quantity whose estimation is deferred to cross-fitting outside the theorem. Theorem 3.2 is presented as a direct consequence of the same pointwise weight identity applied to residuals. The empirical sections validate the predicted ordering using synthetic ground truth, not by fitting a parameter and then reporting a re-scored version of the same fit. The paper does cite the author's prior work ([26], [32], [20], [29], [31]) for the Δ-OPE framework, the DiM/Δβ*-IPS equivalence, and DCG-as-OPE, but those results are parameter-free algebraic identities that are also stated or re-derived in the text (Eqs. 10–12); they do not smuggle in the theorem's conclusion. Section 4.5 candidly flags propensity misspecification, exact-probability cost, and stationarity as limitations; these are scope and robustness caveats, not circular steps. Therefore no load-bearing reduction to inputs was found.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

No free constants are needed for Theorem 3.1; the only fitted quantities are control-variate baselines and reward models in the extensions. The central claim depends on assumptions of known propensities and no interference rather than on fitted parameters.

free parameters (3)
  • β* (scalar baseline for Δβ*-IPS)
    Variance-minimizing baseline estimated from logged data via cross-fitting (Eq. 7). Theorem uses oracle β*; finite-sample estimation adds variance not analyzed.
  • Reward model f_θ (Δ-DR / Δ-MRDR)
    Trained on logged data to minimize MSE or weighted MSE. Not fitted to the ATE itself, but controls estimator variance and is estimated from the same logs with sample splitting.
  • Position-specific baselines β*_{⊥,j} for Δ-DCG
    Estimated from logged data; a rank-wise approximation to the globally optimal joint baseline, assuming rank independence.
axioms (6)
  • domain assumption SUTVA / no interference: outcome depends only on context x and action a, not on other units' assignments.
    Throughout, E[Y|X=x,A=a] is used without spillover or network effects; standard in OPE but not verified for online marketplaces.
  • domain assumption Logging assignment is exactly the mixture π0(a|x)=pπ(a|x)+(1−p)π'(a|x) with known p.
    Eq. (11); needed for unbiasedness of the importance weights.
  • domain assumption Policy probabilities π(a|x) and π'(a|x) are known and correct.
    Required to form wΔ; Section 4.5 explicitly flags this as a limitation.
  • domain assumption For p*, stationarity of P(X) and, for the analytic Δ-IPS solution, homoskedasticity of outcomes.
    Section 3.2 and Section 4.5 acknowledge these assumptions.
  • domain assumption For ranking estimators, the Position-Based Model with known exposure propensities and rank independence.
    Section 3.4 relies on assumptions adapted from Jeunen et al. [31].
  • standard math Convexity of f(y)=1/y and the probabilistic weight identities used in the proof.
    Used to prove J(p) is strictly convex and the pointwise weight gap is non-negative.

pith-pipeline@v1.3.0-alltime-deepseek · 18913 in / 19539 out tokens · 208406 ms · 2026-08-02T01:37:29.356689+00:00 · methodology

0 comments
read the original abstract

Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage $\Delta$-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.

Figures

Figures reproduced from arXiv: 2607.14604 by Olivier Jeunen.

Figure 1
Figure 1. Figure 1: Mean Squared Error (MSE) of the standard DiM estimator and the Policy-Aware [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualising optimal treatment allocation (𝑝 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Empirical variance of top-𝐾 ranking estimators across varying levels of policy divergence (1 − 𝛼), shown on a logarithmic scale. The standard A/B testing procedure (DiM) suffers from high variance regardless of policy overlap. The unbaselined Δ-DCG estimator performs well under high over￾lap but suffers from severe variance explosion as divergence increases. The Δ𝛽 ★-DCG estimator utilises position-specifi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 9 linked inside Pith

  1. [1]

    Shubham Baweja, Neeti Pokharna, Aleksei Ustimenko, and Olivier Jeunen. 2024. Variance Reduction in Ratio Metrics for Efficient Online Experiments. InProc. of the 46th European Conference on Information Retrieval (ECIR ’24). Springer

    2024

  2. [2]

    Charles, D

    Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising.Journal of Machine Learning Research14, 101 (2013), 3207–3260. http://jmlr.org/papers/v14/bottou13a.html

    2013

  3. [3]

    Alexander Buchholz, Ben London, Giuseppe Di Benedetto, Jan Malte Lichtenberg, Yannik Stein, and Thorsten Joachims. 2024. Counterfactual Ranking Evaluation with Flexible Click Models. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). ACM, 1200–1210. doi:10.1145/3626772.3657810

    arXiv 2024

  4. [5]

    Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brun- skill, and Philip S. Thomas. 2021. Universal Off-Policy Evaluation. InAd- vances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 27475–27490. https://proceedings.neurips.cc/paper_files/paper/2021/file/ e71e5cd119bbc5797164fb0cd7fd94a4-Paper.pdf

    2021

  5. [6]

    Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019. Top-k off-policy correction for a REINFORCE recommender system. InProc. of the Twelfth ACM International Conference on Web Search and Data Mining. 456–464

    2019

  6. [7]

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2018. Dou- ble/debiased machine learning for treatment and structural parameters. The Econometrics Journal21, 1 (01 2018), C1–C68. doi:10.1111/ectj.12097 arXiv:https://academic.oup.com/ectj/article-pdf/21/1/C1/27684918/ectj00c1.pdf

    doi:10.1111/ectj.12097 2018

  7. [8]

    Stevens, Tatiana Xifara, and Amit Gandhi

    Alex Deng, Luke Hagar, Nathaniel T. Stevens, Tatiana Xifara, and Amit Gandhi

  8. [10]

    Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. InProc. of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13). ACM, 123–132. doi:10.1145/2433396.2433413

    arXiv 2013

  9. [11]

    Ekstrand, Asia J

    Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM ’20). ACM, 275–284. doi:10.1145/3340531.3411962

    arXiv 2020

  10. [12]

    Pavel Dmitriev, Somit Gupta, Dong Woo Kim, and Garnet Vaz. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Con- trolled Experiments. InProc.of the 23rd ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining (KDD ’17). ACM, 1427–1436. doi:10.1145/3097983.3098024

    arXiv 2017

  11. [13]

    Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly Robust Policy Evaluation and Optimization.Statist. Sci.29, 4 (2014), 485–511

    2014

  12. [14]

    Víctor Elvira, Luca Martino, David Luengo, and Mónica F. Bugallo. 2019. Gen- eralized Multiple Importance Sampling.Statist. Sci.34, 1 (2019), 129 – 155. doi:10.1214/18-STS668

    doi:10.1214/18-sts668 2019

  13. [15]

    Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. 2018. More Robust Doubly Robust Off-policy Evaluation. InProc. of the 35th International Conference on Machine Learning (Proc. of Machine Learning Research, Vol. 80). PMLR, 1447–1456. https://proceedings.mlr.press/v80/farajtabar18a.html

    2018

  14. [16]

    Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline A/B Testing for Recommender Systems. InProc. of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM ’18). ACM, 198–206. https://doi.org/10.1145/3159652.3159687

    arXiv 2018

  15. [17]

    Alois Gruson, Praveen Chandar, Christophe Charbuillet, James McInerney, Samantha Hansen, Damien Tardieu, and Ben Carterette. 2019. Offline Eval- uation to Make Decisions About Playlist Recommendation Algorithms. InProc. of the Twelfth ACM International Conference on Web Search and Data Mining (WSDM ’19). ACM, 420–428. doi:10.1145/3289600.3291027

    arXiv 2019

  16. [18]

    Yongyi Guo, Dominic Coey, Mikael Konutgan, Wenting Li, Chris Schoener, and Matt Goldman. 2021. Machine Learning for Variance Reduction in Online Exper- iments. InAdvances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 8637–8648

    2021

  17. [19]

    Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, and Harrie Oosterhuis

  18. [20]

    Shashank Gupta, Olivier Jeunen, Harrie Oosterhuis, and Maarten de Rijke. 2024. Optimal Baseline Corrections for Off-Policy Contextual Bandits. InProc. of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM, 722–732

    2024

  19. [22]

    2021.Offline Approaches to Recommendation with Online Success

    Olivier Jeunen. 2021.Offline Approaches to Recommendation with Online Success. Ph. D. Dissertation. University of Antwerp

    2021

  20. [23]

    Edward L. Ionides. 2008. Truncated Importance Sampling.Journal of Computa- tional and Graphical Statistics17, 2 (2008), 295–311

    2008

  21. [24]

    Olivier Jeunen. 2025. Counterfactual Inference under Thompson Sampling. arXiv:2504.08773 [cs.IR]

    Pith/arXiv arXiv 2025

  22. [25]

    Olivier Jeunen. 2023. A Common Misassumption in Online Experiments with Machine Learning Models.SIGIR Forum57, 1, Article 13 (dec 2023), 9 pages. doi:10.1145/3636341.3636358

    arXiv 2023

  23. [26]

    Olivier Jeunen. 2026. Unifying On- and Off-Policy Variance Reduction Methods. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26). arXiv:2603.08370

    arXiv 2026

  24. [27]

    Olivier Jeunen. 2025. t-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing. InProceedings of the Nineteenth ACM Conference on Rec- ommender Systems (RecSys ’25). ACM, 1307–1310. doi:10.1145/3705328.3759307

    arXiv 2025

  25. [28]

    Olivier Jeunen and Bart Goethals. 2020. An Empirical Evaluation of Doubly Robust Learning for Recommendation. InREVEAL Workshop at ACM RecSys ’20 (REVEAL ’20)

    2020

  26. [29]

    Olivier Jeunen, Shubham Baweja, Neeti Pokharna, and Aleksei Ustimenko. 2024. Powerful A/B-Testing Metrics and Where to Find Them. InProc.of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM, 816–818. doi:10.1145/ 3640457.3688036

    arXiv 2024

  27. [31]

    Olivier Jeunen and Shashank Gupta. 2026. Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation. InProceedings of the 49th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26). arXiv:2602.14914

    Pith/arXiv arXiv 2026

  28. [32]

    Olivier Jeunen and Aleksei Ustimenko. 2024. Δ-OPE: Off-Policy Estimation with Pairs of Policies. InProc. of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM, 878–883. doi:10.1145/3640457.3688162

    arXiv 2024

  29. [34]

    Thorsten Joachims, Ben London, Yi Su, Adith Swaminathan, and Lequn Wang

  30. [35]

    Olivier Jeunen and Aleksei Ustimenko. 2024. Learning Metrics that Maximise Power for Accelerated A/B-Tests. InProc.of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24). ACM, 5183–5193. https: //doi.org/10.1145/3637528.3671512 Olivier Jeunen

    arXiv 2024

  31. [36]

    Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. 2017. Peeking at A/B Tests: Why it matters, and what to do about it. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17). ACM, 1517–1525. doi:10.1145/3097983.3097992

    arXiv 2017

  32. [37]

    Nathan Kallus, Yuta Saito, and Masatoshi Uehara. 2021. Optimal Off-Policy Evaluation from Multiple Logging Policies. InProceedings of the 38th Inter- national Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 5247–5256. https://proceedings.mlr.press/v139/kallus21a.html

    2021

  33. [38]

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. InProc. of the Tenth ACM International Conference on Web Search and Data Mining (WSDM ’17). ACM, 781–789. doi:10. 1145/3018661.3018699

    arXiv 2017

  34. [39]

    Ron Kohavi and Nanyu Chen. 2024. False Positives in A/B Tests. InProc.of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24). ACM, 5240–5250. doi:10.1145/3637528.3671631

    arXiv 2024

  35. [40]

    Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. 2014. Seven rules of thumb for web site experimenters. InProc.of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’14). ACM, 1857–1866. doi:10.1145/2623330.2623341

    arXiv 2014

  36. [41]

    Eugene Kharitonov, Alexey Drutsa, and Pavel Serdyukov. 2017. Learning Sen- sitive Combinations of A/B Test Metrics. InProc. of the Tenth ACM Interna- tional Conference on Web Search and Data Mining (WSDM ’17). ACM, 651–659. doi:10.1145/3018661.3018708

    arXiv 2017

  37. [42]

    2020.Trustworthy online controlled experi- ments: A practical guide to A/B testing

    Ron Kohavi, Diane Tang, and Ya Xu. 2020.Trustworthy online controlled experi- ments: A practical guide to A/B testing. Cambridge University Press

    2020

  38. [43]

    Koki Konishi, Masataka Ushiku, and Yuta Saito. 2026. A More Accurate Al- gorithm Comparison through A/B Testing using Offline Evaluation Methods. arXiv:2607.01958 [cs.LG] https://arxiv.org/abs/2607.01958

    Pith/arXiv arXiv 2026

  39. [44]

    Ron Kohavi, Alex Deng, and Lukas Vermeer. 2022. A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. InProc. of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22). ACM, 3168–3177. https://doi.org/10.1145/3534678.3539160

    arXiv 2022

  40. [45]

    Jeffrey Li, Yixin Tang, and Jared Bauman. 2020. Improving Experimental Power through Control Using Predictions as Covariate (CUPAC). DoorDash Engineering Blog. https://doordash.engineering/2020/06/08/improving-experimental-power- through-control-using-predictions-as-covariate-cupac/

    2020

  41. [46]

    Muthukrishnan, Vishwa Vinay, and Zheng Wen

    Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S. Muthukrishnan, Vishwa Vinay, and Zheng Wen. 2018. Offline Evaluation of Ranking Policies with Click Models. InProc. of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18). ACM, 1685–1694. doi:10.1145/3219819. 3220028

    doi:10.1145/3219819 2018

  42. [47]

    Damien Lefortier, Adith Swaminathan, Xiaotao Gu, Thorsten Joachims, and Maarten de Rijke. 2016. Large-scale Validation of Counterfactual Learning Meth- ods: A Test-Bed. InNIPS What If Workshop on Inference and Learning of Hypotheti- cal and Counterfactual Interventions in Complex Systems. arXiv:1612.00367 [cs.LG]

    Pith/arXiv arXiv 2016

  43. [48]

    Harrie Oosterhuis and Maarten de Rijke. 2020. Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking. InProceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (ICTIR ’20). ACM, 137–144. doi:10.1145/3409256.3409820

    arXiv 2020

  44. [49]

    Harrie Oosterhuis and Maarten de Rijke. 2021. Unifying Online and Coun- terfactual Learning to Rank: A Novel Counterfactual Estimator that Effec- tively Utilizes Online Interventions. InProceedings of the 14th ACM Interna- tional Conference on Web Search and Data Mining (WSDM ’21). ACM, 463–471. doi:10.1145/3437963.3441794

    arXiv 2021

  45. [50]

    Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-Aware Unbiased Learning to Rank for Top-k Rankings. InProc. of the 43rd International ACM SIGIR Con- ference on Research and Development in Information Retrieval (SIGIR ’20). ACM, 489–498. doi:10.1145/3397271.3401102

    arXiv 2020

  46. [51]

    Art B. Owen. 2013.Monte Carlo theory, methods and examples

    2013

  47. [52]

    Alexey Poyarkov, Alexey Drutsa, Andrey Khalyavin, Gleb Gusev, and Pavel Serdyukov. 2016. Boosted Decision Tree Regression Adjustment for Variance Reduction in Online Controlled Experiments. InProc. of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, 235–244. doi:10.1145/2939672.2939688

    arXiv 2016

  48. [53]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human f...

    2022

  49. [54]

    Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alexandre Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Tóth, and Sam Work. 2025. Tapered Off-Policy REINFORCE - Stable and efficient reinforcement learning for large language models. InThe Thirty-ninth Annual Conference on Neural Information Processing ...

    2025

  50. [55]

    Hitesh Sagtani, Madan Gopal Jhawar, Rishabh Mehrotra, and Olivier Jeunen

  51. [56]

    Lee Richardson, Alessandro Zito, Dylan Greaves, and Jacopo Soriano. 2023. Pareto optimal proxy metrics. arXiv:2307.01000 [stat.ME]

    Pith/arXiv arXiv 2023

  52. [57]

    Yuta Saito and Thorsten Joachims. 2021. Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent Advances. InProc. of the 15th ACM Conference on Recommender Systems (RecSys ’21). ACM, 828–830. doi:10.1145/3460231.3473320

    arXiv 2021

  53. [58]

    Yuta Saito and Thorsten Joachims. 2022. Off-Policy Evaluation for Large Action Spaces via Embeddings. InProc. of the 39th International Conference on Machine Learning (Proc. of Machine Learning Research, Vol. 162). PMLR, 19089–19122. https://proceedings.mlr.press/v162/saito22a.html

    2022

  54. [59]

    Ad-load Balancing via Off-policy Learning in a Content Marketplace. In Proc. of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). ACM, 586–595. doi:10.1145/3616855.3635846

    arXiv

  55. [60]

    Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. 2021. Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation. InProc. of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1

    2021

  56. [61]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  57. [62]

    Adith Swaminathan and Thorsten Joachims. 2015. Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization.Journal of Ma- chine Learning Research16, 52 (2015), 1731–1755. http://jmlr.org/papers/v16/ swaminathan15a.html

    2015

  58. [63]

    Yuta Saito, Qingyang Ren, and Thorsten Joachims. 2023. Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. InProc. of the 40th Inter- national Conference on Machine Learning (Proc. of Machine Learning Research, Vol. 202). PMLR, 29734–29759. https://proceedings.mlr.press/v202/saito23b.html

    2023

  59. [64]

    Otmane Sakhi, Alexandre Gilotte, and David Rohde. 2025. Practical Improve- ments of A/B Testing with Off-Policy Estimation. arXiv:2506.10677 [stat.ML] https://arxiv.org/abs/2506.10677

    Pith/arXiv arXiv 2025

  60. [65]

    Flavian Vasile, David Rohde, Olivier Jeunen, and Amine Benhalloum. 2020. A Gentle Introduction to Recommendation as Counterfactual Policy Learning. In Proc. of the 28th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’20). ACM, 392–393. doi:10.1145/3340631.3398666

    arXiv 2020

  61. [66]

    Huizhi Xie and Juliette Aurisset. 2016. Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix. InProc. of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, 645–654. doi:10.1145/2939672.2939733

    arXiv 2016

  62. [67]

    Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. 2023. Choosing a Proxy Metric from Past Experiments. arXiv:2309.07893 [stat.ME]

    Pith/arXiv arXiv 2023

  63. [68]

    Bram van den Akker, Olivier Jeunen, Ying Li, Ben London, Zahra Nazari, and Devesh Parekh. 2024. Practical Bandits: An Industry Perspective. InProc. of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). ACM, 1132–1135. doi:10.1145/3616855.3636449

    arXiv 2024

  64. [71]

    Qing Zhang, Alex Deng, Michelle Du, Huiji Gao, Liwei He, and Sanjeev Katariya

  65. [73]

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group Sequence Policy Optimization. arXiv:2507.18071 [cs.LG] https://arxiv.org/abs/2507.18071 A Contemporaneous Work Table 1 provides a systematic overview of contemporaneous ap- proaches to policy-aware ...

    Pith/arXiv arXiv 2025

  66. [2021]

    2021), 19–30

    Recommendations as Treatments.AI Magazine42, 3 (Nov. 2021), 19–30

    2021

  67. [2025]

    InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25)

    Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25). ACM, 5205–5214. doi:10. 1145/3711896.3737232

    arXiv