When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits

Sang Su Lee; Shishir Dash; Vijay Raghavan; Vineeth Loganathan

REVIEW 2 major objections 6 minor 30 references

When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard offline checks can systematically mislead reward and policy selection in delayed-feedback contextual bandits; an ordered alignment-and-learnability screen catches the failures before deployment.

desk verdict A careful, honest applied paper with a genuine contribution (the P1–P5 screen and the N1/N2 lessons), whose main soft spot is an acknowledged but untested replay-stability assumption. read the letter →

arxiv 2608.11560 v1 pith:KBERR6JH submitted 2026-08-12 cs.LG

Sang Su Lee , Vineeth Loganathan , Shishir Dash , Vijay Raghavan This is my paper · ORCID

classification cs.LG

keywords contextualbanditsdelayedfeedbackoff-policyevaluationrewarddesignsurrogatemetricsonlinereplaypersonalizationpushnotifications

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual offline checks for choosing a proxy reward and deciding whether a contextual bandit is worth deploying—a batch off-policy estimate, a marginal arm-discrimination test, and a confidence interval—can mislead systematically when the true business outcome arrives only after a long delay. It proposes an ordered five-step diagnostic protocol that screens candidates on two axes: alignment, meaning optimizing the reward actually moves the north-star, and learnability, meaning the bandit can identify the reward-optimal policy. Two lessons carry the argument: a denser reward can look tied with a sparser one in a static batch value yet learn faster online, so batch lift mis-ranks rewards; and when the best single arm is unidentifiable from training data, a contextual policy's edge is at least as consistent with robustness as with personalization, so a personalization premium is easily overstated. The protocol is validated where ground truth is known and illustrated on a deployed push-notification system, with the deployment evidence offered as directional rather than powered.

What carries the argument

The load-bearing mechanism is an ordered diagnostic protocol, P1–P5, run before any reported lift is believed. P1 checks alignment via arm-level rank correlation between per-arm reward and per-arm north-star, not unit-level correlation; P2 tiers fixed best arm, context-free bandit, and contextual bandit to test whether the contextual tool is warranted; P3 checks learnability through arm discrimination plus reward density read from a step-by-step replay of the randomized log; P4 enforces off-policy hygiene with percentile bootstrap intervals, a doubly robust estimator, and respect for the $1/K$ overlap variance ceiling; P5 budgets the reward window and north-star maturation into a time-to-significance. The genuinely new pieces are narrow: replaying a randomized log to rank rewards by the learning trajectory they induce, and an achievable-ceiling estimate—the doubly robust headroom of a contextual oracle over the best fixed arm—as a fast screen for real personalization headroom. The paper uses a ridge-regression contextual bandit as the learner, self-normalized importance weighting and doubly robust estimators for off-policy values, and the surrogate-paradox logic as the mechanism behind the directionality check.

What would settle it

Run the decisive live comparison the paper leaves undone: randomize which proxy reward the bandit optimizes, with the protocol-chosen reward versus a naive reward as the randomized unit, and compare north-star conversion by a pre-registered horizon. If the denser or aligned reward does not produce faster or higher north-star conversion, or if replay run across many random seeds reverses the reward ordering, the learnability gate and N1 fail.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that under delayed-feedback contextual bandits the familiar offline workflow—one off-policy value estimate, a marginal test of whether arms differ, and a confidence interval—systematically flips the two upstream decisions it is used to settle. A correctly computed interval often shows that nothing is significant; a marginal test can call a reward flat when the effect the bandit exploits is conditional on context; and a single batch number cannot see that one reward induces faster online learning than another even when their batch values are identical. The paper supplies an ordered screen, P1–P5, whose first axis is alignment—arm-level directionality between reward and north-star, which catches surrogate-paradox rewards that a unit-level correlation waves through—and whose second axis is learnability, read from step-by-step replay rather than batch value. Two empirical findings carry the argument: reward density, not the reward's target, governs online learning efficiency (N1), and when the best single arm is unidentifiable, the contextual policy's measured edge cannot be cleanly attributed to per-user personalization (N2). The authors state plainly that the individual tests are not new; the contribution is the ordering, the composition, and the two lessons the screen surfaces.

Load-bearing premise

The load-bearing premise is that replaying the uniform-random logged half through a single deterministic path of the bandit faithfully reproduces the learning trajectory and reward ordering a live deployment would show; if replay curves are not stable across stochastic paths, or if adaptive exploration changes the reward ranking, the learnability gate and the N1 ranking lose their foundation.

Editorial extensions

If this is right

Batch lift should not be used to rank candidate proxy rewards; the ranking should come from the learning speed each reward induces in replay.
A non-significant marginal arm-discrimination test is not evidence that a reward is flat, because the exploitable effect may be conditional on context; the contextual policy must be compared against fixed-arm and context-free tiers.
A near-zero achievable ceiling means an apparent personalization premium should be read as robustness to uncertainty about the best arm, not as per-user targeting value; comparisons should use a deployable best-arm baseline rather than an oracle.
Confidence intervals must be read against estimator assumptions: a tight interval from an outcome-model-based estimator can be confidently wrong, so percentile intervals with a variance-reduced estimator and the $1/K$ overlap check are necessary hygiene.
Delayed north-star maturation sets a traffic-independent time floor: more traffic shrinks the minimum detectable effect but cannot shorten the cold-start plus maturation wait.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The ordered-screen logic extends beyond delayed feedback: any proxy-trained decision system that chooses between model families on the basis of one offline number could adopt the alignment-first, learnability-second ordering.
N1 implies reward density is itself a design variable: a slightly less aligned but much denser reward may beat a better-aligned sparse one in realized north-star value, a trade-off the paper surfaces but does not formalize.
N2 suggests a renormalized measure of personalization: report the contextual policy's gain against the best identifiable arm, not an oracle best arm, to separate robustness from true heterogeneity.
The replay's reported orderings rest on a single deterministic path; a natural testable extension is to bootstrap the replay over many random seeds or logged trajectories to put confidence intervals on the learning-curve ordering itself.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The first thing to know: this is a useful, honest paper about a real production problem—picking a proxy reward and deciding whether a contextual bandit is worth it when the north-star is delayed. It makes no grand algorithmic claims. Its contribution is an ordered diagnostic protocol (P1–P5) that screens alignment and learnability before trusting a batch lift, and two lessons that come out of running it: (N1) a single offline number can mis-rank rewards, because a denser reward signal lets the bandit learn faster online even when batch values look tied; and (N2) when the best single arm is unidentifiable, a contextual policy's edge may be robustness to that unidentifiability rather than real per-user personalization. The paper says the individual steps are not new, and that is accurate—the novelty is the composition and the specific use of replay to rank rewards by induced learning speed, plus the achievable-ceiling headroom screen.

What it does well: it validates against known ground truth across several sources—a controllable synthetic generator, the Open Bandit Pipeline (including real ZOZOTOWN logs), and public classification data—and it is unusually explicit about what each finding can and cannot support. The deployment case study is appropriately labeled as underpowered and directional.

The soft spots are real but not fatal. The load-bearing link is the replay of the uniform-random logged slice through LinUCB: the paper runs a single deterministic path and states that only orderings and cold-start shape are trustworthy. That is an acknowledgement, not a demonstration. There is no repeated-seed or bootstrap distribution for the ordering, so we don't know whether the dense-over-sparse ranking is stable across stochastic resamples. The live deployment also uses a 10% explore floor, so the online logging distribution differs from the replay's 1/K acceptance; the authors assert orderings transfer but never test invariance to that shift. These are stated in the threats section, not hidden, but they are the difference between a protocol that is internally plausible and one that is fully evidenced. A second gap is that no code or data are released, which makes independent reimplementation harder even though the open-data validation helps.

My recommendation: send it to peer review. The core reasoning holds, the writing is clear, and the limitations are owned. The referee should ask for a seed-sweep or bootstrap over replay paths and ideally a powered deployment or a formal treatment of replay stability. This is exactly the kind of paper that benefits a workshop or applied track.

Referee Report

2 major / 6 minor

Summary. This paper addresses the problem of selecting a proxy reward and deciding whether to deploy a contextual bandit in delayed-feedback settings, where the business north-star is observed only after a long delay. The authors propose an ordered diagnostic protocol (P1–P5) that screens candidate rewards on alignment with the north-star (P1), tool-fit (P2), learnability via offline replay (P3), off-policy estimation hygiene (P4), and delay budgeting (P5). They claim that standard offline checks—batch off-policy estimates, marginal arm-discrimination tests, and confidence intervals—can systematically mislead, and that their protocol catches these failures before deployment. Validation is provided on a controllable synthetic generator with known ground truth, on the Open Bandit Pipeline (both synthetic and real logged data), on public classification datasets (digits, covertype), and through a deployment case study. Two lessons are highlighted: (N1) reward density governs online learning efficiency, so two rewards that look tied in a batch estimate can separate once a bandit learns online; and (N2) when the best single arm is unidentifiable, a contextual policy's edge over single arms may reflect robustness to that unidentifiability rather than true personalization. The paper is explicit about the provenance of individual tools and about the limited, directional nature of the deployment evidence.

Significance. If the claims hold, the paper makes a useful and practical contribution: it integrates existing ideas (surrogate directionality, reward informativeness, off-policy evaluation, value of personalization) into a single ordered screen that practitioners can apply before running live bandit experiments. The two lessons, especially N1 about reward density and N2 about the robustness-vs-personalization distinction, are likely to be valuable for industrial bandit deployment. Strengths include the use of independent open data and tooling, the honest labeling of evidence quality, and the clear statement of limitations. The main methodological risk is that the replay-based demonstration of N1 rests on a single deterministic path without quantified stability across stochastic resamples; this needs to be addressed before the practical prescriptions can be fully trusted.

major comments (2)

[4.4, 4.7, Figure 2, Section 6] The central support for lesson N1 and for the learnability gate P3 is the claim that a step-by-step replay of logged data reveals that a dense reward learns faster online than a sparse one even when batch values are tied. This claim is currently supported only by single deterministic replay paths: the synthetic M2 run (Section 4.4), the OBP synthetic reproduction (Section 4.7), and the covertype learning curve (Figure 2). The paper itself states in Section 6 that 'the replay runs on the uniform-random half (a counterfactual, not the live contextual policy) along a single deterministic path, so its CIs are optimistic and only orderings and cold-start shape are trustworthy,' but no repeated-seed, bootstrap, or confidence-interval analysis is provided to demonstrate that the dense-over-sparse ordering is stable. Because the replayer accepts only roughly 1/K of logged rounds (with K=5, about 20%), early-bucket differences such as the reported +50pp gap in M2 could be dominated by sampling noise. The authors should report the distribution of the learning curves (e.g., across resamples of the logged data or seeds of the generator) and quantify the probability that the ordering reverses, or explicitly restrict the protocol's claims to settings where such stability can be established. As written, this is a load-bearing gap for N1 and for the practical prescription to 'rank rewards by replayed learning speed.'
[5 (Deployment case study)] The deployment evidence for N2 hinges on the achievable-ceiling estimate being '≈0' for the contextual policy's headroom over the best fixed arm. The paper does not report a confidence interval or bootstrap distribution for this estimate, even though the synthetic validation in Section 4.6 shows that the doubly-robust ceiling estimate can vary across seeds (e.g., -0.1pp at K=3 to -5.9pp at K=20). With a single split and approximately five arms, the deployment's near-zero reading may be consistent with a range of true headrooms, so the 'robustness rather than personalization' interpretation is weaker than the presentation suggests. Given that N2 is a headline lesson and the deployment is the only natural-occurring real-world instance, the paper should either provide an uncertainty measure for the deployment ceiling or state more prominently that the deployment provides only anecdotal support for N2.

minor comments (6)

[1 (Abstract)] The abstract contains a typo: 'downstreamconversion' should be 'downstream conversion'. Also, hyphenated compounds such as 'north-star' are used inconsistently; consider standardizing.
[3 (Protocol)] The protocol description in Section 3 is verbose; the checklist in Section 6 is clearer. Consider moving the checklist earlier or adding a concise summary table after Table 1.
[4.4] In Mechanism 2, the sentence 'Trained on the full batch, both recover the optimal policy, so their batch values are tied (+0.0pp gap)' could be misread as a trivial consequence of using the same policy; clarify that the batch value is computed for the policy learned from the full batch under each reward.
[Figure 2] The learning curves in Figure 2 would benefit from shaded confidence bands or error bars; without them, the claim that the dense reward 'learns a better policy at every horizon' is not visually supported.
[4.7] The description of the Open Bandit Dataset example is dense; consider splitting the long sentence about the ZOZOTOWN sample for readability.
[5] The deployment numbers are relative and absolute volumes withheld; consider adding a short table with the key directional findings and their significance labels for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: protocol validated against independent ground truth, with explicit anti-circularity safeguards.

full rationale

This is a methodological and empirical protocol paper, not a derivation chain in which a fitted parameter is renamed as a prediction. The protocol's checks P1-P5 are ordered screening rules with explicit pass/fail semantics ('necessary-condition screens, not pass/fail certifications'), and no parameter is fitted to a subset of data and then reported as an independent prediction. The two headline lessons, N1 and N2, are demonstrated on a synthetic generator that the authors openly disclose is 'deliberately built to contain each effect, so it tests whether the protocol detects known structure—not whether the structure exists'; this is a validation of the protocol's detection ability against known ground truth, not a derivation of the effect from the protocol itself. The paper also provides a real anti-circularity safeguard: the same failure modes reproduce on independent open data and tooling the authors did not write, including the Open Bandit Pipeline, UCI covertype, and sklearn-digits, and the authors explicitly disclaim novelty for the individual tests ('The individual steps are not new'). There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The only notable residual concern—replay-order stability across stochastic resamples of a single deterministic path—is a validity threat that the authors themselves acknowledge in Section 6 ('only orderings and cold-start shape are trustworthy'), and it does not amount to circularity. The paper's claims are therefore self-contained against external benchmarks, and the circularity burden is minimal.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The protocol introduces no new physical or model entities: no new particles, mediators, forces, or conserved quantities. It introduces two methodological constructs, the replay-to-rank-rewards screen and the achievable-ceiling estimate, but these are statistical quantities computed from existing estimators rather than invented entities. The free parameters listed are hand-chosen simulation constants, explicitly not fitted estimates, and the axioms are the standard OPE assumptions plus the replay-faithfulness assumption that the authors themselves flag as the fragile link.

free parameters (3)

Synthetic outcome conversion probabilities (0.40 when arm matches segment, 0.15 otherwise) = 0.40 / 0.15
Hand-chosen to give a clear conditional contrast in the controllable generator; authors state these are illustrative design choices, not estimates (Section 4.1), and verdicts are stable across K-sweeps and seeds.
Reward density levels in N1 demonstrations (100% vs 10%) and no-structure mean band [0.17, 0.23] = 100%/10%; [0.17, 0.23]
Hand-set simulation knobs chosen so density and structure effects are cleanly exhibited; not fitted to real data. The protocol itself sets no numeric thresholds.
Class-rebalancing strength gamma on covertype = gamma in {0,1,2,3,4}
Swept knob that induces a surrogate paradox on public data; used to validate the arm-level directionality check, not fitted to make a prediction.

assumptions (5)

domain assumption Uniform-random logging with known propensity 1/K and a temporal train/eval split.
Section 3 setting; the replayer and doubly-robust estimators require known propensities, and the deployment is a near-uniform A/B split.
domain assumption Replaying the uniform-random logged slice through the chosen learner (disjoint LinUCB) faithfully reproduces online learning trajectories and reward orderings.
Section 5 and Section 6 threats: authors state confidence intervals are optimistic and only orderings and cold-start shape are trustworthy; this assumption is load-bearing for N1 and P3.
domain assumption Arm-level rank correlation between mean proxy reward and mean north-star across K arms is a reliable directionality signal.
Section 3 P1; validated on synthetic and covertype, but with only five arms the deployment reading is directional, as the authors note.
standard math Standard OPE unbiasedness: with known propensities, IPW/SNIPS and doubly-robust estimators are unbiased if the outcome model or propensities are correct.
Section 2; standard results from Dudik et al. 2011 and Swaminathan and Joachims 2015, used as background.
domain assumption Classes-as-arms conversion of public classification datasets gives the true policy value as label accuracy.
Section 4.7; used to measure estimator bias directly on digits and covertype.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits." pith.science (2026). https://pith.science/paper/KBERR6JH

@misc{pith2026260811560,
  author       = {Pith},
  title        = {Pith review of: When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBERR6JH}},
  note         = {Machine review of arXiv:2608.11560}
}

read the original abstract

Personalizing marketing messages with contextual multi-armed bandits (CMABs) drives real business value, yet the objective that ultimately matters - a downstream conversion - is observed only weeks later, too late to drive online learning. Teams therefore train the bandit on a fast proxy reward, and separately must judge whether a contextual bandit is worth its complexity over sending one best message. Settling both decisions with the usual offline checks - a batch off-policy estimate, a marginal arm-discrimination test, a confidence interval - can mislead systematically under delayed feedback. We give an ordered diagnostic protocol that screens a reward-and-policy candidate on two axes, alignment (does optimizing the reward move the north-star?) and learnability (can the bandit identify the reward-optimal policy?), before trusting any reported lift. We validate it where the truth is known - a public off-policy-evaluation benchmark and a controllable synthetic generator - and illustrate it on a deployed large-marketplace push system (where, with five arms and one split, the evidence is directional rather than powered). Two lessons recur. (N1) A single offline number can mis-rank rewards: a denser reward signal gives the bandit more to learn from, so rewards that look tied in a static estimate pull apart once learning happens online. (N2) If you cannot tell in advance which single message is best, a per-user policy partly just avoids betting on the wrong one - that looks like personalization but is really robustness, so a "personalization premium" is easily overstated. Our contribution is methodological rather than algorithmic: the ordered protocol, the two lessons it surfaces, and the end-to-end experience of applying it to a delayed-feedback CMAB.

Figures

Figures reproduced from arXiv: 2608.11560 by the authors.

**Figure 2.** N1 on a public dataset (UCI covertype recast as a [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 23 canonical work pages

[1]

Imbens, and Hyunseung Kang

Susan Athey, Raj Chetty, Guido W. Imbens, and Hyunseung Kang. 2019.The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely. Technical Report w26463. National Bureau of Economic Research

work page 2019
[2]

Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. 2024. Informa- tiveness of Reward Functions in Reinforcement Learning. InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

work page 2024
[3]

Pavel Dmitriev and Xian Wu. 2016. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. InProceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

work page 2016
[4]

Weitao Duan, Shan Ba, and Chunzhe Zhang. 2021. Online Experimentation with Surrogate Metrics: Guidelines and a Case Study. InProceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM)

work page 2021
[5]

Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. InProceedings of the 28th International Conference on Machine Learning (ICML)

work page 2011
[6]

Hitsch, Sanjog Misra, and Walter W

Günter J. Hitsch, Sanjog Misra, and Walter W. Zhang. 2024. Heterogeneous Treat- ment Effects and Optimal Targeting Policy Evaluation.Quantitative Marketing and Economics(2024). doi:10.1007/s11129-023-09278-5 SSRN:3111957

work page doi:10.1007/s11129-023-09278-5 2024
[7]

Olivier Jeunen, Jatin Mandav, Ivan Potapov, Nakul Agarwal, Sourabh Vaid, Wen- zhe Shi, and Aleksei Ustimenko. 2024. Multi-Objective Recommendation via Multivariate Policy Learning. InProceedings of the 18th ACM Conference on Recommender Systems (RecSys)

work page 2024
[8]

Pooria Joulani, András György, and Csaba Szepesvári. 2013. Online Learning under Delayed Feedback. InProceedings of the 30th International Conference on Machine Learning (ICML)

work page 2013

Show all 30 references

[9]

2020.Trustworthy Online Controlled Experi- ments: A Practical Guide to A/B Testing

Ron Kohavi, Diane Tang, and Ya Xu. 2020.Trustworthy Online Controlled Experi- ments: A Practical Guide to A/B Testing. Cambridge University Press

2020
[10]

Schapire

Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A Contextual- Bandit Approach to Personalized News Article Recommendation. InProceedings of the 19th International Conference on World Wide Web (WWW)

2010
[11]

Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011. Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algo- rithms. InProceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM). arXiv:1003.5956

2011 arXiv
[12]

David Manheim and Scott Garrabrant. 2018. Categorizing Variants of Goodhart’s Law.arXiv preprint arXiv:1803.04585(2018)

2018 arXiv
[13]

James McInerney, Ehtsham Elahi, Justin Basilico, Yves Raimond, and Tony Jebara
[14]

Ng, Daishi Harada, and Stuart Russell

Andrew Y. Ng, Daishi Harada, and Stuart Russell. 1999. Policy Invariance un- der Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the 16th International Conference on Machine Learning (ICML)

1999
[15]

Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. InInternational Conference on Learning Representations (ICLR)

2022
[16]

Ciara Pike-Burke, Shipra Agrawal, Csaba Szepesvári, and Steffen Grünewälder
[17]

Prentice

Ross L. Prentice. 1989. Surrogate endpoints in clinical trials: definition and operational criteria.Statistics in Medicine8, 4 (1989), 431–440

1989
[18]

Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley

Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley
[19]

Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. 2021. Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off- Policy Evaluation. InProceedings of the NeurIPS Datasets and Benchmarks Track. arXiv:2008.07146

2021 arXiv
[20]

Anya Shchetkina and Ron Berman. 2024. When Is Heterogeneity Actionable for Personalization?arXiv preprint arXiv:2411.16552(2024)

2024
[21]

Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger
[22]

Adith Swaminathan and Thorsten Joachims. 2015. The Self-Normalized Estima- tor for Counterfactual Learning. InAdvances in Neural Information Processing Systems (NeurIPS)

2015
[23]

Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. 2024. Choosing a Proxy Metric from Past Experiments. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). arXiv:2309.07893

2024 arXiv
[24]

VanderWeele

Tyler J. VanderWeele. 2013. Surrogate Measures and Consistent Surrogates. Biometrics69, 3 (2013), 561–565

2013
[25]

Claire Vernade, Olivier Cappé, and Vianney Perchet. 2017. Stochastic Bandit Mod- els for Delayed Conversions. InProceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI)

2017
[26]

temporal70 /30

Cameron Voloshin, Hoang M. Le, Nan Jiang, and Yisong Yue. 2021. Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning. InProceedings of the NeurIPS Datasets and Benchmarks Track. arXiv:1911.06854. Sang Su Lee, Vineeth Loganathan, Shishir Dash, and Vijay Ra...

2021 arXiv
[2013]

A Survey of Multi-Objective Sequential Decision-Making.Journal of Artificial Intelligence Research (JAIR)48 (2013), 67–113

2013
[2018]

InProceedings of the 35th International Conference on Machine Learning (ICML)

Bandits with Delayed, Aggregated Anonymous Feedback. InProceedings of the 35th International Conference on Machine Learning (ICML)
[2022]

InAdvances in Neural Information Processing Systems (NeurIPS)

Defining and Characterizing Reward Hacking. InAdvances in Neural Information Processing Systems (NeurIPS)
[2023]

InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Imbens, and Hyunseung Kang

Susan Athey, Raj Chetty, Guido W. Imbens, and Hyunseung Kang. 2019.The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely. Technical Report w26463. National Bureau of Economic Research

work page 2019

[2] [2]

Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. 2024. Informa- tiveness of Reward Functions in Reinforcement Learning. InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

work page 2024

[3] [3]

Pavel Dmitriev and Xian Wu. 2016. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. InProceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

work page 2016

[4] [4]

Weitao Duan, Shan Ba, and Chunzhe Zhang. 2021. Online Experimentation with Surrogate Metrics: Guidelines and a Case Study. InProceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM)

work page 2021

[5] [5]

Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. InProceedings of the 28th International Conference on Machine Learning (ICML)

work page 2011

[6] [6]

Hitsch, Sanjog Misra, and Walter W

Günter J. Hitsch, Sanjog Misra, and Walter W. Zhang. 2024. Heterogeneous Treat- ment Effects and Optimal Targeting Policy Evaluation.Quantitative Marketing and Economics(2024). doi:10.1007/s11129-023-09278-5 SSRN:3111957

work page doi:10.1007/s11129-023-09278-5 2024

[7] [7]

Olivier Jeunen, Jatin Mandav, Ivan Potapov, Nakul Agarwal, Sourabh Vaid, Wen- zhe Shi, and Aleksei Ustimenko. 2024. Multi-Objective Recommendation via Multivariate Policy Learning. InProceedings of the 18th ACM Conference on Recommender Systems (RecSys)

work page 2024

[8] [8]

Pooria Joulani, András György, and Csaba Szepesvári. 2013. Online Learning under Delayed Feedback. InProceedings of the 30th International Conference on Machine Learning (ICML)

work page 2013