{"id":"d16ceca1-ed20-4d0e-a1df-675a4aa4c851","arxiv_id":"2608.11560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ordered diagnostic protocol screens proxy rewards and contextual-bandit policies for alignment and learnability before deployment, and shows offline batch estimates can mislead under delayed feedback.","lead":"This paper shows that the usual offline numbers used to pick a reward signal for a personalized marketing bandit can be misleading when the real business goal is only observed weeks later. It contributes an ordered pre-deployment diagnostic protocol, plus two lessons: denser rewards can look tied offline yet learn faster online, and a contextual policy's edge can be robustness rather than real personalization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Replay-order stability is the load-bearing link: N1's reward ranking and P3's learnability gate rest on a single deterministic replay path, whose ordering across stochastic resamples is never quantified.","rationale":"The reader identified the same load-bearing assumption I would: the replay of the uniform-random logged slice through disjoint LinUCB must faithfully reproduce the online learning trajectory and reward ordering that live deployment would exhibit, at least at the level of relative orderings. My review concludes that this is indeed the single most fragile link in the paper's central claim. The paper is methodologically careful in many respects: it states provenance, validates mechanisms on controlled synthetic ground truth and on independent open data, and explicitly labels the deployment evidence as directional and underpowered. Those are real strengths, and the ground-truth demonstrations of the interval trap, the surrogate paradox, density-driven learning speed, and the achievable-ceiling check are credible as mechanism-level validations. But the jump from those validations to the practical pre-deployment screen relies on replay-order stability, and that stability is asserted rather than shown. The authors' own Section 6 threat list concedes the single-path issue and optimistically hand-waves that 'only orderings and cold-start shape are trustworthy'; no repeated-seed or resampling analysis supports even that restricted claim. This is not a reason to reject the paper: the concern is testable, and the synthetic and OBP infrastructure already in the paper makes the test cheap. The reader's CONDITIONAL verdict is therefore appropriate, and my stress-test does not move it. I would keep the same recommendation and add the replay-stability check as an explicit condition for acceptance. I did not find a separate, more damaging internal inconsistency; the acknowledged limitations (no code/data release, one underpowered deployment, N2's zero-headroom regime existing only in synthetic data) are real but less load-bearing than the replay-order assumption.","tokens_in":15126,"tokens_out":4839,"duration_ms":57795,"concrete_test":"Run the synthetic M2 comparison (Section 4.4), or its OBP synthetic analogue, for 100 independent repetitions with freshly drawn logged data (or 100 bootstrap resamples of the fixed log), each time replaying both dense and 10%-density rewards under disjoint LinUCB. Record, at the pre-convergence horizon, the sign and magnitude of the dense-minus-sparse learning-curve gap and the final reward ordering. If the dense reward wins in fewer than 95 of 100 repetitions, or if the ordering flips under bootstrap resampling, then the N1 ranking prescription and P3's learnability gate lack the stability the protocol requires. Separately, if feasible, replay on the deployment's live 10% explore slice rather than only the uniform-random half and compare the resulting reward ordering to the uniform-random replay ordering to test whether the counterfactual transfer is valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a batch estimate can mis-rank rewards (N1) and that the protocol's learnability gate (P3) catches this depends on a specific premise: the step-by-step replay of the uniform-random logged slice through disjoint LinUCB reliably recovers the relative ordering of rewards by online learning speed. The paper states this premise almost verbatim in Section 6: 'the replay runs on the uniform-random half (a counterfactual, not the live contextual policy) along a single deterministic path, so its CIs are optimistic and only orderings and cold-start shape are trustworthy.' That is an acknowledgement, not a demonstration, and the premise is load-bearing. The replayer accepts only the fraction of logged rounds where the target policy's chosen arm equals the logged arm; with K=5 that discards roughly 80% of rounds. Early-bucket differences such as the +50pp gap in Mechanism 2 (Section 4.4) may therefore be dominated by sampling noise. No repeated-seed or bootstrap distribution for the N1 replay ordering is reported for the synthetic M2 run, the OBP synthetic reproduction, the covertype learning curve (Figure 2), or the deployment's directional comparison. If the dense-over-sparse ordering flips across stochastic resamples, then N1's practical prescription ('rank rewards by replayed learning speed') and P3's reliance on replay are not stable. A second, related issue: the live policy in the deployment uses a 10% uniform explore floor and 90% exploitation, so the context distribution and action-selection frequencies the learner experiences online differ from the replay's uniform-random 1/K acceptance. The authors assert that only orderings and cold-start shape transfer, but do not test whether the ranking is invariant to this distribution shift. This is not a hidden flaw: it is stated as a threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the problem of selecting a proxy reward and deciding whether to deploy a contextual bandit in delayed-feedback settings, where the business north-star is observed only after a long delay. The authors propose an ordered diagnostic protocol (P1–P5) that screens candidate rewards on alignment with the north-star (P1), tool-fit (P2), learnability via offline replay (P3), off-policy estimation hygiene (P4), and delay budgeting (P5). They claim that standard offline checks—batch off-policy estimates, marginal arm-discrimination tests, and confidence intervals—can systematically mislead, and that their protocol catches these failures before deployment. Validation is provided on a controllable synthetic generator with known ground truth, on the Open Bandit Pipeline (both synthetic and real logged data), on public classification datasets (digits, covertype), and through a deployment case study. Two lessons are highlighted: (N1) reward density governs online learning efficiency, so two rewards that look tied in a batch estimate can separate once a bandit learns online; and (N2) when the best single arm is unidentifiable, a contextual policy's edge over single arms may reflect robustness to that unidentifiability rather than true personalization. The paper is explicit about the provenance of individual tools and about the limited, directional nature of the deployment evidence.","tokens_in":15401,"tokens_out":13534,"duration_ms":131634,"significance":"If the claims hold, the paper makes a useful and practical contribution: it integrates existing ideas (surrogate directionality, reward informativeness, off-policy evaluation, value of personalization) into a single ordered screen that practitioners can apply before running live bandit experiments. The two lessons, especially N1 about reward density and N2 about the robustness-vs-personalization distinction, are likely to be valuable for industrial bandit deployment. Strengths include the use of independent open data and tooling, the honest labeling of evidence quality, and the clear statement of limitations. The main methodological risk is that the replay-based demonstration of N1 rests on a single deterministic path without quantified stability across stochastic resamples; this needs to be addressed before the practical prescriptions can be fully trusted.","major_comments":[{"comment":"The central support for lesson N1 and for the learnability gate P3 is the claim that a step-by-step replay of logged data reveals that a dense reward learns faster online than a sparse one even when batch values are tied. This claim is currently supported only by single deterministic replay paths: the synthetic M2 run (Section 4.4), the OBP synthetic reproduction (Section 4.7), and the covertype learning curve (Figure 2). The paper itself states in Section 6 that 'the replay runs on the uniform-random half (a counterfactual, not the live contextual policy) along a single deterministic path, so its CIs are optimistic and only orderings and cold-start shape are trustworthy,' but no repeated-seed, bootstrap, or confidence-interval analysis is provided to demonstrate that the dense-over-sparse ordering is stable. Because the replayer accepts only roughly 1/K of logged rounds (with K=5, about 20%), early-bucket differences such as the reported +50pp gap in M2 could be dominated by sampling noise. The authors should report the distribution of the learning curves (e.g., across resamples of the logged data or seeds of the generator) and quantify the probability that the ordering reverses, or explicitly restrict the protocol's claims to settings where such stability can be established. As written, this is a load-bearing gap for N1 and for the practical prescription to 'rank rewards by replayed learning speed.'","section":"4.4, 4.7, Figure 2, Section 6"},{"comment":"The deployment evidence for N2 hinges on the achievable-ceiling estimate being '≈0' for the contextual policy's headroom over the best fixed arm. The paper does not report a confidence interval or bootstrap distribution for this estimate, even though the synthetic validation in Section 4.6 shows that the doubly-robust ceiling estimate can vary across seeds (e.g., -0.1pp at K=3 to -5.9pp at K=20). With a single split and approximately five arms, the deployment's near-zero reading may be consistent with a range of true headrooms, so the 'robustness rather than personalization' interpretation is weaker than the presentation suggests. Given that N2 is a headline lesson and the deployment is the only natural-occurring real-world instance, the paper should either provide an uncertainty measure for the deployment ceiling or state more prominently that the deployment provides only anecdotal support for N2.","section":"5 (Deployment case study)"}],"minor_comments":[{"comment":"The abstract contains a typo: 'downstreamconversion' should be 'downstream conversion'. Also, hyphenated compounds such as 'north-star' are used inconsistently; consider standardizing.","section":"1 (Abstract)"},{"comment":"The protocol description in Section 3 is verbose; the checklist in Section 6 is clearer. Consider moving the checklist earlier or adding a concise summary table after Table 1.","section":"3 (Protocol)"},{"comment":"In Mechanism 2, the sentence 'Trained on the full batch, both recover the optimal policy, so their batch values are tied (+0.0pp gap)' could be misread as a trivial consequence of using the same policy; clarify that the batch value is computed for the policy learned from the full batch under each reward.","section":"4.4"},{"comment":"The learning curves in Figure 2 would benefit from shaded confidence bands or error bars; without them, the claim that the dense reward 'learns a better policy at every horizon' is not visually supported.","section":"Figure 2"},{"comment":"The description of the Open Bandit Dataset example is dense; consider splitting the long sentence about the ZOZOTOWN sample for readability.","section":"4.7"},{"comment":"The deployment numbers are relative and absolute volumes withheld; consider adding a short table with the key directional findings and their significance labels for clarity.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and honest about its limitations. The main issue is the missing stability analysis for the replay-based demonstration of N1, which is central to the protocol's learnability gate. I would like to see the authors provide repeated-seed or bootstrap evidence for the replay ordering. The paper's use of open data and independent tooling is a strength. There is no concern about novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know: this is a useful, honest paper about a real production problem—picking a proxy reward and deciding whether a contextual bandit is worth it when the north-star is delayed. It makes no grand algorithmic claims. Its contribution is an ordered diagnostic protocol (P1–P5) that screens alignment and learnability before trusting a batch lift, and two lessons that come out of running it: (N1) a single offline number can mis-rank rewards, because a denser reward signal lets the bandit learn faster online even when batch values look tied; and (N2) when the best single arm is unidentifiable, a contextual policy's edge may be robustness to that unidentifiability rather than real per-user personalization. The paper says the individual steps are not new, and that is accurate—the novelty is the composition and the specific use of replay to rank rewards by induced learning speed, plus the achievable-ceiling headroom screen.\n\nWhat it does well: it validates against known ground truth across several sources—a controllable synthetic generator, the Open Bandit Pipeline (including real ZOZOTOWN logs), and public classification data—and it is unusually explicit about what each finding can and cannot support. The deployment case study is appropriately labeled as underpowered and directional.\n\nThe soft spots are real but not fatal. The load-bearing link is the replay of the uniform-random logged slice through LinUCB: the paper runs a single deterministic path and states that only orderings and cold-start shape are trustworthy. That is an acknowledgement, not a demonstration. There is no repeated-seed or bootstrap distribution for the ordering, so we don't know whether the dense-over-sparse ranking is stable across stochastic resamples. The live deployment also uses a 10% explore floor, so the online logging distribution differs from the replay's 1/K acceptance; the authors assert orderings transfer but never test invariance to that shift. These are stated in the threats section, not hidden, but they are the difference between a protocol that is internally plausible and one that is fully evidenced. A second gap is that no code or data are released, which makes independent reimplementation harder even though the open-data validation helps.\n\nMy recommendation: send it to peer review. The core reasoning holds, the writing is clear, and the limitations are owned. The referee should ask for a seed-sweep or bootstrap over replay paths and ideally a powered deployment or a formal treatment of replay stability. This is exactly the kind of paper that benefits a workshop or applied track.","headline":"A careful, honest applied paper with a genuine contribution (the P1–P5 screen and the N1/N2 lessons), whose main soft spot is an acknowledged but untested replay-stability assumption.","tokens_in":16033,"tokens_out":2623,"would_cite":true,"duration_ms":25905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard offline checks can systematically mislead reward and policy selection in delayed-feedback contextual bandits; an ordered alignment-and-learnability screen catches the failures before deployment.","keywords":["contextual bandits","delayed feedback","off-policy evaluation","reward design","surrogate metrics","online replay","personalization","push notifications"],"falsifier":"Run the decisive live comparison the paper leaves undone: randomize which proxy reward the bandit optimizes, with the protocol-chosen reward versus a naive reward as the randomized unit, and compare north-star conversion by a pre-registered horizon. If the denser or aligned reward does not produce faster or higher north-star conversion, or if replay run across many random seeds reverses the reward ordering, the learnability gate and N1 fail.","tokens_in":1855,"feed_emoji":"🎯","tokens_out":3944,"duration_ms":112672,"temperature":0.7,"pith_summary":"The paper argues that the usual offline checks for choosing a proxy reward and deciding whether a contextual bandit is worth deploying—a batch off-policy estimate, a marginal arm-discrimination test, and a confidence interval—can mislead systematically when the true business outcome arrives only after a long delay. It proposes an ordered five-step diagnostic protocol that screens candidates on two axes: alignment, meaning optimizing the reward actually moves the north-star, and learnability, meaning the bandit can identify the reward-optimal policy. Two lessons carry the argument: a denser reward can look tied with a sparser one in a static batch value yet learn faster online, so batch lift mis-ranks rewards; and when the best single arm is unidentifiable from training data, a contextual policy's edge is at least as consistent with robustness as with personalization, so a personalization premium is easily overstated. The protocol is validated where ground truth is known and illustrated on a deployed push-notification system, with the deployment evidence offered as directional rather than powered.","feed_headline":"Don't rank bandit rewards by offline lift alone","feed_subtitle":"Five checks on alignment and learnability catch the offline traps before a bandit ships.","key_machinery":"The load-bearing mechanism is an ordered diagnostic protocol, P1–P5, run before any reported lift is believed. P1 checks alignment via arm-level rank correlation between per-arm reward and per-arm north-star, not unit-level correlation; P2 tiers fixed best arm, context-free bandit, and contextual bandit to test whether the contextual tool is warranted; P3 checks learnability through arm discrimination plus reward density read from a step-by-step replay of the randomized log; P4 enforces off-policy hygiene with percentile bootstrap intervals, a doubly robust estimator, and respect for the $1/K$ overlap variance ceiling; P5 budgets the reward window and north-star maturation into a time-to-significance. The genuinely new pieces are narrow: replaying a randomized log to rank rewards by the learning trajectory they induce, and an achievable-ceiling estimate—the doubly robust headroom of a contextual oracle over the best fixed arm—as a fast screen for real personalization headroom. The paper uses a ridge-regression contextual bandit as the learner, self-normalized importance weighting and doubly robust estimators for off-policy values, and the surrogate-paradox logic as the mechanism behind the directionality check.","core_discovery":"On its own terms, the paper's central claim is that under delayed-feedback contextual bandits the familiar offline workflow—one off-policy value estimate, a marginal test of whether arms differ, and a confidence interval—systematically flips the two upstream decisions it is used to settle. A correctly computed interval often shows that nothing is significant; a marginal test can call a reward flat when the effect the bandit exploits is conditional on context; and a single batch number cannot see that one reward induces faster online learning than another even when their batch values are identical. The paper supplies an ordered screen, P1–P5, whose first axis is alignment—arm-level directionality between reward and north-star, which catches surrogate-paradox rewards that a unit-level correlation waves through—and whose second axis is learnability, read from step-by-step replay rather than batch value. Two empirical findings carry the argument: reward density, not the reward's target, governs online learning efficiency (N1), and when the best single arm is unidentifiable, the contextual policy's measured edge cannot be cleanly attributed to per-user personalization (N2). The authors state plainly that the individual tests are not new; the contribution is the ordering, the composition, and the two lessons the screen surfaces.","pith_inferences":["The ordered-screen logic extends beyond delayed feedback: any proxy-trained decision system that chooses between model families on the basis of one offline number could adopt the alignment-first, learnability-second ordering.","N1 implies reward density is itself a design variable: a slightly less aligned but much denser reward may beat a better-aligned sparse one in realized north-star value, a trade-off the paper surfaces but does not formalize.","N2 suggests a renormalized measure of personalization: report the contextual policy's gain against the best identifiable arm, not an oracle best arm, to separate robustness from true heterogeneity.","The replay's reported orderings rest on a single deterministic path; a natural testable extension is to bootstrap the replay over many random seeds or logged trajectories to put confidence intervals on the learning-curve ordering itself."],"forward_implications":["Batch lift should not be used to rank candidate proxy rewards; the ranking should come from the learning speed each reward induces in replay.","A non-significant marginal arm-discrimination test is not evidence that a reward is flat, because the exploitable effect may be conditional on context; the contextual policy must be compared against fixed-arm and context-free tiers.","A near-zero achievable ceiling means an apparent personalization premium should be read as robustness to uncertainty about the best arm, not as per-user targeting value; comparisons should use a deployable best-arm baseline rather than an oracle.","Confidence intervals must be read against estimator assumptions: a tight interval from an outcome-model-based estimator can be confidently wrong, so percentile intervals with a variance-reduced estimator and the $1/K$ overlap check are necessary hygiene.","Delayed north-star maturation sets a traffic-independent time floor: more traffic shrinks the minimum detectable effect but cannot shorten the cold-start plus maturation wait."],"supporting_citations":[{"why":"Supplies the ridge-regression contextual bandit learner used in the replay and in deployment.","marker":"[10]"},{"why":"Introduced the replayer mechanism that the paper repurposes to rank rewards by the learning trajectory they induce.","marker":"[11]"},{"why":"Supplies the doubly robust estimator used for policy values and for the achievable-ceiling check.","marker":"[5]"},{"why":"Provides the surrogate-paradox mechanism behind the P1 arm-level directionality check.","marker":"[24]"},{"why":"Supplies independent open-source logged data and synthetic benchmarks on which the interval trap, N1, and N2 reproduce.","marker":"[19]"},{"why":"Supplies the self-normalized importance-sampling estimator used for counterfactual value estimates on the randomized log.","marker":"[22]"},{"why":"Defines the value of an optimal targeting policy over the best uniform action, which underlies the achievable-ceiling screen.","marker":"[6]"},{"why":"Defines reward informativeness, used to explain why density governs online learning efficiency in N1.","marker":"[2]"}],"fun_headline_variants":["Offline lift misranks bandit rewards under delayed feedback","Two traps: reward density and fake personalization in bandits","Diagnostic protocol for delayed-feedback bandits: align and learn","Don't trust offline lift when rewards arrive late"],"cache_read_input_tokens":18048,"weakest_assumption_plain":"The load-bearing premise is that replaying the uniform-random logged half through a single deterministic path of the bandit faithfully reproduces the learning trajectory and reward ordering a live deployment would show; if replay curves are not stable across stochastic paths, or if adaptive exploration changes the reward ranking, the learnability gate and the N1 ranking lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Offline lift misranks bandit rewards under delayed feedback","Two traps: reward density and fake personalization in bandits","Diagnostic protocol for delayed-feedback bandits: align and learn","Don't trust offline lift when rewards arrive late"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1427,"prompt_tokens":1122,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":738,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":738,"tokens_out":305,"duration_ms":3787,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:35:10.655850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the decisive live comparison the paper leaves undone: randomize which proxy reward the bandit optimizes, with the protocol-chosen reward versus a naive reward as the randomized unit, and compare north-star conversion by a pre-registered horizon. If the denser or aligned reward does not produce faster or higher north-star conversion, or if replay run across many random seeds reverses the reward ordering, the learnability gate and N1 fail.","supporting_citations":[{"cited_title":"Schapire","cited_arxiv_id":null,"evidence_quote":"Supplies the ridge-regression contextual bandit learner used in the replay and in deployment."},{"cited_title":"Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms","cited_arxiv_id":"1003.5956","evidence_quote":"Introduced the replayer mechanism that the paper repurposes to rank rewards by the learning trajectory they induce."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the doubly robust estimator used for policy values and for the achievable-ceiling check."},{"cited_title":"VanderWeele","cited_arxiv_id":null,"evidence_quote":"Provides the surrogate-paradox mechanism behind the P1 arm-level directionality check."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-normalized importance-sampling estimator used for counterfactual value estimates on the randomized log."},{"cited_title":"Hitsch, Sanjog Misra, and Walter W","cited_arxiv_id":null,"evidence_quote":"Defines the value of an optimal targeting policy over the best uniform action, which underlies the achievable-ceiling screen."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines reward informativeness, used to explain why density governs online learning efficiency in N1."}],"review_version":1}