{"id":"d1f74905-96ee-48d7-bc18-6818196fd7cc","arxiv_id":"1908.07808","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A δ-window extension of Li et al.'s offline bandit evaluation lets logged actions near a policy's choice count, giving a biased but rank-preserving (at coarse level) way to compare continuous-armed bandit policies.","lead":"This paper proposes an offline method to evaluate continuous-armed bandit policies by accepting logged actions within a small distance of the policy's suggested action. The method is tested in simulations and a real-world rebate setting, and it produces a coarse ranking of good versus bad policies, though the fine-grained ranking can flip.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that offline rankings stay 'relatively intact' is undermined by the paper's own TBL/LiF reversal in Table 2 and by the unanalyzed policy-update mismatch in Algorithm 2; an ablation test is needed to see which bias causes the reordering.","rationale":"The reader's conditional verdict already captures the main weakness, and my stress test agrees with its weakest-assumption point: rank preservation rests on an unproven claim that policy-dependent biases do not alter relative orderings. The simulations are transparent, use 1,000 repetitions and multiple δ values, and the paper honestly flags limitations, but the observed TBL/LiF reversal in Table 2 is an in-sample counterexample to the headline claim. The proposed ablation is a minimal experiment that separates the two bias sources: acceptance-window bias versus updating with the proposed action. Because the paper already received a CONDITIONAL verdict, no verdict change is needed; the condition should require either this ablation or a softened conclusion that the method provides only coarse good-versus-bad separation.","tokens_in":11925,"tokens_out":6392,"duration_ms":65273,"concrete_test":"Run an ablation of Algorithm 2 on the 1,000 random f2 reward functions of Simulation Study 2. In arm A, update the policy and stored history with the proposed action Π(h_{t-1}) exactly as in Algorithm 2; in arm B, keep the same δ-acceptance rule but update with the logged action a and store (a, r_a). Compare the offline order of TBL and LiF at T=1750 in each arm with the online order. If the TBL/LiF reversal disappears in arm B, the policy-update mismatch in Algorithm 2 is the cause; if the reversal persists in both arms, the δ-window reward bias alone is responsible. Either outcome would show which unanalyzed assumption in Section 2.1 drives the rank distortion and whether the central claim needs to be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and Section 5, is that the δ-window procedure 'provides a relatively consistent ranking of policies.' For this to hold, the two biases identified in Section 2.1 and Algorithm 2 must not reorder policies: accepting logged actions only within δ substitutes a noisy reward f(a) for f(Π(h_{t-1})), and the policy is updated with the proposed action Π(h_{t-1}) plus r_a instead of with the logged action a. Section 2.1 only shows that the reward bias is downward for concave functions; it does not prove that rankings are preserved. The paper's own Table 2 provides a direct counterexample: in the complex model, LiF ranks above TBL online, but TBL ranks above LiF offline for δ=.5, .2, and .1. Since the offline procedure changes both the reward signal and the policy's internal history, this reversal may be an artifact of the evaluation method rather than a true performance difference. The conclusion that rank ordering 'stays relatively intact' is therefore stronger than the evidence supports; at most, coarse separation of good versus bad policies (TBL/LiF vs EF/UR) is demonstrated. The field application has no ground-truth ranking and cannot validate the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses offline policy evaluation for continuous-armed bandits (CABs). The authors propose a δ-tolerance modification of Li et al.'s replay method: a logged event (a, r_a) is accepted when the policy's proposed action is within δ of a, and the policy is updated using the proposed action and the logged reward. They report two simulation studies (unimodal and bimodal reward functions) comparing four policies under online and offline evaluation, plus a field deployment with a rebate company. The paper claims that the method yields a \"relatively consistent ranking\" of CAB policies and can therefore help select policies without field trials.","tokens_in":12151,"tokens_out":6767,"duration_ms":68779,"significance":"If the ranking claim held, the method would be a cheap and practical tool for CAB policy selection, extending Li et al.'s unbiased offline evaluation to continuous action spaces. The paper is transparent about the main bias (rewards are evaluated at a different point than the policy's action), provides fully specified algorithms, and honestly lists limitations (δ selection, multi-dimensional actions, bias correction). However, the central empirical claim is not supported by the reported evidence: the complex-model results contain an explicit rank reversal between TBL and LiF, and the fixed comparison horizon T=1750 makes the EF/UR tie a design artifact. The core idea is reasonable, but the manuscript needs either stronger evidence or a more modest claim.","major_comments":[{"comment":"The offline ranking reverses the online ranking between TBL and LiF in the complex model: online LiF is first and TBL second, while for δ = .5, .2, and .1 the offline evaluation ranks TBL first and LiF second. This is a direct counterexample to the Section 5 claim that \"the rank ordering of CAB policies stays relatively intact.\" No statistical test or uncertainty measure is attached to the ranks, and Table 1 even uses question marks at δ=.5. The evidence supports at most a coarse grouping of policies, not a consistent fine-grained ranking; the conclusion should be tempered or a formal criterion for \"relatively intact\" provided.","section":"§3.3, Table 2"},{"comment":"The rank comparison at T = 1750 is made during EF's exploration phase; the note in Table 1 states that at this point EF and UR are tied \"by design,\" and the text in Section 3.2 indicates that EF moves to exploitation at 2,000 observations. Consequently the separation between {TBL, LiF} and {EF, UR} is largely a horizon artifact: a policy that is intentionally random for the first 2,000 interactions is expected to look like UR regardless of its post-exploitation performance. The authors should also report ranks at a horizon beyond EF's exploration stage, or at several horizons, and show that the claimed ordering is not an artifact of evaluating at T=1750.","section":"§3.1, Tables 1 and 2"},{"comment":"The procedure updates the policy with the proposed action Π(h_{t−1}) while the reward r_a is generated at the logged action a. Section 2.1 only demonstrates downward bias in expected reward for concave f, which says nothing about whether rankings are preserved; the reward substitution and the history/update mismatch are two separate bias sources that could each reorder policies. The paper needs an ablation (e.g., updating with the logged action a versus the proposed action) or a formal monotonicity argument before the ranking claim can be supported. This is not a minor presentation issue because Algorithm 2 defines the method.","section":"Algorithm 2 and §2.1"}],"minor_comments":[{"comment":"The formula in the sentence \"with length L = 2δ/(a−b)/T′\" appears to have the reciprocal and sign wrong; the preceding expression for p_accept implies L = (b−a)T′/(2δ).","section":"§2"},{"comment":"There are several typographical errors: \"Cleary\" (Section 1), \"the this probability\" (Section 1), \"UF\" instead of \"UR\" in the Table 1 note, and \"content\" instead of \"contend\" in Section 5.","section":"Throughout"},{"comment":"The x-axis label \"Time\" is used for both the online and offline panels; for offline panels it should indicate the number of accepted events T, since different δ values produce different effective evaluation lengths.","section":"Figures 2 and 3"},{"comment":"The question marks after TBL and LiF at δ=.5 are not explained; please clarify whether those ranks are uncertain due to overlapping confidence intervals or for another reason.","section":"Table 1"},{"comment":"The expression \"Π(ht−1, )\" in the if statement contains a stray comma; also, Algorithm 3 does not define how the noise term ε_t enters the displayed equations.","section":"Algorithm 2"},{"comment":"The field application cannot validate the ranking claim because no ground-truth online ranking is available; it is a deployment illustration. The text should state this explicitly so that the reader does not count the field study as evidence for the consistency of the offline ranking.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the method has practical potential, but the advertised ranking result overstates what the tables show. I would ask the authors to (i) define and statistically assess \"relatively intact\" rankings, (ii) run an ablation separating the two bias sources in Algorithm 2, and (iii) report additional horizons beyond EF's exploration phase. If these can be addressed, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know two things about this paper. First, it builds a genuinely useful bridge: it takes Li et al.'s exact-match offline policy evaluator for discrete MABs and relaxes it to continuous action spaces via a tolerance window δ. That is a simple and plausible idea, and the paper is admirably transparent about the fact that the resulting estimates are biased and that δ has no principled selection rule. Second, the main empirical claim — that policy rankings stay 'relatively intact' — is not actually supported by the paper's own results.\n\nThe new ingredient is Algorithm 2. Instead of accepting only logged actions that exactly equal the policy's proposed action, it accepts any logged action within δ, and updates the policy with the proposed action rather than the logged one. Section 2.1 shows the reward estimate is downwardly biased for concave reward functions, which the authors freely admit. That is fine for coarse comparisons, but it is not a proof of ranking preservation.\n\nThe simulations use four policies and two reward functions: a simple parabola and a bimodal polynomial. In the simple model, the offline ranking matches the online ranking for δ = .5, .2, .1. In the complex model, Table 2 tells a different story: online, LiF ranks above TBL; offline, TBL ranks above LiF for every δ. That is a ranking reversal, and it directly contradicts the 'relatively intact' claim. The authors note TBL is 'more robust to the noise introduced by the offline evaluation method,' which may be true, but it means the offline ordering is not the online ordering. The evidence only supports coarse separation between clearly good (TBL/LiF) and clearly bad (EF/UR) policies.\n\nThe second soft spot is the policy-update mismatch. Algorithm 2 inserts the proposed action into the policy's history, not the logged action. That changes the policy's internal state, so the reversal could be an artifact of the update rule rather than the reward bias. The paper mentions this in passing but never isolates it. An ablation — for instance, updating with the logged action or reweighting by distance — would clarify.\n\nThere is also no shared code or data for the simulations, and the field study, while a nice illustration, has no ground truth and cannot validate ranking.\n\nNet: the method is cheap and likely useful for screening out poor policies, but the central claim needs to be softened and the update-rule effect analyzed. I would send this to a workshop or a short applied paper, not a top venue as is. For anyone working on offline evaluation for continuous arms, it is worth reading.","headline":"A sensible δ-window extension of Li et al. with honest limitations, but the central claim of consistent rankings is contradicted by the paper's own Table 2 in the complex model.","tokens_in":12707,"tokens_out":4352,"would_cite":false,"duration_ms":38533,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A δ-window lets offline data rank continuous-armed bandit policies.","keywords":["continuous-armed bandit","offline policy evaluation","counterfactual evaluation","replay method","policy ranking","bandit algorithms","delta window","Thompson sampling"],"falsifier":"Run the offline δ-window evaluation on a logged dataset from a bimodal reward function where two policies are close in online performance; if the offline ranking places the policies in the opposite order from the online ranking at a comparable evaluation length, the ordering guarantee fails — the paper's own Table 2 reports exactly such a reversal (LiF above TBL online, TBL above LiF offline) in its complex model.","tokens_in":11710,"feed_emoji":"🎰","tokens_out":8134,"duration_ms":73634,"temperature":0.7,"pith_summary":"This paper proposes an offline evaluation method for the continuous-armed bandit (CAB) problem, where the candidate actions form a continuous range. The method replays a dataset of uniformly random actions and accepts an event when the logged action falls within a tuning parameter δ of the action a policy would have taken, instead of requiring an exact match. In simulations with four policies and two reward functions, the resulting rankings remain largely consistent with the online rankings, so the authors argue the method is useful for selecting among applied CAB policies.","feed_headline":"Offline replay ranks continuous-action bandit policies","feed_subtitle":"Accepting near-matching actions preserves enough ordering to pick a policy without new field trials.","key_machinery":"The load-bearing mechanism is Algorithm 2, the δ-window acceptance rule. It replaces the exact equality check of the original replay evaluator with the distance check |a − Π(h_{t−1})| < δ, giving a positive acceptance probability in a continuous action space, and it updates the policy using the proposed action rather than the logged action. This introduces evaluation noise that grows with δ, and the method's usefulness depends on that noise not destroying the relative ordering of policies. The tuning parameter δ controls the number of accepted events and the magnitude of the bias; the field application selects δ to match an expected number of accepted observations.","core_discovery":"The paper's central claim is that the δ-window replay algorithm is a workable offline evaluator for continuous-armed bandits. When a logged action a lies within δ of a policy's proposed action Π(h_{t−1}), that event is accepted and the policy is updated using the proposed action and the observed reward. The resulting estimate of a policy's cumulative reward is downwardly biased for concave reward functions, but the relative ordering of policies is preserved across several δ values and two data-generating models. The paper therefore positions the method as a tool for ranking and selecting policies rather than for unbiased performance measurement, and illustrates it with a small field application to choosing rebate splits where a simple ε-first policy ranks first.","pith_inferences":["The δ-window rule can be viewed as a moving bin centered on the policy's own action, making the method a policy-centered alternative to fixed binning of the action space; one could replace the hard threshold with a kernel weight based on |a − Π| to reduce bias while retaining more data.","The paper's own Table 2 shows a reversal between TBL and LiF in the complex model, so the ordering guarantee is not universal; a practitioner should compare confidence bands on cumulative reward rather than trusting point ranks.","Ranking reliability should depend on reward-function curvature and smoothness: sharply peaked or multimodal reward functions are more likely to distort the relative ordering under δ-noise, which is a testable prediction.","The method could be extended to contextual CAB problems by applying the same distance check within context-dependent policies, though that would require logged contexts and more data to keep acceptance rates acceptable."],"forward_implications":["Practitioners can compare multiple continuous-action policies on a single logged dataset, avoiding the cost of repeated field trials.","The output is a ranking, not an unbiased value estimate, so the method supports policy selection rather than performance auditing.","Because δ does not affect data collection, evaluations can be re-run at several δ values on the same dataset to check whether rankings are stable.","In the field example, the ranking favored a simple ε-first policy, consistent with earlier findings that simple heuristics often win in small-sample applied problems.","The reliability of the ordering can differ with the complexity of the reward function, so applying the method to multimodal or high-curvature problems calls for checking δ sensitivity."],"supporting_citations":[{"why":"Supplies the original exact-match replay evaluator for offline MAB policy evaluation that this paper extends to continuous actions.","marker":"Li et al (2011)"},{"why":"Provides the high-confidence off-policy evaluation alternative that the paper argues cannot be applied directly to continuous actions.","marker":"Thomas, Theocharous, and Ghavamzadeh (2015)"},{"why":"Defines the continuum-armed bandit problem and the continuous action setting the paper targets.","marker":"Agrawal (1995)"},{"why":"Introduces the LiF policy, one of the four policies used in the simulations.","marker":"Kaptein and Ianuzzi (2015)"},{"why":"Describes the simulation and deployment platform used to collect data and run the evaluations.","marker":"Kaptein and Kruijswijk (2016)"},{"why":"Documents the small-sample advantage of simple heuristics used to interpret the field result favoring ε-first.","marker":"Kuleshov and Precup (2014)"},{"why":"Specifies the Thompson sampling scheme underlying the TBL policy evaluated in the simulations.","marker":"Scott (2010)"},{"why":"Provides the Bayesian linear regression update used by the TBL policy.","marker":"Box and Tiao (1992)"}],"fun_headline_variants":["Offline δ-window replay ranks continuous bandit policies","Near-matching actions preserve policy ranking in offline bandit evaluation","Biased offline scores still order continuous bandit policies","Tolerance-based replay evaluates continuous-armed bandits offline","Offline ranking of continuous bandit policies without field trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation bias introduced by accepting actions within δ of the policy's choice and updating with the proposed action does not change the relative order of the policies being compared.","fun_headline_variants_meta":{"raw":{"variants":["Offline δ-window replay ranks continuous bandit policies","Near-matching actions preserve policy ranking in offline bandit evaluation","Biased offline scores still order continuous bandit policies","Tolerance-based replay evaluates continuous-armed bandits offline","Offline ranking of continuous bandit policies without field trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3310,"prompt_tokens":833,"completion_tokens":2477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2396}},"tokens_in":449,"tokens_out":2477,"duration_ms":17999,"temperature":1.0,"reasoning_tokens":2396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:55:13.199380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the offline δ-window evaluation on a logged dataset from a bimodal reward function where two policies are close in online performance; if the offline ranking places the policies in the opposite order from the online ranking at a comparable evaluation length, the ordering guarantee fails — the paper's own Table 2 reports exactly such a reversal (LiF above TBL online, TBL above LiF offline) in its complex model.","supporting_citations":[{"cited_title":"In: AAAI, pp 3000--3006","cited_arxiv_id":null,"evidence_quote":"Provides the high-confidence off-policy evaluation alternative that the paper argues cannot be applied directly to continuous actions."},{"cited_title":"SIAM journal on control and optimization 33(6):1926--1951","cited_arxiv_id":null,"evidence_quote":"Defines the continuum-armed bandit problem and the continuous action setting the paper targets."},{"cited_title":"Lock in Feedback in Sequential Experiments","cited_arxiv_id":"1502.00598","evidence_quote":"Introduces the LiF policy, one of the four policies used in the simulations."},{"cited_title":"Applied Stochastic Models in Business and Industry 26(6):639--658","cited_arxiv_id":null,"evidence_quote":"Specifies the Thompson sampling scheme underlying the TBL policy evaluated in the simulations."},{"cited_title":"John Wiley & Sons","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian linear regression update used by the TBL policy."}],"review_version":1}