Exploring Offline Policy Evaluation for the Continuous-Armed Bandit Problem

Jules Kruijswijk; Maurits Kaptein; Petri Parvinen

REVIEW 3 major objections 6 minor 30 references

Exploring Offline Policy Evaluation for the Continuous-Armed Bandit Problem

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A δ-window lets offline data rank continuous-armed bandit policies.

desk verdict A sensible δ-window extension of Li et al. with honest limitations, but the central claim of consistent rankings is contradicted by the paper's own Table 2 in the complex model. read the letter →

arxiv 1908.07808 v1 pith:4PANWYCD submitted 2019-08-21 cs.LG stat.ML

Jules Kruijswijk , Petri Parvinen , Maurits Kaptein This is my paper · ORCID

classification cs.LGstat.ML

keywords continuous-armedbanditofflinepolicyevaluationcounterfactualreplaymethodrankingalgorithmsdeltawindowThompsonsampling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an offline evaluation method for the continuous-armed bandit (CAB) problem, where the candidate actions form a continuous range. The method replays a dataset of uniformly random actions and accepts an event when the logged action falls within a tuning parameter δ of the action a policy would have taken, instead of requiring an exact match. In simulations with four policies and two reward functions, the resulting rankings remain largely consistent with the online rankings, so the authors argue the method is useful for selecting among applied CAB policies.

What carries the argument

The load-bearing mechanism is Algorithm 2, the δ-window acceptance rule. It replaces the exact equality check of the original replay evaluator with the distance check |a − Π(h_{t−1})| < δ, giving a positive acceptance probability in a continuous action space, and it updates the policy using the proposed action rather than the logged action. This introduces evaluation noise that grows with δ, and the method's usefulness depends on that noise not destroying the relative ordering of policies. The tuning parameter δ controls the number of accepted events and the magnitude of the bias; the field application selects δ to match an expected number of accepted observations.

What would settle it

Run the offline δ-window evaluation on a logged dataset from a bimodal reward function where two policies are close in online performance; if the offline ranking places the policies in the opposite order from the online ranking at a comparable evaluation length, the ordering guarantee fails — the paper's own Table 2 reports exactly such a reversal (LiF above TBL online, TBL above LiF offline) in its complex model.

Watch

Extended reading notes

Core claim

The paper's central claim is that the δ-window replay algorithm is a workable offline evaluator for continuous-armed bandits. When a logged action a lies within δ of a policy's proposed action Π(h_{t−1}), that event is accepted and the policy is updated using the proposed action and the observed reward. The resulting estimate of a policy's cumulative reward is downwardly biased for concave reward functions, but the relative ordering of policies is preserved across several δ values and two data-generating models. The paper therefore positions the method as a tool for ranking and selecting policies rather than for unbiased performance measurement, and illustrates it with a small field application to choosing rebate splits where a simple ε-first policy ranks first.

Load-bearing premise

The evaluation bias introduced by accepting actions within δ of the policy's choice and updating with the proposed action does not change the relative order of the policies being compared.

Editorial extensions

If this is right

Practitioners can compare multiple continuous-action policies on a single logged dataset, avoiding the cost of repeated field trials.
The output is a ranking, not an unbiased value estimate, so the method supports policy selection rather than performance auditing.
Because δ does not affect data collection, evaluations can be re-run at several δ values on the same dataset to check whether rankings are stable.
In the field example, the ranking favored a simple ε-first policy, consistent with earlier findings that simple heuristics often win in small-sample applied problems.
The reliability of the ordering can differ with the complexity of the reward function, so applying the method to multimodal or high-curvature problems calls for checking δ sensitivity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The δ-window rule can be viewed as a moving bin centered on the policy's own action, making the method a policy-centered alternative to fixed binning of the action space; one could replace the hard threshold with a kernel weight based on |a − Π| to reduce bias while retaining more data.
The paper's own Table 2 shows a reversal between TBL and LiF in the complex model, so the ordering guarantee is not universal; a practitioner should compare confidence bands on cumulative reward rather than trusting point ranks.
Ranking reliability should depend on reward-function curvature and smoothness: sharply peaked or multimodal reward functions are more likely to distort the relative ordering under δ-noise, which is a testable prediction.
The method could be extended to contextual CAB problems by applying the same distance check within context-dependent policies, though that would require logged contexts and more data to keep acceptance rates acceptable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

You should know two things about this paper. First, it builds a genuinely useful bridge: it takes Li et al.'s exact-match offline policy evaluator for discrete MABs and relaxes it to continuous action spaces via a tolerance window δ. That is a simple and plausible idea, and the paper is admirably transparent about the fact that the resulting estimates are biased and that δ has no principled selection rule. Second, the main empirical claim — that policy rankings stay 'relatively intact' — is not actually supported by the paper's own results.

The new ingredient is Algorithm 2. Instead of accepting only logged actions that exactly equal the policy's proposed action, it accepts any logged action within δ, and updates the policy with the proposed action rather than the logged one. Section 2.1 shows the reward estimate is downwardly biased for concave reward functions, which the authors freely admit. That is fine for coarse comparisons, but it is not a proof of ranking preservation.

The simulations use four policies and two reward functions: a simple parabola and a bimodal polynomial. In the simple model, the offline ranking matches the online ranking for δ = .5, .2, .1. In the complex model, Table 2 tells a different story: online, LiF ranks above TBL; offline, TBL ranks above LiF for every δ. That is a ranking reversal, and it directly contradicts the 'relatively intact' claim. The authors note TBL is 'more robust to the noise introduced by the offline evaluation method,' which may be true, but it means the offline ordering is not the online ordering. The evidence only supports coarse separation between clearly good (TBL/LiF) and clearly bad (EF/UR) policies.

The second soft spot is the policy-update mismatch. Algorithm 2 inserts the proposed action into the policy's history, not the logged action. That changes the policy's internal state, so the reversal could be an artifact of the update rule rather than the reward bias. The paper mentions this in passing but never isolates it. An ablation — for instance, updating with the logged action or reweighting by distance — would clarify.

There is also no shared code or data for the simulations, and the field study, while a nice illustration, has no ground truth and cannot validate ranking.

Net: the method is cheap and likely useful for screening out poor policies, but the central claim needs to be softened and the update-rule effect analyzed. I would send this to a workshop or a short applied paper, not a top venue as is. For anyone working on offline evaluation for continuous arms, it is worth reading.

Referee Report

3 major / 6 minor

Summary. This paper addresses offline policy evaluation for continuous-armed bandits (CABs). The authors propose a δ-tolerance modification of Li et al.'s replay method: a logged event (a, r_a) is accepted when the policy's proposed action is within δ of a, and the policy is updated using the proposed action and the logged reward. They report two simulation studies (unimodal and bimodal reward functions) comparing four policies under online and offline evaluation, plus a field deployment with a rebate company. The paper claims that the method yields a "relatively consistent ranking" of CAB policies and can therefore help select policies without field trials.

Significance. If the ranking claim held, the method would be a cheap and practical tool for CAB policy selection, extending Li et al.'s unbiased offline evaluation to continuous action spaces. The paper is transparent about the main bias (rewards are evaluated at a different point than the policy's action), provides fully specified algorithms, and honestly lists limitations (δ selection, multi-dimensional actions, bias correction). However, the central empirical claim is not supported by the reported evidence: the complex-model results contain an explicit rank reversal between TBL and LiF, and the fixed comparison horizon T=1750 makes the EF/UR tie a design artifact. The core idea is reasonable, but the manuscript needs either stronger evidence or a more modest claim.

major comments (3)

[§3.3, Table 2] The offline ranking reverses the online ranking between TBL and LiF in the complex model: online LiF is first and TBL second, while for δ = .5, .2, and .1 the offline evaluation ranks TBL first and LiF second. This is a direct counterexample to the Section 5 claim that "the rank ordering of CAB policies stays relatively intact." No statistical test or uncertainty measure is attached to the ranks, and Table 1 even uses question marks at δ=.5. The evidence supports at most a coarse grouping of policies, not a consistent fine-grained ranking; the conclusion should be tempered or a formal criterion for "relatively intact" provided.
[§3.1, Tables 1 and 2] The rank comparison at T = 1750 is made during EF's exploration phase; the note in Table 1 states that at this point EF and UR are tied "by design," and the text in Section 3.2 indicates that EF moves to exploitation at 2,000 observations. Consequently the separation between {TBL, LiF} and {EF, UR} is largely a horizon artifact: a policy that is intentionally random for the first 2,000 interactions is expected to look like UR regardless of its post-exploitation performance. The authors should also report ranks at a horizon beyond EF's exploration stage, or at several horizons, and show that the claimed ordering is not an artifact of evaluating at T=1750.
[Algorithm 2 and §2.1] The procedure updates the policy with the proposed action Π(h_{t−1}) while the reward r_a is generated at the logged action a. Section 2.1 only demonstrates downward bias in expected reward for concave f, which says nothing about whether rankings are preserved; the reward substitution and the history/update mismatch are two separate bias sources that could each reorder policies. The paper needs an ablation (e.g., updating with the logged action a versus the proposed action) or a formal monotonicity argument before the ranking claim can be supported. This is not a minor presentation issue because Algorithm 2 defines the method.

minor comments (6)

[§2] The formula in the sentence "with length L = 2δ/(a−b)/T′" appears to have the reciprocal and sign wrong; the preceding expression for p_accept implies L = (b−a)T′/(2δ).
[Throughout] There are several typographical errors: "Cleary" (Section 1), "the this probability" (Section 1), "UF" instead of "UR" in the Table 1 note, and "content" instead of "contend" in Section 5.
[Figures 2 and 3] The x-axis label "Time" is used for both the online and offline panels; for offline panels it should indicate the number of accepted events T, since different δ values produce different effective evaluation lengths.
[Table 1] The question marks after TBL and LiF at δ=.5 are not explained; please clarify whether those ranks are uncertain due to overlapping confidence intervals or for another reason.
[Algorithm 2] The expression "Π(ht−1, )" in the if statement contains a stray comma; also, Algorithm 3 does not define how the noise term ε_t enters the displayed equations.
[§4] The field application cannot validate the ranking claim because no ground-truth online ranking is available; it is a deployment illustration. The text should state this explicitly so that the reader does not count the field study as evidence for the consistency of the offline ranking.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the offline CAB evaluation method is an empirical proposal whose ranking claim is tested against simulation, not derived from the fitted data or from self-cited theorems.

full rationale

The paper's central contribution is Algorithm 2, an adaptation of Li et al.'s (2011) exact-match rejection sampler to continuous actions via a δ-window. No parameter is fitted and then renamed a prediction: δ is a user-chosen tuning parameter, and the paper explicitly explores multiple values of δ. The claim that the method 'provides a relatively consistent ranking of policies' is an empirical observation made by comparing offline regret curves against an online simulation benchmark (Tables 1 and 2), not a consequence of the definitions. The bias analysis in Section 2.1 establishes only that large δ gives noisy reward evaluations; it does not define the ranking result. The self-citations (LiF algorithm, StreamingBandit) describe implementations and benchmark policies; they are not invoked as uniqueness theorems or as proof of the offline-ranking claim, so they are not load-bearing. The paper's own limitations—no principled δ selection, sensitivity to data-generating complexity, and the policy-update mismatch in Algorithm 2—are acknowledged validity threats rather than circular steps. Because the conclusion is an empirical finding that could in principle be contradicted (and indeed Table 2 shows a reversal between LiF and TBL), it is not equivalent to its inputs by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on δ and on the uniform-random logging and smoothness assumptions. No new entities are introduced.

free parameters (1)

δ (tolerance window) = chosen as 0.01, 0.05, 0.1, 0.2, 0.5 in simulations; 0.1 in field study
The method's acceptance probability, bias, and effective evaluation length T all depend on δ; the paper provides no principled way to choose it (conclusion: 'no clear guidance on how to set δ').

assumptions (4)

domain assumption Actions in the logged dataset S are drawn uniformly at random from [a,b].
Required for the acceptance probability p_accept = 2δ/(b-a) and for the Li et al. baseline; in the field study this was arranged.
domain assumption The true reward function f() is stationary over the logging period.
Needed to treat the logged data as valid for evaluating a policy; not tested.
domain assumption f() is smooth enough that rewards at actions within δ of the proposed action approximate the reward at the proposed action.
The method's rationale in Section 2.1 relies on this without formally stating or verifying it.
standard math Reward noise ε has mean zero.
Used in the regret definition and in interpreting average rewards.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Offline Policy Evaluation for the Continuous-Armed Bandit Problem." pith.science (2026). https://pith.science/paper/4PANWYCD

@misc{pith2026190807808,
  author       = {Pith},
  title        = {Pith review of: Exploring Offline Policy Evaluation for the Continuous-Armed Bandit Problem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PANWYCD}},
  note         = {Machine review of arXiv:1908.07808}
}

read the original abstract

The (contextual) multi-armed bandit problem (MAB) provides a formalization of sequential decision-making which has many applications. However, validly evaluating MAB policies is challenging; we either resort to simulations which inherently include debatable assumptions, or we resort to expensive field trials. Recently an offline evaluation method has been suggested that is based on empirical data, thus relaxing the assumptions, and can be used to evaluate multiple competing policies in parallel. This method is however not directly suited for the continuous armed (CAB) problem; an often encountered version of the MAB problem in which the action set is continuous instead of discrete. We propose and evaluate an extension of the existing method such that it can be used to evaluate CAB policies. We empirically demonstrate that our method provides a relatively consistent ranking of policies. Furthermore, we detail how our method can be used to select policies in a real-life CAB problem.

Figures

Figures reproduced from arXiv: 1908.07808 by the authors.

**Figure 2.** The results for simulation Study 1 using a simple unimodal data generating model in terms of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** The results for simulation Study 2 using a more complex data generating model in terms of [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Revenue of the participating rebate company as a function of the proposed split. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: Average cumulative reward and confidence bounds of four different CAB policies using our [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

[1]

SIAM journal on control and optimization 33(6):1926--1951

Agrawal R (1995) The continuum-armed bandit problem. SIAM journal on control and optimization 33(6):1926--1951

work page 1995
[2]

CoRR abs/1111.1797

Agrawal S, Goyal N (2011) Analysis of Thompson Sampling for the multi-armed bandit problem . CoRR abs/1111.1797

arXiv 2011
[3]

Springer

Berry DA, Fristedt B (1985) Bandit Problems: Sequential Allocation of Experiments . Springer

work page 1985
[4]

In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, (AISTATS-11), pp 19--26

Beygelzimer A, Langford J, Li L, Reyzin L, Schapire RE (2011) Contextual bandit algorithms with supervised learning guarantees. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, (AISTATS-11), pp 19--26

work page 2011
[5]

John Wiley & Sons

Box GE, Tiao GC (1992) Bayesian inference in statistical analysis. John Wiley & Sons

work page 1992
[6]

Foundations and Trends in Machine Learning 5(1):1--122

Bubeck S, Cesa-Bianchi N (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning 5(1):1--122

work page 2012
[7]

In: Proceedings of the third ACM international conference on Web search and data mining, ACM, pp 351--360

Cheng H, Cant \'u -Paz E (2010) Personalized click prediction in sponsored search. In: Proceedings of the third ACM international conference on Web search and data mining, ACM, pp 351--360

work page 2010
[8]

In: Advances in Neural Information Processing Systems, pp 1025--1033

Djolonga J, Krause A, Cevher V (2013) High-dimensional gaussian process bandits. In: Advances in Neural Information Processing Systems, pp 1025--1033

work page 2013

Show all 30 references

[9]

Marketing Science 28(2):202--223

Hauser JR, Urban GL, Liberali G, Braun M (2009) Website Morphing . Marketing Science 28(2):202--223

2009
[10]

Journal of artificial intelligence research 4:237--285

Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: A survey. Journal of artificial intelligence research 4:237--285

1996
[11]

CoRR abs/1502.00598

Kaptein M, Ianuzzi D (2015) Lock in feedback in sequential experiments. CoRR abs/1502.00598

2015 arXiv
[12]

CoRR abs/1602.06700

Kaptein M, Kruijswijk J (2016) Streamingbandit: Developing adaptive persuasive systems. CoRR abs/1602.06700

2016 arXiv
[13]

Palgrave Communications 2

Kaptein M, Van Emden R, Iannuzzi D (2016) Tracking the decoy: Maximizing the decoy effect through sequential experimentation. Palgrave Communications 2

2016
[14]

In: Advances in Neural Information Processing Systems, vol 17, pp 697--704

Kleinberg RD (2004) Nearly tight bounds for the continuum-armed bandit problem. In: Advances in Neural Information Processing Systems, vol 17, pp 697--704

2004
[15]

In: Advances in Neural Information Processing Systems, pp 2447--2455

Krause A, Ong CS (2011) Contextual gaussian process bandit optimization. In: Advances in Neural Information Processing Systems, pp 2447--2455

2011
[16]

CoRR abs/1402.6028

Kuleshov V, Precup D (2014) Algorithms for multi-armed bandit problems. CoRR abs/1402.6028

2014 arXiv
[17]

Advances in Applied Mathematics 6(1):4--22

Lai T, Robbins H (1985) Asymptotically efficient adaptive allocation rules . Advances in Applied Mathematics 6(1):4--22

1985
[18]

In: Advances in Neural Information Processing Systems, pp 817--824

Langford J, Zhang T (2008) The epoch-greedy algorithm for multi-armed bandits with side information. In: Advances in Neural Information Processing Systems, pp 817--824

2008
[19]

In: Proceedings of the 19th international conference on World Wide Web, p 10, 1003.0146

Li L, Chu W, Langford J, Schapire RE (2010) A Contextual-Bandit Approach to Personalized News Article Recommendation . In: Proceedings of the 19th international conference on World Wide Web, p 10, 1003.0146

2010 arXiv
[20]

In: Proceedings of the fourth ACM international conference on Web search and data mining, ACM, pp 297--306

Li L, Chu W, Langford J, Wang X (2011) Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In: Proceedings of the fourth ACM international conference on Web search and data mining, ACM, pp 297--306

2011
[21]

In: Proceedings of the National Academy of Sciences, vol 106, pp 22,387--22,392

Press WH (2009) Bandit solutions provide unified ethical models for randomized clinical trials and comparative effectiveness research. In: Proceedings of the National Academy of Sciences, vol 106, pp 22,387--22,392

2009
[22]

American Journal of Physics 62(2):129

Scofield JH (1994) Frequency-domain description of a lock-in amplifier . American Journal of Physics 62(2):129

1994
[23]

Applied Stochastic Models in Business and Industry 26(6):639--658

Scott SL (2010) A modern Bayesian look at the multi-armed bandit . Applied Stochastic Models in Business and Industry 26(6):639--658

2010
[24]

MIT press

Sutton RS, Barto AG (1998) Reinforcement learning: An introduction. MIT press

1998
[25]

In: AAAI, pp 3000--3006

Thomas PS, Theocharous G, Ghavamzadeh M (2015) High-confidence off-policy evaluation. In: AAAI, pp 3000--3006

2015
[26]

Biometrika (3-4):285--294

Thompson WR (1933) On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika (3-4):285--294

1933
[27]

IEEE Transactions on Automatic Control 50(3):338--355

Wang CC, Kulkarni SR, Poor HV (2005) Bandit problems with side observations. IEEE Transactions on Automatic Control 50(3):338--355

2005
[28]

Journal of the Royal Statistical Society Series B Methodological 42(2):143--149

Whittle P (1980) Multi-armed bandits and the gittins index. Journal of the Royal Statistical Society Series B Methodological 42(2):143--149

1980
[29]

Computational Statistics & Data Analysis 113:136--153

Williamson SF, Jacko P, Villar SS, Jaki T (2017) A bayesian adaptive design for clinical trials in rare diseases. Computational Statistics & Data Analysis 113:136--153

2017
[30]

Am J Phys 59(6):569--572

Wolfson R (1991) The lock-in amplifier: A student experiment . Am J Phys 59(6):569--572

1991

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

SIAM journal on control and optimization 33(6):1926--1951

Agrawal R (1995) The continuum-armed bandit problem. SIAM journal on control and optimization 33(6):1926--1951

work page 1995

[2] [2]

CoRR abs/1111.1797

Agrawal S, Goyal N (2011) Analysis of Thompson Sampling for the multi-armed bandit problem . CoRR abs/1111.1797

arXiv 2011

[3] [3]

Springer

Berry DA, Fristedt B (1985) Bandit Problems: Sequential Allocation of Experiments . Springer

work page 1985

[4] [4]

In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, (AISTATS-11), pp 19--26

Beygelzimer A, Langford J, Li L, Reyzin L, Schapire RE (2011) Contextual bandit algorithms with supervised learning guarantees. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, (AISTATS-11), pp 19--26

work page 2011

[5] [5]

John Wiley & Sons

Box GE, Tiao GC (1992) Bayesian inference in statistical analysis. John Wiley & Sons

work page 1992

[6] [6]

Foundations and Trends in Machine Learning 5(1):1--122

Bubeck S, Cesa-Bianchi N (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning 5(1):1--122

work page 2012

[7] [7]

In: Proceedings of the third ACM international conference on Web search and data mining, ACM, pp 351--360

Cheng H, Cant \'u -Paz E (2010) Personalized click prediction in sponsored search. In: Proceedings of the third ACM international conference on Web search and data mining, ACM, pp 351--360

work page 2010

[8] [8]

In: Advances in Neural Information Processing Systems, pp 1025--1033

Djolonga J, Krause A, Cevher V (2013) High-dimensional gaussian process bandits. In: Advances in Neural Information Processing Systems, pp 1025--1033

work page 2013