Pith. sign in

REVIEW 4 major objections 5 minor 50 references

From Theory to Practice with RAVEN-UCB: Addressing Non-Stationarity in Multi-Armed Bandits through Variance Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a variance-adaptive upper confidence bound algorithm, RAVEN-UCB, achieves regret bounds tighter than UCB1 and UCB-V and performs well in non-stationary bandit environments.

desk verdict The paper's advertised regret bounds rest on an impossible algebraic simplification, so the theory collapses; the empirical work is not strong enough to carry the paper. read the letter →

arxiv 2506.02933 v1 pith:YTMIQ7F5 submitted 2025-06-03 cs.LG stat.ML

Junyi Fang , Yuxun Chen , Yuxin Chen , Chen Zhang This is my paper · ORCID
classification cs.LGstat.ML MSC 62L0568T05
keywords multi-armedbanditsnon-stationaryenvironmentsvariance-adaptiveexplorationupperconfidenceboundregretboundsconceptdriftlogisticsoptimizationrecursivevarianceupdate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-armed bandit policy can get both tighter regret bounds and better practical performance in changing environments by using the sample variance of each arm as an exploration signal. The proposed algorithm, RAVEN-UCB, adds a variance term proportional to $\sqrt{\hat{\sigma}_k^2/(N_k+1)}$ and a logarithmically decaying exploration coefficient to the usual upper confidence bound score. On the theory side, it reports regret bounds of order $K\sigma_{\max}^2\log T/\Delta$ when the gap $\Delta$ between the best and second-best arm is fixed, and $\sqrt{K T\log T}$ in the worst case, which it says are tighter than the corresponding UCB1 and UCB-V bounds. On the practical side, recursive mean and variance updates keep each step at constant cost, and simulations across distributional changes, periodic shifts, and temporary fluctuations show lower regret than several baselines, including a logistics scenario with one hundred warehouses.

What carries the argument

The object that carries the argument is the upper confidence bound (UCB) index $\mathrm{score}(k) = M(k) + \alpha_t \sqrt{\ln(t+1)/(N(k)+1)} + \beta_0 \sqrt{S^2(k)/(N(k)+1+\epsilon)}$, with $\alpha_t = \alpha_0/\log(t+\epsilon)$, which combines the empirical mean, a log-time exploration term, and a variance exploration term. Two recursive identities, $\bar{X}_{n+1} = \bar{X}_n + (X_{n+1}-\bar{X}_n)/(n+1)$ and $S^2_{n+1} = (1-1/n)S^2_n + (n+1)(\bar{X}_{n+1}-\bar{X}_n)^2$, let the algorithm update both statistics in constant time. The regret proof's load-bearing simplification is the claim that the confidence radius collapses to $\sqrt{4\sigma_k^2\log t / N_k(t)}$ for all $t$, which requires $\alpha_0\sqrt{\ln t} + \beta_0\sigma_k = 2\sigma_k\sqrt{\ln t}$.

What would settle it

At two horizons, say $t=10$ and $t=10^4$, evaluate $\alpha_0\sqrt{\ln t} + \beta_0\sigma$ against $2\sigma\sqrt{\ln t}$ for any fixed positive $\alpha_0$, $\beta_0$, and $\sigma$: the ratio $(\alpha_0\sqrt{\ln t} + \beta_0\sigma)/(\sigma\sqrt{\ln t}) = \alpha_0/\sigma + \beta_0/\sqrt{\ln t}$ changes with $t$, so the equality cannot hold at both points. A direct test is to replace the simplified confidence radius in Eq. (27) with the full radius from Eq. (26) and re-derive the expected suboptimal pulls, checking whether the bound acquires extra terms that grow with $t$ or requires a different parameter schedule.

Watch

Extended reading notes

Core claim

RAVEN-UCB's central claim is that replacing the standard log-time exploration term with a variance-scaled term, simplified to $\sqrt{4\sigma_k^2\log t / N_k(t)}$, lets the algorithm concentrate exploration on arms whose rewards are genuinely uncertain, giving suboptimal-pull counts of order $(\sigma_k^2/\Delta_k^2)\log T$ per arm. The paper derives this by decomposing regret as $\sum_k \Delta_k \, E[N_k(T)]$ and bounding events where the optimal arm's confidence bound falls below its mean or where a suboptimal arm's bound exceeds it, using a sub-Gaussian concentration inequality. It then states gap-dependent and gap-independent regret bounds and claims they dominate the corresponding UCB1 and UCB-V bounds. Empirically, it reports that the variance-aware pull count yields roughly $1-\sigma_{\max}^2$ regret reduction for Bernoulli rewards and that the algorithm ranks first among nine algorithms in a non-stationary logistics simulation.

Load-bearing premise

The regret bound rests on an equality, $\alpha_0\sqrt{\ln t} + \beta_0\sigma_k = 2\sigma_k\sqrt{\ln t}$, that is supposed to hold at every time step even though $\alpha_0$ and $\beta_0$ are fixed constants and $\sqrt{\ln t}$ grows; if that simplification fails, the stated rates are not proven.

Editorial extensions

If this is right

  • Suboptimal arms get pulled on the order of $(\sigma_k^2/\Delta_k^2)\log T$ times, a factor $\sigma_k^2$ smaller than UCB1's $(1/\Delta_k^2)\log T$, so regret scales with the largest arm variance rather than with the inverse gap.
  • The gap-independent bound $O(\sqrt{K T\log T})$ matches UCB1 and UCB-V, meaning the variance-aware exploration does not worsen the worst-case regret in this analysis.
  • The logarithmically decaying exploration coefficient $\alpha_t = \alpha_0/\log(t+\epsilon)$ makes exploration fade over time, which the paper argues helps in non-stationary environments where early uncertainty is high and later exploitation is safer.
  • Constant-time recursive updates make the per-step computation independent of history length, so the algorithm can be applied at the scale of the 100-warehouse logistics simulation.
  • In the paper's experiments, RAVEN-UCB reports roughly an 84 percent regret reduction over UCB1 with Bernoulli rewards and the lowest cumulative regret among nine baselines in the logistics scenario.
  • If the simulation results generalize, RAVEN-UCB offers a drop-in replacement for UCB-style policies in dynamic settings, combining the theoretical regret rate with constant-time updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corrected proof would likely need to treat $\alpha_t$ and $\beta_0$ as a time-dependent pair, or bound the un-simplified confidence radius; the stated $O(K\sigma_{\max}^2\log T/\Delta)$ rate is not established unless the simplifying equality is repaired.
  • The full-history variance estimate reacts slowly to abrupt regime changes, so pairing the variance term with a sliding-window or discounted variance estimator is a natural testable extension that might improve performance on the periodic and blip scenarios the paper considers.
  • The paper's hyperparameter guidance can be read as a scheduling rule, with $\alpha_0$ growing with the horizon and $\beta_0$ falling with volatility; an automated online scheduler for $\alpha_t$ and $\beta_t$ could be compared against the fixed-parameter version across the three non-stationarity classes.
  • The predicted regret reduction of $1-\sigma_{\max}^2$ is calibrated to Bernoulli rewards with means near one; testing RAVEN-UCB with sub-Gaussian rewards of different variance ranges would show whether that formula is specific to the experiment or a general property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. RAVEN-UCB is proposed as a variance-adaptive UCB algorithm whose arm score adds a logarithmically decaying exploration coefficient α_t = α0/log(t+ε) and a variance-proportional bonus β0 sqrt(σ̂²/(N+1)) to the empirical mean, with recursive O(1) updates for mean and variance. The paper claims gap-dependent regret O(K σ_max² log T / Δ) and gap-independent regret O(√(KT log T)), asserting these bounds are tighter than those of UCB1 and UCB-V. Three experiments are presented: a Bernoulli-reward regret comparison reporting an 84% regret reduction over UCB1, a hyperparameter sensitivity study across three non-stationary scenarios, and a 100-warehouse logistics simulation with normal rewards in which RAVEN-UCB reports the lowest cumulative regret. The theoretical proof in Appendix A.2 is the basis for the claimed bounds.

Significance. If the theoretical claims were valid, RAVEN-UCB would be an attractive, simple variance-adaptive policy with low per-step computational cost and a provable regret advantage in certain stationary settings. The recursive update formulas appear correct, the code is explicitly made available, and the empirical setup is clearly described, which are strengths for reproducibility. However, the central proof is not valid as written: the reduction of the confidence radius to sqrt(4σ_k² log t / N_k(t)) in Eq. (27) is algebraically impossible with fixed hyperparameters, and the non-stationary performance claims are not covered by the proof at all. The gap-independent bound is also derived heuristically. The paper therefore does not currently establish its main theoretical or non-stationary claims.

major comments (4)
  1. [Appendix A.2, Eq. (27)] The simplification of the confidence radius is invalid. From Eq. (26), c_k(t) = α_t sqrt(log(t)/(N_k(t)+1)) + β0 sqrt(σ̂_k²/(N_k(t)+1+ε)) with α_t = α0/log(t+ε). The manuscript then states 'Suppose α0 sqrt(ln t) + β0 σ_k = 2 σ_k sqrt(ln t)' and obtains c_k(t) = sqrt(4σ_k² log t / N_k(t)). This omits the factor 1/log(t+ε) multiplying α0 and ignores the +ε and +1 offsets. For fixed α0 and β0, the identity α0 sqrt(ln t) + β0 σ_k = 2 σ_k sqrt(ln t) cannot hold for all t because the left-hand side is bounded or tends to zero while the right-hand side grows without bound. Since Eq. (27) is used to derive the 1/t² tail probabilities in Eqs. (30)–(31), the expected sub-pull count, and the gap-dependent bound in Eq. (32), the advertised regret bounds in Section 3.4 and Table 2 are not established by the presented proof. No alternative argument is supplied.
  2. [Sections 2.2, 3.4, and Appendix A.2] The regret analysis does not treat non-stationarity. The proof assumes fixed means µ_k, fixed gaps Δ_k, and a fixed sub-Gaussian parameter σ_k² for each arm; there is no change-point model, drift model, periodic variation, or blip model anywhere in the derivation. Yet the abstract and Section 3.2 claim that RAVEN-UCB is designed for and validated in non-stationary environments. The variance-adaptive exploration is a heuristic in that setting, and no regret bound is provided under the DPC/PC/TF classes defined in Section 2.2. The non-stationary performance claims therefore have no theoretical support in this manuscript.
  3. [Appendix A.2, Eqs. (33)–(36)] The gap-independent regret bound is derived heuristically, not proven. The text 'assume E[N_k(T)] ≈ l' and 'Δ_k ≈ c_k(t)' replaces a random pull count with a single value and treats the gap as equal to the confidence radius; the subsequent Cauchy-Schwarz step in Eq. (34) and the substitutions leading to Eq. (36) are not a rigorous regret derivation. The claimed O(√(KT log T)) gap-independent bound therefore lacks a valid proof, even if the earlier gap-dependent argument were repaired.
  4. [Section 4.1 and Eq. (9)] The empirical 84% regret-reduction claim is presented as confirmation of the theory, but it is a consistency check rather than a parameter-free prediction. The explanation uses the true values of σ_i² and σ_max², which are known only because the simulation means θ_k are generated by the authors; additionally, the hyperparameters α0, β0, and ε were tuned with Optuna per configuration. The reported reduction therefore depends on in-hindsight knowledge and tuning, not on the theoretical constants alone. The paper should report results with fixed, theory-prescribed hyperparameters (or a clear sensitivity range) and provide confidence intervals over independent trials.
minor comments (5)
  1. [Algorithm 1 and Eq. (26)] The score formula in Algorithm 1 uses sqrt(ln(t+1)/(N(k)+1)) and a variance denominator N(k)+1+ε, while the proof in Eq. (26) writes sqrt(ln(t)/(N_k(t)+1)) and sqrt(σ̂_k²/(N_k(t)+1+ε)); these should be aligned exactly, including whether the logarithmic term is ln(t) or ln(t+1).
  2. [Throughout] The algorithm name is written inconsistently as 'RAVEN-UCB', 'RA VEN-UCB', and 'Raven-UCB'; the manuscript should use one consistent spelling.
  3. [Table 2] The regret bounds are compared without explicit constants, which is acceptable for order notation but potentially misleading because constants matter for practical comparison; state the actual UCB1 and UCB-V constants or note that only the dependence on K, T, and Δ is being compared.
  4. [References] Several citations appear imprecise: reference [45] for 'WLS + Optimistic TS' points to Chapelle and Li, but that paper is not obviously the source of that baseline, and the 'CCB' baseline is cited to a Monte-Carlo tree search paper [1]; please verify that each baseline is cited to its actual source.
  5. [Figure 4] The caption contains a typo, 'umulative reward' instead of 'cumulative reward', and the boxplot in Figure 5 is referenced in the text but not described in the caption or the body.

Circularity Check

1 steps flagged · score 3.0 of 10

The theoretical regret analysis is not circular, but the empirical '84% regret reduction' claim is partially circular because it is evaluated using the known simulation variance and tuned hyperparameters as if it were an independent prediction.

  1. fitted input called prediction [Section 4.1, Eq. (9), Figure 2]
    "The asymptotic improvement ratio converges to 1− σ2 max. In our Bernoulli experiment, arm variances p(1 − p) for p ∼ Uniform(0.8, 0.95) lie in [0.05, 0.16], giving σ2 max ≈ 0.16, so the regret reduction is approximately 1 − σ2 max = 0.84, i.e., 84% regret reduction, which aligns with both theory and empirical results in Figure 2."

    The 84% figure is presented as a theoretical prediction matching the experiments, but the σ_max used in the calculation is read off from the same known Bernoulli parameters that generated the simulation, and the algorithm's hyperparameters were tuned on the same task via Optuna before the comparison was reported. Equation (9) is not an independent rate law; it is the paper's own algebraic subtraction of the assumed UCB1 pull count log T/Δ² and the assumed RAVEN-UCB pull count σ² log T/Δ². Thus the claimed agreement is a consistency check between the simulation's generating variances and the paper's assumed constants, not a parameter-free prediction. The central regret bound itself is not circular, though it contains a separate algebraic flaw in Appendix A.2.

full rationale

The actual regret-bound derivation is attempted from concentration inequalities and is not circular: the confidence radius is simplified to Eq. (27), then used to bound P(E_k(t)) and expected sub-pulls. That chain is not equivalent to its inputs by construction, although the simplification "Suppose α0√ln t + β0σ_k = 2σ_k√ln t" is algebraically unjustifiable; that is a correctness problem, not a circularity problem. No load-bearing self-citation chain appears in the paper. The one genuinely circular element is empirical: the 84% regret-reduction claim is computed from the known generative variances of the synthetic experiment after the same experiment was used for hyperparameter tuning, so the experiment largely restates the inputs rather than providing an independent test. This is a localized fitted-input-as-prediction issue and does not undermine the derivation chain itself, hence a score of 3 rather than a higher circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the sub-Gaussian assumption, the invalid confidence-radius simplification, and standard concentration results. The hyperparameters α0, β0, and ε are tuned ad hoc in experiments and are free parameters that the theory cannot pin down.

free parameters (3)
  • alpha0 = tuned per scenario (e.g., 0.5 to 5.0)
    Scales the UCB log-exploration term; the theory requires it to satisfy an impossible equality with sqrt(log t).
  • beta0 = tuned per scenario (e.g., 0.5 to 10.0)
    Weights the variance exploration bonus; no theoretical guidance is provided.
  • epsilon = tuned in [1e-3, 0.5]
    Small constant in the log denominator and inside the variance square root; its placement inside the sqrt in Algorithm 1 creates a permanent exploration offset.
assumptions (4)
  • domain assumption Each arm's rewards are sub-Gaussian with known variance parameter σ_k^2.
    Invoked in the concentration inequality in Appendix A.2, Eq. (28).
  • ad hoc to paper α0√ln t + β0σ_k = 2σ_k√ln t for all t.
    This condition, used to simplify the confidence radius to Eq. (27), cannot hold with constant α0 and β0 because √ln t grows with t.
  • standard math Sample variance converges to true variance as N grows.
    Used in the proof to replace σ̂² with σ²; the Law of Large Numbers supports this in the limit, but finite-time behavior is not handled.
  • standard math The Basel sum ∑ 2/t² converges to π²/6.
    Used to bound the total probability of error events in the regret proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Theory to Practice with RAVEN-UCB: Addressing Non-Stationarity in Multi-Armed Bandits through Variance Adaptation." pith.science (2026). https://pith.science/paper/YTMIQ7F5

@misc{pith2026250602933,
  author       = {Pith},
  title        = {Pith review of: From Theory to Practice with RAVEN-UCB: Addressing Non-Stationarity in Multi-Armed Bandits through Variance Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTMIQ7F5}},
  note         = {Machine review of arXiv:2506.02933}
}
abstract

The Multi-Armed Bandit (MAB) problem is challenging in non-stationary environments where reward distributions evolve dynamically. We introduce RAVEN-UCB, a novel algorithm that combines theoretical rigor with practical efficiency via variance-aware adaptation. It achieves tighter regret bounds than UCB1 and UCB-V, with gap-dependent regret of order $K \sigma_{\max}^2 \log T / \Delta$ and gap-independent regret of order $\sqrt{K T \log T}$. RAVEN-UCB incorporates three innovations: (1) variance-driven exploration using $\sqrt{\hat{\sigma}_k^2 / (N_k + 1)}$ in confidence bounds, (2) adaptive control via $\alpha_t = \alpha_0 / \log(t + \epsilon)$, and (3) constant-time recursive updates for efficiency. Experiments across non-stationary patterns - distributional changes, periodic shifts, and temporary fluctuations - in synthetic and logistics scenarios demonstrate its superiority over state-of-the-art baselines, confirming theoretical and practical robustness.

Figures

Figures reproduced from arXiv: 2506.02933 by the authors.

Figure 1
Figure 1. Distributions of sample means and variances for a bandit arm [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Regret reduction in horizon T Both algorithms pull each suboptimal arm i on the order of log T times, but with different scaling factors dependent on reward variance: UCB1 selects suboptimal arms with a rate proportional to log T ∆2 i , our RAVEN-UCB reduces this rate to σ 2 i log T ∆2 i by incorporating variance exploration (See (32) in A.2). The difference in their pull counts is therefore:  log T ∆2 i − σ 2 i lo… view at source ↗
Figure 3
Figure 3. Cumulative Regret versus α0 for different β0 values In the Variance Drift scenario, optimal parameters transition from (α0 = 0.5, β0 = 0.5) with regret 6.55 at T = 1000 to (α0 = 1.0, β0 = 5.0) with regret 27.14 at T = 10000. The flat minimal curve for β0 = 0.5 at T = 1000 indicates conservative exploration in short-term high-variance settings, while the sharp regret decline for β0 = 5.0 at T = 10000 validates enhanc… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Cumulative regret, (b) umulative reward [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Boxplots of the results RAVEN-UCB performs well because it uses a variance-adaptive exploration strategy. This helps it quickly find and adapt to changes in warehouse efficiency. This is particularly valuable in logistics, where rapid adaptation to operational changes …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 48 canonical work pages

  1. [1]

    Modification of improved upper confidence bounds for regulating exploration in Monte-Carlo tree search

    Yun-Ching Liu and Yoshimasa Tsuruoka. “Modification of improved upper confidence bounds for regulating exploration in Monte-Carlo tree search”. Theoretical Computer Science 644 2016. pp. 92–105

    work page 2016

  2. [2]

    UCB revisited: Improved regret bounds for the stochas- tic multi-armed bandit problem

    Peter Auer and Ronald Ortner. “UCB revisited: Improved regret bounds for the stochas- tic multi-armed bandit problem”. Periodica Mathematica Hungarica 61 2010. pp. 55– 65

    work page 2010

  3. [3]

    Some aspects of the sequential design of experiments

    Herbert Robbins. “Some aspects of the sequential design of experiments”. Bulletin of the American Mathematical Society 58 1952. pp. 527–535

    work page 1952

  4. [4]

    A contextual-bandit approach to personalized news article recommen- dation

    Lihong Li et al. “A contextual-bandit approach to personalized news article recommen- dation”. 2010. pp. 661–670

    work page 2010

  5. [5]

    Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms

    Lihong Li et al. “Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms”. In: Proceedings of the fourth ACM international confer- ence on Web search and data mining . 2011, pp. 297–306

    work page 2011

  6. [6]

    Combinatorial multi-armed bandit: General framework and applications

    Wei Chen, Yajun Wang, and Yang Yuan. “Combinatorial multi-armed bandit: General framework and applications”. In: International conference on machine learning. PMLR. 2013, pp. 151–159

    work page 2013

  7. [7]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv´ ari. Bandit algorithms . Cambridge University Press, 2020

    work page 2020

  8. [8]

    Stochastic multi-armed-bandit problem with non-stationary rewards

    Omar Besbes, Yonatan Gur, and Assaf Zeevi. “Stochastic multi-armed-bandit problem with non-stationary rewards”. Advances in neural information processing systems 27 2014

    work page 2014

Show all 50 references
  1. [9]

    On upper-confidence bound policies for switching bandit problems

    Aur´ elien Garivier and Eric Moulines. “On upper-confidence bound policies for switching bandit problems”. In: International conference on algorithmic learning theory. Springer. 2011, pp. 174–188

    2011

  2. [10]

    Discounted ucb

    Levente Kocsis and Csaba Szepesv´ ari. “Discounted ucb”. In: 2nd PASCAL Challenges Workshop. Vol. 2. 2006, pp. 51–134

    2006

  3. [11]

    Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm

    Emanuele Cavenaghi et al. “Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm”. Entropy 23 2021. pp. 380

    2021

  4. [12]

    Exploration–exploitation tradeoff using variance estimates in multi-armed bandits

    Jean-Yves Audibert, R´ emi Munos, and Csaba Szepesv´ ari. “Exploration–exploitation tradeoff using variance estimates in multi-armed bandits”. Theoretical Computer Sci- ence 410 2009. pp. 1876–1902

    2009

  5. [13]

    Regret Analysis of Stochastic and Non- stochastic Multi-armed Bandit Problems

    S´ ebastien Bubeck and Nicol` o Cesa-Bianchi. Regret Analysis of Stochastic and Non- stochastic Multi-armed Bandit Problems . en. arXiv:1204.5721 [cs]. Nov. 2012. 22

    2012 arXiv

  6. [14]

    Finite-time analysis of the multi- armed bandit problem

    Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. “Finite-time analysis of the multi- armed bandit problem”. Machine learning 47 2002. pp. 235–256

    2002

  7. [15]

    Adaptive ε-greedy exploration in reinforcement learning based on value differences

    Michel Tokic. “Adaptive ε-greedy exploration in reinforcement learning based on value differences”. In: Annual conference on artificial intelligence . Springer. 2010, pp. 203– 210

    2010

  8. [16]

    Real-time self-adaptive Q-learning controller for energy man- agement of conventional autonomous vehicles

    Mojgan Fayyazi et al. “Real-time self-adaptive Q-learning controller for energy man- agement of conventional autonomous vehicles”. Expert Systems with Applications 222

  9. [17]

    A note on the advantage of context in Thompson sam- pling

    Michael Byrd and Ross Darrow. “A note on the advantage of context in Thompson sam- pling”. In: Artificial Intelligence and Machine Learning in the Travel Industry: Simplify- ing Complex Decision Making . Ed. by Ben Vinod. Cham: Springer Nature Switzerland, 2023, pp. 109–114

    2023

  10. [18]

    Efficient-ucbv: An almost optimal algorithm using vari- ance estimates

    Subhojyoti Mukherjee et al. “Efficient-ucbv: An almost optimal algorithm using vari- ance estimates”. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 32. 1. 2018

    2018

  11. [19]

    Applying Multi-armed Bandit Algorithms to Computational Advertising

    Kazem Jahanbakhsh. “Applying Multi-armed Bandit Algorithms to Computational Advertising”. CoRR abs/2011.10919 2020

    2011 arXiv

  12. [20]

    Recommendation System-based Upper Confidence Bound for Online Advertising

    Nhan Nguyen-Thanh et al. “Recommendation System-based Upper Confidence Bound for Online Advertising”. CoRR abs/1909.04190 2019

    1909 arXiv

  13. [21]

    Improvement of the Recommendation System Based on the Multi-Armed Bandit Algorithm

    Y. Li. “Improvement of the Recommendation System Based on the Multi-Armed Bandit Algorithm”. Applied and Computational Engineering 36 2024. pp. 237–241

    2024

  14. [22]

    Applying Multi-Armed Bandit algorithms for music recommendations at Spo- tify

    Y. Xia. “Applying Multi-Armed Bandit algorithms for music recommendations at Spo- tify”. Applied and Computational Engineering 68 2024. pp. 54–64

    2024

  15. [23]

    An efficient learning framework for multiproduct inventory systems with customer choices

    Xiangyu Gao and Huanan Zhang. “An efficient learning framework for multiproduct inventory systems with customer choices”. Production and Operations Management 31

  16. [24]

    Multi-Armed Bandit Approaches for Lo- cation Planning with Dynamic Relief Supplies Allocation Under Disaster Uncertainty

    Jun Liang, Zongjia Zhang, and Yanpeng Zhi. “Multi-Armed Bandit Approaches for Lo- cation Planning with Dynamic Relief Supplies Allocation Under Disaster Uncertainty”. Smart Cities 8 2024. pp. 5

    2024

  17. [25]

    A Multi-Armed Bandit Problem with the Optimal Arm Depending on a Hidden Markov Model

    Talha Cihad Gulcu. “A Multi-Armed Bandit Problem with the Optimal Arm Depending on a Hidden Markov Model”. In: 2021 IEEE Information Theory Workshop (ITW) . IEEE. 2021, pp. 1–6

    2021

  18. [26]

    Adaptive Algorithms for Multi-armed Bandit with Composite and Anonymous Feedback

    Siwei Wang, Haoyun Wang, and Longbo Huang. “Adaptive Algorithms for Multi-armed Bandit with Composite and Anonymous Feedback”. Proceedings of the AAAI Confer- ence on Artificial Intelligence 35 2021. pp. 10210–10217. 23

    2021

  19. [27]

    Classifier ensembles for detecting concept change in streaming data: Overview and perspectives

    Ludmila I Kuncheva. “Classifier ensembles for detecting concept change in streaming data: Overview and perspectives”. In: 2nd Workshop SUEMA . Vol. 2008. 2008, pp. 5–10

    2008

  20. [28]

    The impact of digital technology and Industry 4.0 on the ripple effect and supply chain risk analytics

    Dmitry Ivanov, Alexandre Dolgui, and Boris Sokolov. “The impact of digital technology and Industry 4.0 on the ripple effect and supply chain risk analytics”. International journal of production research 57 2019. pp. 829–846

    2019

  21. [29]

    Language models are few-shot learners

    Tom Brown et al. “Language models are few-shot learners”. Advances in neural infor- mation processing systems 33 2020. pp. 1877–1901

    2020

  22. [30]

    A comparison of Softmax proposals

    Konstantinos Velonis and Haridimos T. Vergos. “A comparison of Softmax proposals”. en. In: 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME). Maldives, Maldives: IEEE, Nov. 2022, pp. 1–6

    2022

  23. [31]

    Some formulas of variance of uncertain random variable

    Yuhong Sheng and Kai Yao. “Some formulas of variance of uncertain random variable”. Journal of Uncertainty Analysis and Applications 2 2014. pp. 1–10

    2014

  24. [32]

    An Upper Confidence Bound Approach to Estimating the Maximum Mean

    Zhang Kun, Liu Guangwu, and Shi Wen. An Upper Confidence Bound Approach to Estimating the Maximum Mean . en. arXiv:2408.04179 [math]. Aug. 2024

    2024 arXiv

  25. [33]

    Convergence rates in the law of large numbers

    Leonard E Baum and Melvin Katz. “Convergence rates in the law of large numbers”. Transactions of the American Mathematical Society 120 1965. pp. 108–123

    1965

  26. [34]

    The law of large numbers with a continuum of iid random variables

    Kenneth L Judd. “The law of large numbers with a continuum of iid random variables”. Journal of Economic theory 35 1985. pp. 19–25

    1985

  27. [35]

    A Survey on Practical Applications of Multi-Armed and Contextual Bandits

    Djallel Bouneffouf and Irina Rish. “A Survey on Practical Applications of Multi-Armed and Contextual Bandits”. CoRR abs/1904.10040 2019

    1904 arXiv

  28. [36]

    Optuna: A next-generation hyperparameter optimization frame- work

    Takuya Akiba et al. “Optuna: A next-generation hyperparameter optimization frame- work”. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining . 2019, pp. 2623–2631

    2019

  29. [37]

    Hyperband: A novel bandit-based approach to hyperparameter opti- mization

    Lisha Li et al. “Hyperband: A novel bandit-based approach to hyperparameter opti- mization”. Journal of Machine Learning Research 18 2017. pp. 1–52

    2017

  30. [38]

    Asymptotically optimal multi- armed bandit algorithm and hyperparameter optimization

    Haiguang Huang, Zhaowei Zhang, and Tongliang Zhang. “Asymptotically optimal multi- armed bandit algorithm and hyperparameter optimization”. Advances in Neural Infor- mation Processing Systems 33 2020. pp. 1458–1469

    2020

  31. [39]

    Non-stochastic best arm identification and hyperparameter op- timization

    Kevin Jamieson et al. “Non-stochastic best arm identification and hyperparameter op- timization”. 2016. pp. 240–248

    2016

  32. [40]

    Springer, 2015

    Logistics Management: Contributions of the Section Logistics of the German Academic Association for Business Research, 2015, Braunschweig, Germany . Springer, 2015. 24

    2015

  33. [41]

    Design and application of Internet of things-based warehouse management system for smart logistics

    C. K. M. Lee et al. “Design and application of Internet of things-based warehouse management system for smart logistics”.Computers & Industrial Engineering 123 2018. pp. 130–143

    2018

  34. [42]

    Models for Evaluating and Planning City Logistics Systems

    Teodor G. Crainic, Nicoletta Ricciardi, and Giovanni Storchi. “Models for Evaluating and Planning City Logistics Systems”. Transportation Science 43 2009. pp. 432–454

    2009

  35. [43]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . MIT Press, 2018

    2018

  36. [44]

    On the likelihood that one unknown probability exceeds an- other in view of the evidence of two samples

    William R. Thompson. “On the likelihood that one unknown probability exceeds an- other in view of the evidence of two samples”. Biometrika 25 1933. pp. 285–294

    1933

  37. [45]

    An empirical evaluation of Thompson sampling

    Olivier Chapelle and Lihong Li. “An empirical evaluation of Thompson sampling”. In: Advances in Neural Information Processing Systems . 2011, pp. 2249–2257

    2011

  38. [46]

    Improved algorithms for linear stochastic bandits

    Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. “Improved algorithms for linear stochastic bandits”. In: Advances in Neural Information Processing Systems . 2011, pp. 2312–2320

    2011

  39. [47]

    Introduction to Multi-Armed Bandits

    Aleksandrs Slivkins. Introduction to Multi-Armed Bandits . 2019

    2019

  40. [48]

    Concentration Inequali- ties

    St´ ephane Boucheron, G´ abor Lugosi, and Olivier Bousquet. “Concentration Inequali- ties”. In: Advanced Lectures on Machine Learning: ML Summer Schools 2003, Can- berra, Australia, February 2 - 14, 2003, T¨ ubingen, Germany, August 4 - 16, 2003, Revised Lectures. Ed. by Olivi...

    2003

  41. [49]

    The solution of the problem of integration in finite terms

    Robert H Risch. “The solution of the problem of integration in finite terms”. 1970

    1970

  42. [50]

    A short and elementary proof of the Basel problem

    Samuel G Moreno. “A short and elementary proof of the Basel problem”. The College Mathematics Journal 47 2016. pp. 134–135. 25

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.