Pith. sign in

REVIEW 4 major objections 5 minor 47 references

This paper claims that unobserved Markov states in bandits can be tracked implicitly by feeding LinUCB lagged rewards plus joint probe fingerprints, yielding lower dynamic regret than baselines when states are distinguishable and sticky.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:19 UTC pith:TPV3T6ST

load-bearing objection A plausibly useful heuristic for latent-state bandits, but the headline empirical result is confounded by a two-unit-per-round advantage for the randomized algorithms. the 4 major comments →

arxiv 2602.05139 v3 pith:TPV3T6ST submitted 2026-02-04 cs.LG

Adaptive Exploration for Latent-State Bandits

Jikai Jin , Kenneth Hung , Sanath Kumar Krishnamurthy , Baoyi Shi , Congshan Zhang This is my paper
classification cs.LG
keywords latent-state banditshidden Markov statecontextual banditLinUCBadaptive probingstate fingerprintdynamic regretnon-stationary bandit
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Bandits usually assume rewards depend only on observable context, but many applications hide a time-varying state that changes which arm is best. This paper proposes to ignore the state entirely and instead give a linear contextual bandit two signals: the previous action-reward pair, which reflects the recent state, and a 'fingerprint' formed by briefly playing multiple arms, which disambiguates states that look identical under any single arm. The authors build LinUCB variants that probe on a fixed schedule (RP-UCB and SP-UCB) or adaptively decide when to probe using residual, uncertainty, and staleness gates (AdaRP-UCB and AdaSP-UCB). They show, in a two-state analysis, that the probe interval should scale like the square root of probe cost over transition rate times reward gap. In synthetic stress tests the adaptive methods beat standard, adversarial, and non-stationary baselines whenever the fingerprints separate states and are refreshed quickly enough.

Core claim

Central claim: the optimal policy in a latent-state bandit can be learned without modeling the hidden state. Feed LinUCB a context of the previous action-reward pair plus a state fingerprint — the joint rewards from probing both arms. Different states can give the same single-arm reward but different joint rewards, so fingerprints separate states that lagged context alone cannot. The paper offers fixed-schedule probing (RP-UCB for two simultaneous units, SP-UCB for one) and adaptive variants gated by residual, UCB-margin, and staleness tests. Theorem 1 bounds average regret by Δprobe/τ + Δmax q τ/2 + Δmax εfp + O(1/T), minimized at τ* ≈ sqrt(Δprobe/(Δmax q)) — the clearest statement of the p

What carries the argument

The state fingerprint (the joint reward observation from playing both arms) is the central object; concatenated with the lagged action-reward pair it forms the LinUCB context. The fingerprint disambiguates states whose single-arm reward distributions overlap, and the lagged context keeps tracking current between probes. Adaptive probing is governed by three gates — residual (prediction error), uncertainty (small UCB margin), and staleness (age of last fingerprint) — each a test of whether a probe's information gain is worth its regret. Theorem 1 supplies the quantitative identity that sets probe frequency: the optimal interval is the square root of probe cost divided by the product of transi

Load-bearing premise

The approach assumes the hidden state is sticky enough and the arms' rewards clean enough that a brief probe or the previous round's action-reward pair carries usable information about the state at decision time; if the state flips mid-probe or two states look alike under every arm, the fingerprints mislead rather than help.

What would settle it

Take a two-state, two-arm bandit where both states have identical joint reward fingerprints (e.g., both produce (0.5, 0.5)) but different optimal arms. If AdaRP-UCB still achieves low dynamic regret in this configuration, the paper's identifiability story is wrong; if it collapses toward the single-best-arm baseline, the fingerprint is confirmed as the mechanism doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If fingerprints separate states and are refreshed often enough, AdaRP-UCB achieves lower dynamic regret than UCB1, Thompson sampling, EXP3, sliding-window UCB, and discounted UCB across the tested parameter sweeps (S=2–50, p_stay=0.5–0.99, σ=0.01–0.5, T=500–20000).
  • Adaptive probing beats fixed-schedule probing in volatile environments, because it spends probes when residuals spike, margins shrink, or fingerprints age, and holds off otherwise.
  • The probe interval is a real design parameter with a principled default: τ* ≈ sqrt(Δprobe/(Δmax q)), so probe more often as states turn over faster and less often as probes get costlier.
  • The approach inherits LinUCB's computational cost and needs no state-space model, transition estimates, or belief states, making it deployable where POMDP solvers would be too heavy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Section 6 comparison gives RP-UCB/AdaRP-UCB two action-reward samples per round while single-unit baselines get one; if the intended regret metric is per decision unit, the randomized algorithms are being compared on more data per round, so their edge could shrink under a matched sample budget. (Editorial inference, not paper's claim.)
  • The fingerprint idea suggests a general recipe for hidden-state problems: occasionally play 'sensor' arms that reveal cross-arm payoff structure. The same trick might transfer to recommendation or clinical-trial settings where units can be randomized to different treatments.
  • The named failure modes — weak fingerprint separation, high noise, state changes during sequential probes — are testable diagnostics: given a candidate reward matrix, one could compute expected fingerprint separation before deciding whether probing will pay.
  • Theorem 1 is proved only for S=2 with unique optimal arms; the paper's experiments suggest the mechanism still helps for larger S, but the formal tradeoff is unproven there.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies a latent-state bandit model in which an unobserved Markov state evolves independently of the learner's actions and determines the reward of each arm. The authors propose a family of model-free algorithms built on LinUCB: LC-UCB uses the lagged action-reward pair as context, while RP-UCB/SP-UCB and their adaptive variants AdaRP-UCB/AdaSP-UCB supplement this with a 'fingerprint' obtained by probing both arms either simultaneously (two experimental units) or sequentially. The adaptive variants replace the fixed probing schedule with residual, uncertainty, and staleness gates. The paper's main evidence is an empirical study (Table 3) claiming that the proposed algorithms reduce dynamic regret relative to classical, adversarial, and non-stationary baselines in synthetic stress tests. A simple heuristic bound for an idealized periodic-probing policy is given in Theorem 1, and practical guidance for algorithm selection is provided.

Significance. If the empirical claims were supported by a fair comparison, the paper would offer a practical, computationally lightweight approach to a genuinely hard problem: latent-state bandits with autonomously evolving states and no state observability. The idea of using lagged rewards plus coordinated probing as contextual features is intuitive and could be useful in applications such as advertising and clinical trials. The paper also provides a clear taxonomy of related settings. However, the central empirical claim currently rests on a regret-accounting comparison that is not commensurate across algorithm families, and several promised experiments (ablations, misspecification tests) are absent. The theoretical contribution is limited to an idealized bound and does not by itself support the proposed algorithms. With a corrected evaluation and appropriate caveats, the core ideas could still be a useful contribution.

major comments (4)
  1. [Abstract / Section 6] The regret comparison in Table 3 is not fair across algorithm families. Algorithm 2 (RP-UCB) plays two actions every round (lines 18-19) and observes two rewards; in exploit mode it updates LinUCB with two independent samples of the same context, and in probe mode it charges regret only for the better of the two arms (lines 22-25). The dynamic regret defined in Section 2 is for a single decision unit per round, so a two-unit algorithm that receives two samples per round while being charged as one unit has a hidden data-rate advantage. Doubling the noise variance for the randomized algorithms does not equalize the number of samples per round; two averaged noisy rewards still carry more information than one reward. Since the abstract's superiority claim is based on Table 3, the paper must either compare methods at equal per-unit sample counts, report regret summed over all units played, or
  2. [Section 5.1 / Algorithm 4] The abstract promises 'Ablations and misspecification tests identify the main failure modes: weak fingerprint separation, high noise, and state changes during sequential probes.' The full text contains parameter sweeps (Section 6) but no dedicated ablation study or misspecification experiments. The claimed failure modes are only qualitatively discussed from the parameter sweeps. Either add the stated experiments or revise the abstract so it does not promise evidence that is not in the paper.
  3. [Table 3 / Section 6] The adaptive gates are not fully specified and their sensitivity is not analyzed. Section 5.1 gives z_thresh≈2, λ_h=0.1, δ_h=0.5, but no numerical default is given for m_thresh, and τ_min is not specified in the text. Since AdaRP-UCB's advantage over RP-UCB depends on these hand-chosen thresholds, the paper should report the exact hyperparameter settings used for every configuration in Table 3 and provide a sensitivity analysis (or at least a statement of robustness over a plausible range). Without this, the empirical results are not reproducible and the possibility that the thresholds were tuned to the test configurations cannot be ruled out.
  4. [Table 3] Table 3 reports mean cumulative regrets with no measure of variability. Section 6 states that 128 random instances × 5 independent runs were used, but no standard errors, confidence intervals, or significance tests are reported. This is particularly problematic for close comparisons (e.g., the σ=0.05 row) and for the claim of 'consistently outperform.' The paper should include error bars or a statistical comparison to substantiate the claimed dominance.
minor comments (5)
  1. [Section 4.1] The sentence 'We implemented for a two arms, but believe that it can be easily generalized to multiple arms' is grammatically incorrect and vague. If the algorithms are only implemented for K=2, this should be stated clearly and the generalization claim should be either demonstrated or removed.
  2. [Table 5 / Table 3] The baseline 'Optimal Single Arm' is described as always selecting the arm with highest expected reward under a uniform stationary distribution. This is not generally the optimal stationary policy for a non-uniform ergodic chain, so the label 'Oracle' is misleading. Consider renaming it to 'Best Stationary Arm (uniform)' or computing the true optimal stationary policy.
  3. [Section 3.1, Eq. (1)] The derivation of the conditional expectation would benefit from more careful notation. The approximation E[r_t | a_{t-1}, r_{t-1}] ≈ μ_{s,a} hides the dependence on the algorithm's state-visit distribution π*_{LC-UCB}. A reader may confuse the true environment distribution with the algorithm-induced distribution.
  4. [Figures 2 and 3] The figures have no descriptive captions. Figure 2 shows 'head-to-head winning rates' but the panels are not labeled in the caption; Figure 3's panels are labeled only in the figure. Add a legend and panel descriptions to both.
  5. [Appendix C] The proof of Theorem 1 is clear but relies on an idealized probing policy; it would help to state explicitly that this is not a regret bound for RP-UCB or AdaRP-UCB, to avoid overinterpretation.

Circularity Check

0 steps flagged

No significant circularity; the derivation chain is self-contained, though the empirical comparison has a fairness concern that is a correctness risk rather than circularity.

full rationale

The paper's only formal derivation is Theorem 1, which is a self-contained upper bound for an idealized periodic probing policy. It takes per-probe regret, probe error probability, and state transition probability as assumptions and derives a regret bound via a union bound; the bound's inputs are not the claimed conclusion, and the minimization over probe interval is a genuine tradeoff. The algorithms themselves are LinUCB variants with stated feature constructions; none of the feature constructions or UCB updates reduce to the regret objective by definition. The abstract's claim is an empirical evaluation outcome, not a fitted parameter renamed as a prediction. The adaptive gate thresholds (z_thresh ≈ 2, lambda_h = 0.1, delta_h = 0.5) are presented as design hyperparameters, and the text gives no indication they were fitted to the test configurations; any assertion that they were is speculation, which the rules exclude. There are no load-bearing self-citations and no imported uniqueness theorems. The one in-scope concern is Section 6's accounting: Algorithm 2 plays two actions per round but reports a single per-round regret (lines 22-29), and the paper states 'We double the noise variance for randomized algorithms to compensate for their inherent randomization effects and ensure fair comparison.' That does not equalize the data rate between two-sample and one-sample algorithms, so the headline superiority in Table 3 is not fully substantiated. This is a validity/fairness issue for the empirical claim, but it is not a circular reduction of a derivation to its inputs, so the circularity score remains 0.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on the hidden Markovian state model, the additive reward noise assumption, and the assumption that probing can produce fingerprints that disambiguate states. These are domain assumptions, not free parameters, but the algorithm's many hyperparameters (α, λ, τ, gate thresholds) are effectively free parameters whose values are not reported or analyzed. The absence of a sensitivity analysis means we cannot distinguish a robust algorithmic improvement from tuning artifacts.

free parameters (9)
  • alpha (LinUCB exploration parameter)
    Controls the UCB exploration bonus; used in all algorithms but its value is not reported. Different choices could change results.
  • lambda (regularization)
    Initializes precision matrices A_a; not reported. Sensitivity not tested.
  • tau (probing interval)
    For fixed-schedule RP-UCB/SP-UCB; not reported in experiments. It trades probe cost against stale fingerprint regret.
  • z_thresh (residual gate) = ≈2
    Set to approximately 2, corresponding to a 95% confidence interval; not tuned but not justified against other values.
  • m_thresh (uncertainty gate)
    Not specified; decides when UCB margin is too small.
  • lambda_h (hazard rate) = 0.1
    Used in exponential hazard; example in text, no sensitivity analysis.
  • delta_h (hazard threshold) = 0.5
    Activates staleness gate after ~7 rounds; no sensitivity analysis.
  • tau_min (minimum inter-probe gap)
    Prevents excessive probing; value not reported.
  • sigma_0 (noise floor in residual gate)
    Serves as floor to prevent residual gate over-triggering; value not specified.
axioms (5)
  • domain assumption The hidden state evolves as an ergodic Markov chain with transition matrix P and stationary distribution π*.
    Defines the latent-state bandit setting; used in Section 2 and throughout.
  • domain assumption The action does not affect state transitions; state evolves autonomously.
    Stated in Figure 1 and Section 1; distinguishes from restless bandits.
  • domain assumption Reward is r_t = mu_{s_t,a_t} + eta_t with mean-zero noise of variance σ^2.
    Reward model in Section 2.
  • domain assumption For sequential probing, the state is sufficiently 'sticky' so two consecutive rewards approximate joint observations.
    Section 4.1: 'This approach relies on slow state transitions.'
  • domain assumption The learner observes only past actions and rewards, never the state.
    Core problem definition; also the DAG in Figure 1.

pith-pipeline@v1.3.0-alltime-deepseek · 15666 in / 14533 out tokens · 129130 ms · 2026-08-03T04:19:06.947613+00:00 · methodology

0 comments
read the original abstract

We study bandits whose rewards depend on an unobserved Markov state that evolves independently of the learner's actions. The optimal arm can change even though the learner observes only past actions and rewards. We propose algorithms that feed LinUCB with two summaries of the hidden state: a lagged action-reward pair and, when available, a probe fingerprint formed from rewards of multiple arms. The adaptive variants refresh the fingerprint using residual, margin, and staleness tests. In synthetic stress tests over state count, transition rate, noise, and horizon, these methods reduce dynamic regret relative to standard, adversarial, and non-stationary bandit baselines when the summaries distinguish states and are updated often enough. Ablations and misspecification tests identify the main failure modes: weak fingerprint separation, high noise, and state changes during sequential probes.

Figures

Figures reproduced from arXiv: 2602.05139 by Baoyi Shi, Congshan Zhang, Jikai Jin, Kenneth Hung, Sanath Kumar Krishnamurthy.

Figure 1
Figure 1. Figure 1: A directed acyclic graph (DAG) representing the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Frequency of pulling the optimal arm across time. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Head-to-head winning rates under different [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 1 canonical work pages

  1. [1]

    Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009. Explore/exploit schemes for web content optimization. In2009 Ninth IEEE International Conference on Data Mining. IEEE, 1–10

    2009

  2. [2]

    Dilip Arumugam and Benjamin Van Roy. 2022. Deciding what to model: Value- equivalent sampling for reinforcement learning.Advances in neural information processing systems35 (2022), 9024–9044

    2022

  3. [3]

    Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem.Machine learning47, 2 (2002), 235–256

    2002

  4. [4]

    Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 2002. The nonstochastic multiarmed bandit problem.SIAM journal on computing32, 1 (2002), 48–77

    2002

  5. [5]

    Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar

  6. [6]

    Elias Bareinboim, Andrew Forney, and Judea Pearl. 2015. Bandits with unobserved confounders: a causal approach. InProceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1(Montreal, Canada)(NIPS’15). MIT Press, Cambridge, MA, USA, 1342–1350

    2015

  7. [7]

    Donald A Berry. 2012. Adaptive clinical trials in oncology.Nature reviews Clinical oncology9, 4 (2012), 199–207

    2012

  8. [8]

    Omar Besbes, Yonatan Gur, and Assaf Zeevi. 2019. Optimal exploration– exploitation in a multi-armed bandit problem with non-stationary rewards.Sto- chastic Systems9, 4 (2019), 319–337

    2019

  9. [9]

    Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire

  10. [10]

    Yang Cao, Zheng Wen, Branislav Kveton, and Yao Xie. 2019. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 418–427

    2019

  11. [11]

    Wei Chen, Yajun Wang, and Yang Yuan. 2013. Combinatorial Multi-Armed Bandit: General Framework and Applications. InProceedings of the 30th International Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 28), Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, Georgia, USA, 151–159. Issue 1. https://proceedings.mlr.pr...

    2013

  12. [12]

    Maria Dimakopoulou, Zhengyuan Zhou, Susan Athey, and Guido Imbens. 2019. Balanced linear contextual bandits. InProceedings of the Thirty-Third AAAI Con- ference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence(Honolulu, Hawa...

    work page doi:10.1609/aaai.v33i01.33013445 2019

  13. [13]

    Mitsis, and Joelle Pineau

    Audrey Durand, Charis Achilleos, Demetris Iacovides, Katerina Strati, Georgios D. Mitsis, and Joelle Pineau. 2018. Contextual Bandits for Adapting Treatment in a Mouse Model of de Novo Carcinogenesis. InProceedings of the 3rd Machine Learning for Healthcare Conference (Proceedings of Machine Learning Research, Vol. 85), Finale Doshi-Velez, Jim Fackler, Ke...

    2018

  14. [14]

    Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. 2022. Provable reinforcement learning with a short-term memory. InInternational Conference on Machine Learning. PMLR, 5832–5850

    2022

  15. [15]

    Andrew Forney, Judea Pearl, and Elias Bareinboim. 2017. Counterfactual data- fusion for online reinforcement learners. InInternational conference on machine learning. PMLR, 1156–1164

    2017

  16. [16]

    Aurélien Garivier and Eric Moulines. 2011. On upper-confidence bound policies for switching bandit problems. InInternational conference on algorithmic learning theory. Springer, 174–188

    2011

  17. [17]

    Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015. Contextual Markov Decision Processes. arXiv:1502.02259 [stat.ML] https://arxiv.org/abs/1502.02259

    Pith/arXiv arXiv 2015

  18. [18]

    Botao Hao and Tor Lattimore. 2022. Regret bounds for information-directed reinforcement learning.Advances in neural information processing systems35 (2022), 28575–28587

    2022

  19. [19]

    David P Helmbold, Robert E Schapire, Yoram Singer, and Manfred K Warmuth

  20. [20]

    Peter Henderson, Ben Chugg, Brandon Anderson, and Daniel E. Ho. 2022. Be- yond Ads: Sequential Decision-Making Algorithms in Law and Public Policy. In Proceedings of the 2022 Symposium on Computer Science and Law(Washington DC, USA)(CSLA W ’22). Association for Computing Machinery, New York, NY, USA, 87–100. doi:10.1145/3511265.3550439

    arXiv 2022

  21. [21]

    Chi Jin, Sham Kakade, Akshay Krishnamurthy, and Qinghua Liu. 2020. Sample- efficient reinforcement learning of undercomplete pomdps.Advances in Neural Information Processing Systems33 (2020), 18530–18539

    2020

  22. [22]

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Plan- ning and acting in partially observable stochastic domains.Artificial intelligence 101, 1-2 (1998), 99–134

    1998

  23. [23]

    Wonyoung Kim, Gi-Soo Kim, and Myunghee Cho Paik. 2021. Doubly robust thompson sampling with linear payoffs. InProceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 1211, 11 pages

    2021

  24. [24]

    Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide.Data mining and knowledge discovery18, 1 (2009), 140–181

    2009

  25. [25]

    Akshay Krishnamurthy, Zhiwei Steven Wu, and Vasilis Syrgkanis. 2018. Semi- parametric Contextual Bandits. InProceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 2776–2785. https://proceedings.mlr.press/ v80/krishnamurthy18a.html

    2018

  26. [27]

    Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor

  27. [28]

    Jeongyeol Kwon, Shie Mannor, Constantine Caramanis, and Yonathan Efroni

  28. [29]

    Tze Leung Lai, Philip W Lavori, and Olivia Yueh-Wen Liao. 2014. Adaptive choice of patient subgroup for comparing two treatments.Contemporary clinical trials 39, 2 (2014), 191–200

    2014

  29. [30]

    Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual- bandit approach to personalized news article recommendation. InProceedings of the 19th international conference on World wide web. 661–670

    2010

  30. [31]

    Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang, Dingli Ma, Mladen Kolar, and Zhaoran Wang. 2024. Instrumental variable value iteration for causal offline reinforcement learning.Journal of Machine Learning Research25, 303 (2024), 1–56

    2024

  31. [32]

    Keqin Liu and Qing Zhao. 2010. Indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access.IEEE Transactions on Information Theory56, 11 (2010), 5547–5567

    2010

  32. [33]

    Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. 2018. Markov decision processes with continuous side information. InAlgorithmic learning theory. PMLR, 597–618

    2018

  33. [34]

    Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. 2006. Anytime point-based approximations for large POMDPs.Journal of Artificial Intelligence Research27 (2006), 335–380

    2006

  34. [35]

    Herbert Robbins. 1952. Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc.58, 5 (1952), 527 – 535

    1952

  35. [36]

    Donald B Rubin. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of educational Psychology66, 5 (1974), 688

    1974

  36. [37]

    Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. 2018. A tutorial on thompson sampling.Foundations and Trends®in Machine Learning11, 1 (2018), 1–96

    2018

  37. [38]

    Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. 2013. Automatic ad format selection via contextual bandits. InProceedings of the 22nd ACM inter- national conference on Information & Knowledge Management. 1587–1594

    2013

  38. [39]

    William R Thompson. 1933. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.Biometrika25, 3/4 (1933), 285–294

    1933

  39. [40]

    Sofía S Villar, Jack Bowden, and James Wason. 2015. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges.Statistical science: a review journal of the Institute of Mathematical Statistics30, 2 (2015), 199

    2015

  40. [41]

    Peter Whittle. 1988. Restless bandits: Activity allocation in a changing world. Journal of applied probability25, A (1988), 287–298

    1988

  41. [42]

    Latent-State

    Annie Xie, Logan Mondal Bhamidipaty, Evan Zheran Liu, Joey Hong, Sergey Levine, and Chelsea Finn. 2024. Learning to Explore in POMDPs with In- formational Rewards. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Rus- lan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller...

    2024

  42. [1998]

    On-line portfolio selection using multiplicative updates.Mathematical Finance8, 4 (1998), 325–347

    1998

  43. [2011]

    In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics

    Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 19–26

  44. [2016]

    InConference on Learning Theory

    Reinforcement learning of pomdps using spectral methods. InConference on Learning Theory. PMLR, 193–256

  45. [2021]

    Reinforcement learning in reward-mixing mdps.Advances in Neural Information Processing Systems34 (2021), 2253–2264

    2021

  46. [2023]

    InInternational Conference on Machine Learning

    Reward-mixing mdps with few latent contexts are learnable. InInternational Conference on Machine Learning. PMLR, 18057–18082

  47. [2024]

    Advances in Neural Information Processing Systems37 (2024), 82726–82756

    Rl in latent mdps is tractable: Online guarantees via off-policy evaluation. Advances in Neural Information Processing Systems37 (2024), 82726–82756

    2024