Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Balancing optimism and pessimism in offline-to-online learning

T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One rule matches the best of LCB and UCB in offline-to-online bandits.

desk verdict The OTO algorithm and logging-regret analysis are new and worth reading, but Theorem 1's pointwise anytime guarantee is false as stated. read the letter →

arxiv 2502.08259 v2 pith:SUDKTVIP submitted 2025-02-12 cs.LG cs.AI

Flore Sentenac , Ilbin Lee , Csaba Szepesvari This is my paper · ORCID
classification cs.LGcs.AI MSC 68Q3262L05
keywords offline-to-onlinelearningmulti-armedbanditspessimismandoptimismlowerconfidenceboundupperregretagainstloggingpolicyexplorationbudgetminimax
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what a learner should do when it inherits offline data and then must interact with the same environment for a known or unknown number of rounds. Short horizons favor the pessimistic Lower Confidence Bound (LCB) algorithm, which stays close to the data-generating policy; long horizons favor the optimistic Upper Confidence Bound (UCB) algorithm, which finds the best arm efficiently. The paper introduces OTO, which plays LCB unless a carefully defined exploration budget proves it can afford UCB's exploration. Its main theorem bounds OTO's regret both against the best arm and against the logging policy, showing that OTO is never much worse than the better of LCB and UCB on either measure, up to logarithmic factors and a tunable budget term. The paper also gives a minimax lower bound for the whole offline-to-online spectrum and shows the same budget idea works when the horizon is unknown via a horizon-doubling proxy.

What carries the argument

The engine is an exploration budget $B_{\tilde T}(t)$ that accumulates a safe lower bound on how much reward the algorithm has earned and can still earn relative to a benchmark $\gamma = \mu_{L(0)}(0) - \alpha\beta$, where $L(0)$ is the LCB arm at time zero and $\alpha$ tunes how strict the budget is. Each play of LCB adds $\alpha\beta$ to the budget, while playing UCB charges the budget the gap between UCB's lower confidence value and $\gamma$; at every round the algorithm plays UCB only if $B_{\tilde T}(t) > 0$, otherwise it falls back to LCB. The budget is constructed so that the related pseudo-budget stays positive by induction, and this positivity is what converts LCB plays into a guarantee on regret against the logging policy. The bound on regret against optimality then splits into a UCB-style pull-count bound for arms chosen when the budget is high and an LCB-style bound for rounds when the budget is low.

What would settle it

Construct a two-arm instance with $m_1 = m$, $m_2 = 0$, $\mu_1 = 0.5$, $\mu_2 = 1$, run OTO with known horizon $T$ and $\alpha = 1$, and compare its logging-policy regret to the bound $t(1+\alpha)\beta$ from Eq. (2). If the empirical regret exceeds that bound by a non-logarithmic factor with high probability, the budget induction in Section 7 is wrong; if OTO ever exceeds both the LCB and UCB regret curves by more than the additive $K$ term, Theorem 1 is violated.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 1: for any bandit instance, with probability at least $1 - 2T\delta$, OTO satisfies $R^{\mathrm{log}}_{OTO}(t) \le t(1 + (1 + \mathbf{1}_{T \text{ unknown}})\alpha)\beta$ and $R_{OTO}(t) \le \sum_i \Delta_i (4\log(K/\delta)/\Delta_i^2 - m_i)_+ + 12K\log(K/\delta)/(\alpha\beta) + K$, where $\Delta_i$ is the suboptimality gap, $m_i$ the offline sample count for arm $i$, and $\beta = (\sum_i \sqrt{m_i}/m)\sqrt{2\log(K/\delta)}$ measures the uncertainty in the logging policy's value. Together with a matching minimax lower bound, this says OTO automatically finds the problem-dependent inflection point where optimism overtakes pessimism, keeping the short-horizon safety of LCB without giving up the long-horizon efficiency of UCB. The guarantee holds for both regret against the optimal arm and regret against the logging policy, and it extends to unknown horizons with an extra factor and a time-dependent confidence level.

Load-bearing premise

The whole proof treats the number of offline samples per arm as fixed, exactly known numbers, and it identifies the logging policy with those counts; if the counts are random, the budget and the bounds do not directly apply, and the paper only conjectures that they would extend with little change.

Editorial extensions

If this is right

  • OTO performs nearly as well as the better of LCB and UCB at any time horizon, for both regret against the optimal arm and regret against the logging policy.
  • The algorithm needs no prior knowledge of whether deployment is short or long; the budget determines the switch automatically.
  • When $\alpha = 0$, OTO reduces to LCB, and larger $\alpha$ makes it behave more like UCB, so the trade-off is controlled by one parameter.
  • In the unknown-horizon case, a horizon-doubling proxy preserves the guarantees up to a factor involving $\alpha$ and an additive cost, with the confidence parameter decaying as $1/t^2$.
  • The minimax lower bound shows that offline-to-online difficulty depends on the composition of offline data, with the minimum per-arm offline count limiting small-horizon performance and the usual $\sqrt{KT}$ term emerging for large horizons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The budget principle is more portable than the specific formula in Algorithm 1; alternative budget definitions with similar guarantees are discussed in the appendix, so the portable insight is bounding how far cumulative reward can fall below a safe benchmark.
  • If the conjecture that results extend to random offline sample counts holds, the practical gain is substantial: practitioners could drop the fixed-dataset assumption and still use the same guarantees.
  • A testable extension is to instantiate the budget with confidence intervals for contextual bandits or simple reinforcement-learning value functions; the paper anticipates this transfer but does not prove it.
  • We read the parameter $\alpha$ as an application-level trade-off knob: small values favor the logging-policy guarantee and large values favor the optimality guarantee, so choosing it well is part of deploying the method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies offline-to-online learning in stochastic multi-armed bandits, where a learner has fixed offline sample counts per arm and then interacts online for a horizon T. It introduces an algorithm, OTO, that switches between a Lower Confidence Bound (LCB) arm and an Upper Confidence Bound (UCB) arm according to a budget that accumulates when LCB is played. The central claimed result, Theorem 1, is that for every time 1 ≤ t ≤ T, OTO achieves, with high probability, a bound on regret against the logging policy comparable to LCB's bound and bounds on regret against the optimal arm comparable to UCB's bounds. The paper also analyzes the regret of UCB and LCB in this setting, and reports synthetic and real-data experiments. The overall goal is to show that OTO automatically interpolates between pessimism and optimism across the offline-to-online spectrum.

Significance. The problem is well motivated and the paper contains useful components: a careful comparison of UCB and LCB under both the logging-policy regret and the optimality regret, a detailed appendix with proofs, and reproducible experimental code. If the anytime guarantee of Theorem 1 were correct, the budget-based interpolation idea would be a valuable contribution to the offline-to-online bandit literature. However, the headline anytime bound on logging regret is false as stated, and the failure is not merely a missing constant in the proof but a consequence of the budget anticipating future LCB plays. This undermines the abstract's central claim, although the final-time bounds and the UCB/LCB analysis may still be salvageable with a modified budget or a weakened theorem.

major comments (2)
  1. [Section 7, Theorem 1, Eq. (2)] The claimed pointwise bound Rlog_OTO(t) ≤ t(1 + (1 + 1_{T unknown})α)β for every 1 ≤ t ≤ T is false. The proof in Section 7 only establishes the bound at the final time T: positivity of the pseudo-budget at time T gives (T − T)αβ = 0, which removes the horizon-dependent term. Repeating the same argument at an intermediate t yields Rlog_OTO(t) ≤ t(1 + α)β + (T − t)αβ for the known-horizon case, with an extra (T − t)αβ that does not vanish for t < T. This is not a harmless proof gap: the term (T − t)αβ is exactly what makes the budget positive at early rounds, so the algorithm is allowed to take an early action that violates Eq. (2). A concrete instance is K = 2, m1 = 10^4, m2 = 1, μ1 = 0.5, μ2 = 0, T = 2000, α = 1, δ = 1/(2T^2). For realized offline means such as μhat_1 ≈ 0.5 and μhat_2 ∈ [−2, 2], which occur with probability close to 0.95 under N(0.5,1) and N(0,1) rewards, the concentration inequalities of Lemma 1 hold, yet at t = 1 the budget is positive, OTO pulls arm 2, and Rlog(1) = μ0 ≈ 0.5, while the right-hand side of Eq. (2) is (1 + α)β ≈ 0.116. Thus the theorem's claimed high-probability anytime bound fails on a substantial-probability event, contradicting the stated 1 − 2Tδ guarantee.
  2. [Section 4, Eq. (1) and Algorithm 1] The root cause of the pointwise failure is that the budget B_T(t) contains the term (T − t)αβ, which credits the algorithm for LCB plays that have not yet happened but are anticipated by the end of the horizon. This is a non-anticipating violation in the sense that early exploration is justified by future forced LCB plays, so the logging-policy regret at early times can be much larger than the anytime bound claims. A repair should either replace the future term with a term depending only on past LCB plays (e.g., removing (T − t)αβ or using a doubling schedule over the elapsed time), or the main theorem must be weakened to a final-horizon statement. The current abstract and the surrounding text promise an 'at any point in time' guarantee, so this is a load-bearing issue that needs to be resolved before the paper can be accepted.
minor comments (4)
  1. [Section 3 and Eq. (1)] The notation for the upper and lower confidence bounds is not consistently rendered: the budget formula and the proof mix ar{\mu}_i(t) and \underline{\mu}_i(t), and the text in Section 4 describes the second term in Eq. (1) as 'a high probability lower bound' although the displayed symbol is ambiguous. Please define both bounds explicitly in Eq. (1) and in Algorithm 1, since the proof's validity depends on which bound appears in the budget.
  2. [Section 5, Theorem 3 and Propositions 1 and 4] Several statements use probability 1 − 2T^2δ or an additive 2T^2δ, while Lemma 1 and the union bound in the proofs give 2Tδ. For example, Theorem 3 states probability at least 1 − 2T^2δ, but the proof applies Lemma 1 and a union bound over the two inequalities, yielding 2Tδ. Please align the statement with the proof or explain the additional factor.
  3. [Section 4, Remark 2] The time-varying δt extension for unknown horizons is presented as a remark without a complete proof. Since Theorem 1 explicitly includes the unknown-horizon case, please either provide a formal proof for δt = δ0/t^2 or clearly label that part as a conjecture or a sketch.
  4. [Section 6.2, Figures 5 and 6] The captions for Figures 5 and 6 include the phrase 'Mean at 78241' (and 'Mean at 78230'), which appears to be a leftover artifact from the experimental log and is not explained in the text. Please remove or explain these values.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: Theorem 1 is derived from the algorithm's definition and concentration inequalities; self-citations are motivational, not load-bearing.

full rationale

The paper's central result, Theorem 1, is derived in Section 7 from the definition of OTO and standard concentration arguments. The proof bounds UCB pulls by the usual confidence-bound argument, bounds LCB pulls via positivity of the pseudo-budget, and then rearranges algebraic inequalities. No parameter is fitted to the target regret: alpha and delta are user-set, beta is a deterministic function of offline counts and delta, and gamma is subsequently lower-bounded by mu0 - (1+alpha)beta using concentration, not by assuming the desired regret bound. The minimax lower bounds and the LCB/UCB comparisons are proved independently and are not used as inputs to Theorem 1. Self-citations (e.g., Wu et al. 2016 for conservative bandits, Xiao et al. 2021 for LCB optimality) motivate the algorithm design and evaluation metrics but are not load-bearing: the needed UCB and LCB bounds are re-proved in the paper. The reviewer's concern that Theorem 1's 'at any point in time' statement is only proved at the final horizon is a correctness/quantifier gap, not circularity, because the proof does not assume the conclusion it is trying to establish. Overall the derivation is self-contained, so the circularity score is low.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a standard stochastic bandit model with fixed offline counts, concentration inequalities, and a hand-designed budget mechanism. No new physical or mathematical entities are postulated; the only hand-chosen inputs are the algorithm parameters α and δ.

free parameters (2)
  • α = User-set. Synthetics use α=0.2 (known horizon) and α=0.6 (unknown); CTR uses α=0, 0.3, 1, 5.
    Controls budget stringency and the trade-off between logging regret and optimality regret. The theorems hold for any α, so it is not fitted to the target result.
  • δ = 1/T^2 (known horizon), δ_t=0.01/t^2 (unknown horizon).
    Confidence level in the concentration inequalities. It is a standard user-chosen parameter, not fitted to the target result.
assumptions (5)
  • domain assumption Reward distributions Pi are 1-subgaussian with means in [0,1].
    Used throughout to construct confidence intervals via Hoeffding's inequality (Section 3, Lemma 1).
  • domain assumption Offline sample sizes m_i are fixed, known, non-random; the logging policy is π_i = m_i/m.
    Section 3 states this explicitly and only conjectures extension to random counts; the definitions of β and γ and the proof of Eq. (7) rely on fixed m_i.
  • standard math Hoeffding's inequality and the union bound hold for the constructed upper and lower confidence sequences.
    Lemma 1 (Section 3) uses these to justify the high-probability event that all confidence intervals contain the true means for all t ≤ T.
  • standard math Bretagnolle-Huber inequality and data processing inequality give the lower bound in Theorem 2.
    Appendix C.1 uses these standard information-theoretic tools.
  • ad hoc to paper The specific budget formula B_T(t) is a hand-designed mechanism; the proof depends on its exact structure.
    Remark 1 acknowledges the formula may seem arbitrary and that other budgets could work. The induction in Section 7 relies on the pseudo-budget staying non-negative, which is specific to this formula.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Balancing optimism and pessimism in offline-to-online learning." pith.science (2026). https://pith.science/paper/SUDKTVIP

@misc{pith2026250208259,
  author       = {Pith},
  title        = {Pith review of: Balancing optimism and pessimism in offline-to-online learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SUDKTVIP}},
  note         = {Machine review of arXiv:2502.08259}
}
read the original abstract

We consider what we call the offline-to-online learning setting, focusing on stochastic finite-armed bandit problems. In offline-to-online learning, a learner starts with offline data collected from interactions with an unknown environment in a way that is not under the learner's control. Given this data, the learner begins interacting with the environment, gradually improving its initial strategy as it collects more data to maximize its total reward. The learner in this setting faces a fundamental dilemma: if the policy is deployed for only a short period, a suitable strategy (in a number of senses) is the Lower Confidence Bound (LCB) algorithm, which is based on pessimism. LCB can effectively compete with any policy that is sufficiently "covered" by the offline data. However, for longer time horizons, a preferred strategy is the Upper Confidence Bound (UCB) algorithm, which is based on optimism. Over time, UCB converges to the performance of the optimal policy at a rate that is nearly the best possible among all online algorithms. In offline-to-online learning, however, UCB initially explores excessively, leading to worse short-term performance compared to LCB. This suggests that a learner not in control of how long its policy will be in use should start with LCB for short horizons and gradually transition to a UCB-like strategy as more rounds are played. This article explores how and why this transition should occur. Our main result shows that our new algorithm performs nearly as well as the better of LCB and UCB at any point in time. The core idea behind our algorithm is broadly applicable, and we anticipate that our results will extend beyond the multi-armed bandit setting.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decentralized Relaxed Smooth Optimization with Gradient Descent Methods

    math.OC 2025-08 unverdicted novelty 6.0 of 10

    A decentralized gradient descent method with adaptive clipping is claimed to reach best-known convergence rates for convex and nonconvex problems under (L0,L1)-smoothness without knowing the constants.

Reference graph

Works this paper leans on

61 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution isbn issn journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in "" FUNCTION format.date year ...

  3. [3]

    and Goyal, N

    Agrawal, S. and Goyal, N. (2012). Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39--1. JMLR Workshop and Conference Proceedings

    2012

  4. [4]

    and Ortner, R

    Auer, P. and Ortner, R. (2010). Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica , 61(1-2):55--65

    work page 2010

  5. [5]

    J., Smith, L., Kostrikov, I., and Levine, S

    Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. (2023). Efficient online reinforcement learning with offline data. In International Conference on Machine Learning , pages 1577--1594. PMLR

    work page 2023

  6. [6]

    and Bayati, M

    Bastani, H. and Bayati, M. (2020). Online decision making with high-dimensional covariates. Operations Research , 68(1):276--294

    work page 2020

  7. [7]

    and Nemirovski, A

    Ben-Tal, A. and Nemirovski, A. (2002). Robust optimization--methodology and applications. Mathematical programming , 92:453--480

    work page 2002

  8. [8]

    B., and Caramanis, C

    Bertsimas, D., Brown, D. B., and Caramanis, C. (2011). Theory and applications of robust optimization. SIAM review , 53(3):464--501

    work page 2011

Show all 61 references
  1. [9]

    and Thiele, A

    Bertsimas, D. and Thiele, A. (2006). A robust optimization approach to inventory theory. Operations research , 54(1):150--168

    2006

  2. [10]

    Bu, J., Simchi-Levi, D., and Wang, L. (2023). Offline pricing and demand learning with censored data. Management Science , 69(2):885--903

    2023

  3. [11]

    Bu, J., Simchi-Levi, D., and Xu, Y. (2022). Online pricing with offline data: Phase transition and inverse square law. Management Science , 68(12):8568--8588

    2022

  4. [12]

    Buckman, J., Gelada, C., and Bellemare, M. G. (2020). The importance of pessimism in fixed-dataset policy optimization

    2020

  5. [13]

    Cai, B. (2024). Deepctr difm: Demonstrating deepctr with difm model on kaggle. Accessed: 2024-11-18

    2024

  6. [14]

    and Gallien, J

    Caro, F. and Gallien, J. (2007). Dynamic assortment with demand learning for seasonal consumer goods. Management science , 53(2):276--292

    2007

  7. [15]

    Chen, X., Shi, P., and Pu, S. (2022). Data-pooling reinforcement learning for personalized healthcare intervention. arXiv preprint arXiv:2211.08998

    2022 arXiv

  8. [16]

    Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. (2022). Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning , pages 3852--3878. PMLR

    2022

  9. [17]

    Cheung, W. C. and Lyu, L. (2024). Leveraging ( B iased) information: Multi-armed bandits with offline data. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F., editors, Proceedings of the 41st International Conference on Mach...

    2024

  10. [18]

    and Gu, S

    Fujimoto, S. and Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132--20145

    2021

  11. [19]

    Fujimoto, S., Meger, D., and Precup, D. (2018). Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning

    2018

  12. [20]

    Guo, H., Tang, R., Ye, Y., Li, Z., and He, X. (2017). Deepfm: a factorization-machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence , IJCAI'17, page 1725–1731. AAAI Press

    2017

  13. [21]

    and Momeni, A

    Gur, Y. and Momeni, A. (2022). Adaptive sequential experiments with unknown information arrival processes. Manufacturing & Service Operations Management , 24(5):2666--2684

    2022

  14. [22]

    Jin, Y., Ren, Z., Yang, Z., and Wang, Z. (2022). Policy learning" without''overlap: Pessimism and generalized empirical bernstein's inequality. arXiv preprint arXiv:2212.09900

    2022 arXiv

  15. [23]

    Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. (2020a). Morel: Model-based offline reinforcement learning. Advances in neural information processing systems , 33:21810--21823

    2020

  16. [24]

    Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. (2020b). Morel: Model-based offline reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 21810--...

    2020

  17. [25]

    Lattimore, T. (2016). Regret analysis of the anytime optimally confident ucb algorithm

    2016

  18. [26]

    and Szepesv \'a ri, C

    Lattimore, T. and Szepesv \'a ri, C. (2020). Bandit algorithms . Cambridge University Press

    2020

  19. [27]

    Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. (2021). Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning

    2021

  20. [28]

    Li, G., Ma, C., and Srebro, N. (2024a). Pessimism for offline linear contextual bandits using lp confidence sets. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22, Red Hook, NY, USA. Curran Associates Inc

    2024

  21. [29]

    D., Chi, Y., and Chen, Y

    Li, G., Zhan, W., Lee, J. D., Chi, Y., and Chen, Y. (2023). Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. arXiv preprint arXiv:2305.10282

    2023 arXiv

  22. [30]

    D., Chi, Y., and Chen, Y

    Li, G., Zhan, W., Lee, J. D., Chi, Y., and Chen, Y. (2024b). Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. Advances in Neural Information Processing Systems , 36

    2024

  23. [31]

    Lu, W., Yu, Y., Chang, Y., Wang, Z., Li, C., and Yuan, B. (2020). A dual input-aware factorization machine for ctr prediction. In International Joint Conference on Artificial Intelligence

    2020

  24. [32]

    Pandey, S., Agarwal, D., Chakrabarti, D., and Josifovski, V. (2007). Bandits for taxonomies: A model-based approach. In Proceedings of the 2007 SIAM international conference on data mining , pages 216--227. SIAM

    2007

  25. [33]

    and Roels, G

    Perakis, G. and Roels, G. (2008). Regret in the newsvendor model with partial information. Operations research , 56(1):188--203

    2008

  26. [34]

    F., Jiao, J., and Ramchandran, K

    Rajaraman, N., Yang, L. F., Jiao, J., and Ramchandran, K. (2020). Toward the fundamental limits of imitation learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  27. [35]

    Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2024). Bridging offline reinforcement learning and imitation learning: a tale of pessimism. In Proceedings of the 35th International Conference on Neural Information Processing Systems , NIPS '21, Red Hook, NY, USA....

    2024

  28. [36]

    and Bagnell, D

    Ross, S. and Bagnell, D. (2010). Efficient reductions for imitation learning. In Teh, Y. W. and Titterington, M., editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , ...

    2010

  29. [37]

    M., Bradlow, E

    Schwartz, E. M., Bradlow, E. T., and Fader, P. S. (2017). Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science , 36(4):500--522

    2017

  30. [38]

    Shen, W. (2024). Deepctr-torch: Easy-to-use, modular, and extendible pytorch framework for ctr prediction. Accessed: 2024-11-18

    2024

  31. [39]

    and Joachims, T

    Shivaswamy, P. and Joachims, T. (2012). Multi-armed bandit problems with history. In Lawrence, N. D. and Girolami, M., editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22 of Proceedings of Machine Learning Resear...

    2012

  32. [40]

    Song, Y., Zhou, Y., Sekhari, A., Andrew Bagnell, J., Krishnamurthy, A., and Sun, W. (2022). Hybrid RL : Using both offline and online data can make RL efficient. arXiv [cs.LG]

    2022

  33. [41]

    and Joachims, T

    Swaminathan, A. and Joachims, T. (2015a). Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1):1731--1755

    2015

  34. [42]

    and Joachims, T

    Swaminathan, A. and Joachims, T. (2015b). Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research , 16(52):1731--1755

    2015

  35. [43]

    Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285--294

    1933

  36. [44]

    Vershynin, R. (2018). Frontmatter , page i–ii. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press

    2018

  37. [45]

    and Pacchiano, A

    Wagenmaker, A. and Pacchiano, A. (2023). Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning , pages 35300--35338. PMLR

    2023

  38. [46]

    Wang, R., Fu, B., Fu, G., and Wang, M. (2017). Deep & cross network for ad click predictions

    2017

  39. [47]

    and Cukierski, W

    Wang, S. and Cukierski, W. (2014). Click-through rate prediction. https://kaggle.com/competitions/avazu-ctr-prediction. Kaggle

    2014

  40. [48]

    Wu, Y., Shariff, R., Lattimore, T., and Szepesv\' a ri, C. (2016). Conservative bandits. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, page 1254–1262. JMLR.org

    2016

  41. [49]

    Wu, Y., Tucker, G., and Nachum, O. (2019a). Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361

    2019 arXiv

  42. [50]

    Wu, Y., Tucker, G., and Nachum, O. (2019b). Behavior regularized offline reinforcement learning

    2019

  43. [51]

    Xiao, C., Wu, Y., Lattimore, T., Dai, B., Mei, J., Li, L., Szepesvari, C., and Schuurmans, D. (2021a). On the optimality of batch policy optimization algorithms. In International Conference on Machine Learning

    2021

  44. [52]

    Xiao, C., Wu, Y., Lattimore, T., Dai, B., Mei, J., Li, L., Szepesvari, C., and Schuurmans, D. (2021b). On the optimality of batch policy optimization algorithms. In International Conference on Machine Learning

    2021

  45. [53]

    Xie, T., Bhardwaj, M., Jiang, N., and Cheng, C.-A. (2022). Armor: A model-based framework for improving arbitrary baseline policies with offline data

    2022

  46. [54]

    Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2024). Policy finetuning: bridging sample-efficient offline and online reinforcement learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems , NIPS '21, Red Hook, NY, USA. Curr...

    2024

  47. [55]

    Xu, L., Zheng, Y., and Jiang, L. (2022). A robust data-driven approach for the newsvendor problem with nonparametric information. Manufacturing & Service Operations Management , 24(1):504--523

    2022

  48. [56]

    Yin, M., Bai, Y., and Wang, Y.-X. (2021). Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics

    2021

  49. [57]

    and Wang, Y.-X

    Yin, M. and Wang, Y.-X. (2021). Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065--4078

    2021

  50. [58]

    Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. (2020). Mopo: model-based offline policy optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  51. [59]

    Zheng, H., Luo, X., Wei, P., Song, X., Li, D., and Jiang, J. (2023). Adaptive policy learning for offline-to-online reinforcement learning. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of...

    2023

  52. [60]

    C., and Ryzhov, I

    Zhou, Y., Fu, M. C., and Ryzhov, I. O. (2024). Sequential learning with a similarity selection index. Operations Research , 72(6):2526--2542

    2024

  53. [61]

    Zhou, Y., Sekhari, A., Song, Y., and Sun, W. (2023). Offline data enhanced on-policy policy gradient with provable guarantees

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.