Pith. sign in

REVIEW 4 major objections 4 minor 95 references

Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regret

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that log-Sobolev isoperimetry of posteriors is enough for posterior-sampling RL to achieve order-optimal Bayesian regret, even when the posterior is sampled approximately by Langevin dynamics.

desk verdict A real extension of PSRL regret to LSI posteriors, but the headline sqrt(T) rate rests on an explicitly unproven linear-growth assumption on LSI constants. read the letter →

arxiv 2412.20824 v2 pith:VRN65EDE submitted 2024-12-30 cs.LG stat.ML

Emilio Jorge , Christos Dimitrakakis , Debabrota Basu This is my paper · ORCID
classification cs.LGstat.ML
keywords log-SobolevinequalityposteriorsamplingBayesianregretLangevindynamicsreinforcementlearningThompsonisoperimetricdistributions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the success of posterior-sampling reinforcement learning can be guaranteed for a wider class of models than the usual linear, kernel, Gaussian, or log-concave assumptions. It answers that if the data likelihood or the posterior satisfies the log-Sobolev inequality (LSI), then PSRL has sublinear Bayesian regret, and if the LSI constants grow linearly with the number of episodes, the regret is order-optimal $\tilde O(\sqrt{dHT})$. Because exact posteriors are rarely available, the paper introduces LaPSRL, which uses the SARAH-LD Langevin sampler to approximate the posterior with a decaying error, preserving order-optimal regret while keeping gradient complexity subquadratic. The claim matters because it extends provably efficient exploration to multimodal, perturbed, and other non-log-concave posterior families that appear in practice but are excluded by existing theory.

What carries the argument

The load-bearing object is the log-Sobolev inequality (LSI), a strong isoperimetric condition on a distribution $\nu$: for all smooth $g$, $\mathbb{E}_\nu[g^2\log g^2] - \mathbb{E}_\nu[g^2]\log\mathbb{E}_\nu[g^2] \le \frac{2}{\alpha}\mathbb{E}_\nu[\|\nabla g\|^2]$, with constant $\alpha>0$. LSI implies sub-Gaussian concentration of Lipschitz functions, which the paper converts into a posterior KL-concentration bound, and it also guarantees fast mixing of Langevin samplers. The second piece of machinery is the LSI constant's growth rate: linear growth ($\alpha_l = \Omega(Hl)$) is what upgrades the generic $O(T^{3/4})$ regret to order-optimal $\tilde O(\sqrt{dHT})$. The third piece is SARAH-LD, a variance-reduced Langevin sampler whose KL-accurate approximate samples with controlled gradient complexity let LaPSRL trade a decaying posterior error $\epsilon_{\text{post},l}$ against regret.

What would settle it

Run PSRL or LaPSRL on an episodic MDP whose posterior is a proven LSI distribution with sublinear LSI growth, for example a constructed multimodal posterior with $\alpha_n$ growing like $n^{1/2}$; if the empirical Bayesian regret grows faster than $\sqrt{T}$ and closer to the $T^{3/4}$ rate of Lemma 1, the order-optimal claim of Theorem 4 would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that isoperimetry, specifically the log-Sobolev inequality, is a sufficient condition for sublinear Bayesian regret in posterior-sampling RL. The proof route is that LSI gives Gaussian concentration of Lipschitz functions, which yields a polylogarithmic KL-concentration bound for the Bayesian posterior around the true MDP (Theorem 3). Feeding this bound into the PSRL regret decomposition gives sublinear regret (Lemma 1), and when the posterior's LSI constants grow at least linearly with episodes, the regret tightens to $\tilde O(\sqrt{dHT})$ (Theorem 4). For the approximate setting, LaPSRL controls the sampling error $\epsilon_{\text{post},l} = O(1/l)$, which is exactly the decay needed to retain order-optimal regret, and the SARAH-LD sampler keeps the total gradient complexity subquadratic (Corollary 2, Lemma 2).

Load-bearing premise

The near-optimal regret bound rests on the unproven premise that the posterior's log-Sobolev constant grows at least linearly with the number of episodes; the authors show this for Gaussian, log-concave, and mixture-of-log-concave posteriors but only conjecture it in general.

Editorial extensions

If this is right

  • Any Bayesian RL setting whose likelihood or posterior is LSI, and whose prior has positive mass around the true parameter, inherits sublinear Bayesian regret for PSRL without linear, kernel, or log-concavity assumptions.
  • If the LSI constants of the posterior grow linearly with the number of episodes, PSRL and LaPSRL both achieve $\tilde O(\sqrt{dHT})$ Bayesian regret, order-optimal in $H$ and $T$.
  • Approximate inference is not an obstacle: a posterior error that decays as $O(1/l)$ keeps the same regret order as exact PSRL.
  • LaPSRL with SARAH-LD runs with total gradient complexity $\tilde O(\tau T + \tau T^{1.5}/\sqrt{d})$ while preserving $\tilde O(\sqrt{dHT})$ regret, subquadratic in $T$.
  • The guarantees cover log-concave posteriors, mixtures of log-concave posteriors, Gaussian posteriors, and any LSI posterior with bounded likelihood ratio, including some multimodal and perturbed distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear-growth conjecture for LSI constants holds for general LSI posteriors, the same framework could cover many Bayesian neural-network posteriors, whose bounded likelihood ratios preserve LSI; the paper only demonstrates the linear scaling for specific families.
  • The regret bound's dependence on the LSI constant suggests a practical diagnostic: if an empirical estimate of $\alpha_l$ grows slower than linearly, one should expect the $T^{3/4}$ rate of Lemma 1 rather than the $\sqrt{T}$ rate.
  • The same posterior-concentration argument could likely be transplanted to other Bayesian algorithms whose regret decomposes through KL-concentration of models, such as UCB-style schemes.
  • Chained sampling is presented as a practical accelerator, but the variance-control argument for general LSI posteriors is deferred, so for LSI families beyond log-concave, chain initialization is an engineering choice rather than a proven speedup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies Bayesian regret for posterior-sampling RL when the prior/posterior is log-Sobolev (LSI). It proves a concentration bound for LSI likelihoods (Theorem 3), derives a sublinear O~(H T^{3/4}) regret for PSRL from that bound (Lemma 1), and shows that if posterior LSI constants grow linearly with the number of episodes, PSRL achieves O~(sqrt(dHT)) regret (Theorem 4). It then introduces LaPSRL, an approximate-posterior version that uses SARAH-LD, and analyzes the gradient complexity needed to preserve the regret order (Corollary 2, Lemma 2). A chained-sampling variant and its variance control are also discussed (Theorem 6). The paper closes with experiments on a Gaussian bandit, Cartpole, and Reacher, including bimodal and neural-network models.

Significance. If the main results were fully unconditional, the paper would make a useful contribution: it replaces log-concavity and Gaussian assumptions with LSI, covers mixtures and bounded perturbations, and provides a concrete Langevin sampling scheme with subquadratic complexity. The paper is also commendably explicit about its limitations, particularly the unproven linear growth of LSI constants. However, the headline order-optimal O~(sqrt(dHT)) claim is conditional on that unproven assumption, and the final confidence-interval proof in Theorem 3 contains a technical gap. The contribution is therefore best read as a conditional framework and a set of reusable tools, not as an unconditional proof that LSI posteriors alone suffice for order-optimal posterior-sampling RL.

major comments (4)
  1. [§5, after Theorem 8; abstract; Table 1] The advertised O~(sqrt(dHT)) regret is not established for general LSI posteriors. Theorem 4 gives the bound only when the posterior LSI constants satisfy alpha_l = Omega(Hl), and the paper itself states, in the paragraph after Theorem 8, that it has 'been unable to prove that the log-Sobolev constant of a posterior ... will always scale as Omega(n)' and merely conjectures this. Without that linear-growth assumption, Lemma 1 yields only O~(H T^{3/4}). Moreover, the Holley-Stroock bound in Theorem 2 allows the LSI constant to degrade by a factor of order e^{-2 osc(Phi)}; since each observation can contribute a bounded perturbation to the log-posterior, osc(Phi) may grow linearly with n, so LSI alone is consistent with non-growing or even decaying constants. The abstract's claim that LaPSRL achieves order-optimal regret, and the Table 1 row citing 'LSIP(M), linear growth on alpha', should therefore be stated as conditional on a conjecture that is currently unresolved.
  2. [Appendix C.1, proof of Theorem 3, Eqs. (15)-(16)] The proof of Theorem 3 does not establish Eq. (3) as written. In Eq. (15), the denominator is lower bounded using Jensen's inequality as R_Ξ e^{-nL_theta ||theta* - theta||} dP(theta) >= P(Ξ) (omega(Ξ)/P(Ξ))^n = omega(Ξ)^n / P(Ξ)^{n-1}, which yields a reciprocal factor P(Ξ)^{n-1}/omega(Ξ)^n rather than the displayed 1/(P(Ξ)omega(Ξ)^n). Independently, the final confidence interval in Eq. (17) contains no dependence on L_theta, even though L_theta is used in the denominator bound; the proof does not explain how the L_theta term is eliminated. Since Theorem 3 is the basis for the unconditional sublinear-regret Lemma 1, this gap needs to be repaired or the theorem restated.
  3. [Algorithm 2, Line 3; Corollary 1] The error schedule in Algorithm 2 appears inconsistent with the theorem it implements. Line 3 sets epsilon_post,l = g(H,S,A) / (l Delta_max^2), but Corollary 1 requires epsilon_post,l = O(g(H,S,A)^2 / (l Delta_max^2)). With the schedule as written and g(H,S,A) = O(sqrt(dH)), the accumulated approximation error is O(sqrt(g tau)) for tau episodes, which is O(d^{1/4} T^{1/2} H^{-1/4}) and is not the claimed O~(sqrt(dHT)) order. If this is a typo in the algorithm listing, it should be corrected; as printed, the algorithm does not satisfy the stated regret guarantee.
  4. [§4, paragraph after Theorem 6] The chained-sampling variant is not fully analyzed. Theorem 6 bounds the KL divergence between consecutive approximate posteriors only up to Var(P̂(M|H_l)), and the paragraph immediately after the theorem states that controlling this variance for LSI distributions 'demands analysing Langevin samplers independent of RL, which we defer to future work.' Since Algorithm 2 presents chaining as one of its two modes and the experiments in Appendix F use chained sampling, the paper's claim that 'LaPSRL with SARAH-LD' has the stated regret and gradient complexity is presently established only for the non-chained mode.
minor comments (4)
  1. [Abstract and Section 2] There are several typos and stylistic issues: 'expectated regret' in Section 2, 'is a a Bayesian algorithm' in Section 1, 'ineuqalities' in Section 5, and 'neccesary' in Appendix F. These should be cleaned up.
  2. [Section 3, Step 3] The quantity ξ(T) is introduced in the regret decomposition but is never defined before it is used in the bound BR + 2Hσ_R sqrt(T ξ(T)); please define it explicitly.
  3. [Table 1 and Theorem 4] Notation for the LSI constants is inconsistent: Theorem 4 states the bound in terms of α_{\bar R,l} and α_{\bar T,l}, but the following paragraph and Lemma 2 write α_T = Omega(T) and α_l = Omega(Hl). Please align the notation and state the scaling assumptions in the same symbols used by Corollary 2 and Lemma 2.
  4. [Algorithm 4, line 11] The update formula appears to use the prior density P(θ) where the log-prior should appear; from the definition f_i(θ) = -log P0(θ)/n - log L(x_i|θ), the gradient should involve ∇ log P(θ), not ∇ P(θ). Please check and correct.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the regret bounds are conditional on explicit LSI growth assumptions, not fitted to data; the unproven linear-growth premise is a correctness gap, not a circular step.

full rationale

The paper's derivation chain is not circular. Theorem 3 derives posterior KL concentration from the standard sub-Gaussian concentration consequence of LSI (Eq. 2) together with the explicit true-mass prior assumption; it does not assume the KL bound it claims. Lemma 1 and Theorem 4 then convert this concentration into regret by the standard PSRL decomposition, and the O~(sqrt(dHT)) statement appears only after the paper explicitly assumes posterior LSI constants grow linearly, alpha_l = Omega(Hl) (Theorem 4 discussion and Lemma 2). This is a conditional theorem, not a fitted parameter renamed as a prediction. The paper is transparent that the linear-growth scaling is unproven in general: Section 5 states 'we have been unable to prove that the log-Sobolev constant of a posterior... will always scale as Omega(n)' and calls it a conjecture. That is an unproven premise and a correctness risk for the headline rate, but the regret bound is not equivalent to the assumption by construction. The approximate-posterior analysis (Theorem 5, Corollaries 1-2, Lemma 2) transfers the exact-posterior regret guarantee to LaPSRL by choosing the sampling tolerance eps_post,l = O(g(H,S,A)/(l Delta_max^2)); this preserves the already-derived exact regret order and does not inject the conclusion into the premise. SARAH-LD's gradient complexity is imported from an external result (Kinoshita & Suzuki, 2022) and is not authored by this paper's authors. The only self-citations (Buening et al., 2023; Ouhamma et al., 2022; related-work entries) are prior-work or robustness references and are not load-bearing in the proofs. No equation is defined in terms of the quantity it is supposed to prove, and no parameter is fit to force a regret bound. Thus the paper receives a low circularity score, with the linear-growth assumption flagged as an external correctness gap.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The central theory rests on standard LSI properties, a set of domain assumptions about the MDP and likelihood, and one explicitly unproven scaling assumption (linear growth of LSI constants). No new physical or mathematical entities are introduced. The free parameters are experimental hyperparameters, not fitted constants in the proof.

free parameters (3)
  • LSI constant alpha in experiments = 100n, 1000n, 10000n for Cartpole; 10000n, 100000n for Reacher
    Chosen by hand in Appendix F.2 rather than estimated; needed to set SARAH-LD step sizes.
  • L parameter in experiments = alpha/n
    Set to alpha/n in both continuous control experiments, simplifying the smoothness parameter.
  • Batched bandit action repetitions = 20
    Each action is played 20 times per sampled model to reduce computational cost.
assumptions (7)
  • domain assumption Data distribution satisfies LSI with constant alpha_theta* (Definition 1)
    Central assumption of Theorem 3 and all regret bounds.
  • domain assumption L-smoothness of the log-likelihood (Assumption 1)
    Required for the SARAH-LD gradient complexity bound and for the sampling analysis.
  • ad hoc to paper True-mass prior (Assumption 2)
    Prior must put non-negligible mass in neighborhoods of the true parameter; used in Theorem 3's concentration proof.
  • domain assumption One-step value function is Lipschitz in the mean state distribution (Assumption 3)
    Required for Theorem 4's sqrt(T) regret bound.
  • ad hoc to paper Posterior LSI constants grow linearly, alpha_l = Omega(Hl)
    Not proven for general LSI posteriors; the paper conjectures it in Section 5 after Theorem 8.
  • standard math SARAH-LD gradient complexity bound (Theorem 9 of Kinoshita and Suzuki)
    External result used to convert sampling error into gradient complexity.
  • standard math Mixture LSI constant bounds (Theorem 10 from Koehler and Vuong)
    Used to derive LSI constants for mixtures of log-concave posteriors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regret." pith.science (2026). https://pith.science/paper/VRN65EDE

@misc{pith2026241220824,
  author       = {Pith},
  title        = {Pith review of: Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regret},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VRN65EDE}},
  note         = {Machine review of arXiv:2412.20824}
}
read the original abstract

Common assumptions, like linear or RKHS models, and Gaussian or log-concave posteriors over the models, do not explain practical success of RL across a wider range of distributions and models. Thus, we study how to design RL algorithms with sublinear regret for isoperimetric distributions, specifically the ones satisfying the Log-Sobolev Inequality (LSI). LSI distributions include the standard setups of RL theory, and others, such as many non-log-concave and perturbed distributions. First, we show that the Posterior Sampling-based RL (PSRL) algorithm yields sublinear regret if the data distributions satisfy LSI and some mild additional assumptions. Also, when we cannot compute or sample from an exact posterior, we propose a Langevin sampling-based algorithm design: LaPSRL. We show that LaPSRL achieves order-optimal regret and subquadratic complexity per episode. Finally, we deploy LaPSRL with a Langevin sampler -- SARAH-LD, and test it for different bandit and MDP environments. Experimental results validate the generality of LaPSRL across environments and its competitive performance with respect to the baselines.

Figures

Figures reproduced from arXiv: 2412.20824 by the authors.

Figure 1
Figure 1. Examples of log-Sobolev distributions. Isoperimetry relates to the ratio between the area of the boundary and the volume of a set. It is known that some isoperimetric condition is needed for rapid mixing of Markov chains to avoid the risk of getting stuck in bad regions (Stroock 1Chalmers University of Technology, Sweden 2University of Gothenburg, Sweden 3University of Neuchâtel, Switzerland 4University of Oslo, Nor… view at source ↗
Figure 2
Figure 2. We compare LaPSRL against baselines. In the bandit and Cartpole experiments, we benchmark with PSRL, and in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. We compare LaPSRL versus baselines. In the bandit and Cartpole experiments we benchmark with PSRL, in [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

95 extracted references · 66 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Lazaric, A

    Abeille, M. and Lazaric, A. Improved regret bounds for thompson sampling in linear quadratic control problems. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 1--9. PMLR, 10--15 Jul 2018. URL http://proceedings.mlr.press/v80/abeille18a.html

    2018

  3. [3]

    and Jia, R

    Agrawal, S. and Jia, R. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/f...

    2017

  4. [4]

    Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning

    Anonymous. Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=FvQsk3la17

    2025

  5. [5]

    G., Osband, I., and Munos, R

    Azar, M. G., Osband, I., and Munos, R. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pp.\ 263--272. PMLR, 2017

    2017

  6. [6]

    Analysis and geometry of Markov diffusion operators, volume 103

    Bakry, D., Gentil, I., Ledoux, M., et al. Analysis and geometry of Markov diffusion operators, volume 103. Springer, 2014

    2014

  7. [7]

    and Kolesnikov, A

    Barthe, F. and Kolesnikov, A. V. Mass transport and variants of the logarithmic sobolev inequality. Journal of Geometric Analysis, 18 0 (4): 0 921--979, 2008

    2008

  8. [8]

    G., Sutton, R

    Barto, A. G., Sutton, R. S., and Anderson, C. W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13 0 (5): 0 834--846, 1983. doi:10.1109/TSMC.1983.6313077

    arXiv 1983

Show all 95 references
  1. [9]

    On the log-sobolev constant of log-concave measures, 2023

    Bizeul, P. On the log-sobolev constant of log-concave measures, 2023

    2023

  2. [10]

    Concentration inequalities

    Boucheron, S., Lugosi, G., and Bousquet, O. Concentration inequalities. In Summer school on machine learning, pp.\ 208--240. Springer, 2003

    2003

  3. [11]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  4. [12]

    K., Dimitrakakis, C., Eriksson, H., Grover, D., and Jorge, E

    Buening, T. K., Dimitrakakis, C., Eriksson, H., Grover, D., and Jorge, E. Minimax-bayes reinforcement learning. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Procee...

    2023

  5. [13]

    Bayesian linear regression with sparse priors

    Castillo, I., Schmidt-Hieber, J., and Van der Vaart, A. Bayesian linear regression with sparse priors. The Annals of Statistics, pp.\ 1986--2018, 2015

    1986

  6. [14]

    A note on talagrand’s transportation inequality and logarithmic sobolev inequality

    Cattiaux, P., Guillin, A., and Wu, L.-M. A note on talagrand’s transportation inequality and logarithmic sobolev inequality. Probability theory and related fields, 148: 0 285--304, 2010

    2010

  7. [15]

    and Lehec, J

    Chafa\" i , D. and Lehec, J. Logarithmic sobolev inequalities essentials, 2023. URL https://djalil.chafai.net/docs/M2/chafai-lehec-m2-lsie-lecture-notes.pdf. Accessed on 08/10/2024

    2023

  8. [16]

    Thompson sampling for high-dimensional sparse linear contextual bandits

    Chakraborty, S., Roy, S., and Tewari, A. Thompson sampling for high-dimensional sparse linear contextual bandits. In International Conference on Machine Learning, pp.\ 3979--4008. PMLR, 2023

    2023

  9. [17]

    Dimension-free log-sobolev inequalities for mixture distributions

    Chen, H.-B., Chewi, S., and Niles-Weed, J. Dimension-free log-sobolev inequalities for mixture distributions. Journal of Functional Analysis, 281 0 (11): 0 109236, 2021. ISSN 0022-1236. doi:https://doi.org/10.1016/j.jfa.2021.109236

    2021

  10. [18]

    Chowdhury, S. R. and Gopalan, A. Online learning in kernelized markov decision processes, 2019

    2019

  11. [19]

    R., Gopalan, A., and Maillard, O.-A

    Chowdhury, S. R., Gopalan, A., and Maillard, O.-A. Reinforcement learning in parametric mdps with exponential families. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceeding...

    2021

  12. [20]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Proc...

    2018

  13. [21]

    Bagger: A bayesian algorithm for safe and query-efficient imitation learning

    Cronrath, C., Jorge, E., Moberg, J., Jirstrand, M., and Lennartson, B. Bagger: A bayesian algorithm for safe and query-efficient imitation learning. 2018

    2018

  14. [22]

    Model based B ayesian exploration

    Dearden, R., Friedman, N., and Andre, D. Model based B ayesian exploration. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp.\ 150--159, 1999

    1999

  15. [23]

    Robust Bayesian reinforcement learning through tight lower bounds

    Dimitrakakis, C. Robust Bayesian reinforcement learning through tight lower bounds. In European Workshop on Reinforcement Learning (EWRL 2011), pp.\ 177--188, 2011

    2011

  16. [24]

    A., J Reddi, S., Williamson, S

    Dubey, K. A., J Reddi, S., Williamson, S. A., Poczos, B., Smola, A. J., and Xing, E. P. Variance reduction in stochastic gradient langevin dynamics. Advances in neural information processing systems, 29, 2016

    2016

  17. [25]

    and Van Roy, B

    Dwaracherla, V. and Van Roy, B. Langevin dqn. arXiv preprint arXiv:2002.07282, 2020

    2002 arXiv

  18. [26]

    Exponential families in theory and practice

    Efron, B. Exponential families in theory and practice. Cambridge University Press, 2022

    2022

  19. [27]

    Sentinel: taming uncertainty with ensemble based distributional reinforcement learning

    Eriksson, H., Basu, D., Alibeigi, M., and Dimitrakakis, C. Sentinel: taming uncertainty with ensemble based distributional reinforcement learning. In Cussens, J. and Zhang, K. (eds.), Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume...

    2022

  20. [28]

    and Ming, Y

    Fan, Y. and Ming, Y. Model-based reinforcement learning for continuous control with posterior sampling. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 3078--3...

    2021

  21. [29]

    Bayesian bellman operators

    Fellows, M., Hartikainen, K., and Whiteson, S. Bayesian bellman operators. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 13641--13656. Curran Associates, Inc., 2021. URL ht...

    2021

  22. [30]

    Ferguson, T. S. Location and scale parameters in exponential families of distributions. The Annals of mathematical statistics, 33 0 (3): 0 986--1001, 1962

    1962

  23. [31]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  24. [32]

    J., Tellex, S., Chowdhary, G., Roy, N., and How, J

    Geramifard, A., Walsh, T. J., Tellex, S., Chowdhary, G., Roy, N., and How, J. P. A tutorial on linear function approximators for dynamic programming and reinforcement learning. Foundations and Trends® in Machine Learning, 6 0 (4): 0 375--451, 2013. ISSN 1935-8237. doi:10.1561/...

    2013 doi

  25. [33]

    Convex optimization: Algorithms and complexity

    Ghavamzadeh, M., Mannor, S., Pineau, J., and Tamar, A. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8 0 (5–6): 0 359–483, 2015. ISSN 1935-8245. doi:10.1561/2200000049. URL http://dx.doi.org/10.1561/2200000049

    2015 doi

  26. [34]

    A review of consistency and convergence of posterior distribution

    Ghosal, S. A review of consistency and convergence of posterior distribution. In Varanashi Symposium in Bayesian Inference, Banaras Hindu University. Citeseer, 1997

    1997

  27. [35]

    Bayesian reinforcement learning via deep, sparse sampling

    Grover, D., Basu, D., and Dimitrakakis, C. Bayesian reinforcement learning via deep, sparse sampling. In International Conference on Artificial Intelligence and Statistics, pp.\ 3036--3045. PMLR, 2020

    2020

  28. [36]

    A pac-bayesian link between generalisation and flat minima

    Haddouche, M., Viallard, P., Simsekli, U., and Guedj, B. A pac-bayesian link between generalisation and flat minima. arXiv preprint arXiv:2402.08508, 2024

    2024 arXiv

  29. [37]

    R., Precup, D., and Xu, P

    Haque, I., Tan, Y., Yang, Y., Lan, Q., Lu, J., Mahmood, A. R., Precup, D., and Xu, P. More efficient randomized exploration for reinforcement learning via approximate sampling. Reinforcement Learning Journal, 3 0 (1), 2024

    2024

  30. [38]

    and Stroock, D

    Holley, R. and Stroock, D. Logarithmic sobolev inequalities and stochastic ising models. Journal of Statistical Physics, 46 0 (5-6): 0 1159--1194, 1987

    1987

  31. [39]

    Randomized exploration in cooperative multi-agent reinforcement learning

    Hsu, H.-L., Wang, W., Pajic, M., and Xu, P. Randomized exploration in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2404.10728, 2024

    2024 arXiv

  32. [40]

    Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 1--18, 2022. URL http://jmlr.org/papers...

    2022

  33. [41]

    Optimal regret is achievable with bounded approximate inference error: An enhanced bayesian upper confidence bound framework

    Huang, Z., Lam, H., Meisami, A., and Zhang, H. Optimal regret is achievable with bounded approximate inference error: An enhanced bayesian upper confidence bound framework. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/...

    2023

  34. [42]

    Tight regret and complexity bounds for thompson sampling via langevin monte carlo

    Huix, T., Zhang, M., and Durmus, A. Tight regret and complexity bounds for thompson sampling via langevin monte carlo. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of...

    2023

  35. [43]

    R., Precup, D., Anandkumar, A., and Azizzadenesheli, K

    Ishfaq, H., Lan, Q., Xu, P., Mahmood, A. R., Precup, D., Anandkumar, A., and Azizzadenesheli, K. Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo. In The Twelfth International Conference on Learning Representations, 2024. URL htt...

    2024

  36. [44]

    M., and Jordan, M

    Jin, C., Netrapalli, P., Ge, R., Kakade, S. M., and Jordan, M. I. A short note on concentration inequalities for random vectors with subgaussian norm, 2019. URL https://arxiv.org/abs/1902.03736

    2019 arXiv

  37. [45]

    I Can't Believe It's Not Better!

    Jorge, E., Eriksson, H., Dimitrakakis, C., Basu, D., and Grover, D. Inferential induction: A novel framework for B ayesian reinforcement learning. In Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, volume 137 of Proceedings of Machine Learning Research,...

    2020

  38. [46]

    Kalman, R. E. A new approach to linear filtering and prediction problems. 1960

    1960

  39. [47]

    L., Ma, Y., and Mitra, S

    Karbasi, A., Kuang, N. L., Ma, Y., and Mitra, S. L angevin thompson sampling with logarithmic communication: Bandits and reinforcement learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Co...

    2023

  40. [48]

    Learning linear-quadratic regulators via thompson sampling with preconditioned langevin dynamics, 2023

    Kim, G. Learning linear-quadratic regulators via thompson sampling with preconditioned langevin dynamics, 2023

    2023

  41. [49]

    Approximate thompson sampling for learning linear quadratic regulators with o ( \ T\ ) regret

    Kim, Y., Kim, G., and Yang, I. Approximate thompson sampling for learning linear quadratic regulators with o ( \ T\ ) regret. arXiv preprint arXiv:2405.19380, 2024

    2024 arXiv

  42. [50]

    and Suzuki, T

    Kinoshita, Y. and Suzuki, T. Improved convergence rate of stochastic gradient langevin dynamics with variance reduction and its application to optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. UR...

    2022

  43. [51]

    and Vuong, T.-D

    Koehler, F. and Vuong, T.-D. Sampling multimodal distributions with the vanilla score: Benefits of data-based initialization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=oAMArMMQxb

    2024

  44. [52]

    Statistical efficiency of score matching: The view from isoperimetry

    Koehler, F., Heckett, A., and Risteski, A. Statistical efficiency of score matching: The view from isoperimetry. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TD7AnQjNzR6

    2023

  45. [53]

    and Yin, G

    Krishnamurthy, V. and Yin, G. Langevin dynamics for adaptive inverse reinforcement learning of stochastic gradient algorithms. Journal of Machine Learning Research, 22 0 (121): 0 1--49, 2021. URL http://jmlr.org/papers/v22/20-625.html

    2021

  46. [54]

    L., Yin, M., Wang, M., Wang, Y.-X., and Ma, Y

    Kuang, N. L., Yin, M., Wang, M., Wang, Y.-X., and Ma, Y. Posterior sampling with delayed feedback for reinforcement learning with linear function approximation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=RiyH3z7oIF

    2023

  47. [55]

    and Szepesv \'a ri, C

    Lattimore, T. and Szepesv \'a ri, C. Bandit algorithms. Cambridge University Press, 2020

    2020

  48. [56]

    Concentration of measure and logarithmic sobolev inequalities

    Ledoux, M. Concentration of measure and logarithmic sobolev inequalities. In Seminaire de probabilites XXXIII, pp.\ 120--216. Springer, 2006

    2006

  49. [57]

    E., Bottero, A

    Luis, C. E., Bottero, A. G., Vinogradska, J., Berkenkamp, F., and Peters, J. Model-based uncertainty in value functions. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 ...

    2023

  50. [58]

    E., Bottero, A

    Luis, C. E., Bottero, A. G., Vinogradska, J., Berkenkamp, F., and Peters, J. Value-distributional model-based reinforcement learning. Journal of Machine Learning Research, 25 0 (298): 0 1--42, 2024 a . URL http://jmlr.org/papers/v25/23-0913.html

    2024

  51. [59]

    E., Bottero, A

    Luis, C. E., Bottero, A. G., Vinogradska, J., Berkenkamp, F., and Peters, J. Model-based epistemic variance of values for risk-aware policy optimization, 2024 b . URL https://arxiv.org/abs/2312.04386

    2024 arXiv

  52. [60]

    On approximate thompson sampling with langevin algorithms

    Mazumdar, E., Pacchiano, A., Ma, Y., Jordan, M., and Bartlett, P. On approximate thompson sampling with langevin algorithms. In International Conference on Machine Learning, pp.\ 6797--6807. PMLR, 2020

    2020

  53. [61]

    Bayesian linear regression

    Minka, T. Bayesian linear regression. Technical report, Citeseer, 2000

    2000

  54. [62]

    S., and Aggarwal, V

    Moradipari, A., Pedramfar, M., Zini, M. S., and Aggarwal, V. Improved bayesian regret bounds for thompson sampling in reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=2EVTB1idyR

    2023

  55. [63]

    A diffusion process perspective on posterior contraction rates for parameters

    Mou, W., Ho, N., Wainwright, M., Bartlett, P., and Jordan, M. A diffusion process perspective on posterior contraction rates for parameters. SIAM Journal on Mathematics of Data Science, 6 0 (2): 0 553--577, 2024

    2024

  56. [64]

    Posterior sampling via langevin monte carlo for offline reinforcement learning, 2024

    Nguyen-Tang, T., Yin, M., Uehara, M., Wang, Y.-X., Wang, M., and Arora, R. Posterior sampling via langevin monte carlo for offline reinforcement learning, 2024. URL https://openreview.net/forum?id=WwCirclMvl

    2024

  57. [65]

    Convex analysis of the mean field langevin dynamics

    Nitanda, A., Wu, D., and Suzuki, T. Convex analysis of the mean field langevin dynamics. In Camps-Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine ...

    2022

  58. [66]

    Variational bayesian reinforcement learning with regret bounds

    O' Donoghue, B. Variational bayesian reinforcement learning with regret bounds. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 28208--28221. Curran Associates, Inc., 2021. U...

    2021

  59. [67]

    The uncertainty bellman equation and exploration

    O'Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp.\ 3836--3845, 2018

    2018

  60. [68]

    and Van Roy, B

    Osband, I. and Van Roy, B. Model-based reinforcement learning and the eluder dimension. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https:/...

    2014

  61. [69]

    and Van Roy, B

    Osband, I. and Van Roy, B. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning, pp.\ 2701--2710. PMLR, 2017

    2017

  62. [70]

    Osband, I., Russo, D., and Roy, B. V. (more) efficient reinforcement learning via posterior sampling. In NIPS , 2013 a

    2013

  63. [71]

    (more) efficient reinforcement learning via posterior sampling

    Osband, I., Russo, D., and Van Roy, B. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013 b

    2013

  64. [72]

    M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B

    Osband, I., Wen, Z., Asghari, S. M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B. Approximate T hompson sampling via epistemic neural networks. In Evans, R. J. and Shpitser, I. (eds.), Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligen...

    2023

  65. [73]

    Bilinear exponential family of mdps: Frequentist regret bound with tractable exploration and planning

    Ouhamma, R., Basu, D., and Maillard, O.-A. Bilinear exponential family of mdps: Frequentist regret bound with tractable exploration and planning. arXiv preprint arXiv:2210.02087, 2022

    2022 arXiv

  66. [74]

    Thompson sampling and approximate inference

    Phan, M., Abbasi Yadkori, Y., and Domke, J. Thompson sampling and approximate inference. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc....

    2019

  67. [75]

    Sample-efficient cross-entropy method for real-time planning

    Pinneri, C., Sawant, S., Blaes, S., Achterhold, J., Stueckler, J., Rolinek, M., and Martius, G. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning 2020, 2020. URL https://corlconf.github.io/paper_217

    2020

  68. [76]

    Online kernel selection for bayesian reinforcement learning

    Reisinger, J., Stone, P., and Miikkulainen, R. Online kernel selection for bayesian reinforcement learning. In International Conference on Machine Learning, pp.\ 816--823, 2008

    2008

  69. [77]

    and Van Roy, B

    Russo, D. and Van Roy, B. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39 0 (4): 0 1221--1243, 2014. doi:10.1287/moor.2014.0650. URL https://doi.org/10.1287/moor.2014.0650

    2014

  70. [78]

    V., Kazerouni, A., Osband, I., and Wen, Z

    Russo, D., Roy, B. V., Kazerouni, A., Osband, I., and Wen, Z. A tutorial on thompson sampling, 2020

    2020

  71. [79]

    Posterior sampling for deep reinforcement learning

    Sasso, R., Conserva, M., and Rauber, P. Posterior sampling for deep reinforcement learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedin...

    2023

  72. [80]

    Poincar \'e and log--sobolev inequalities for mixtures

    Schlichting, A. Poincar \'e and log--sobolev inequalities for mixtures. Entropy, 21 0 (1): 0 89, 2019

    2019

  73. [81]

    Proximal policy optimization algorithms, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  74. [82]

    J., Lykouris, T., Dudik, M., and Schapire, R

    Simchowitz, M., Tosh, C., Krishnamurthy, A., Hsu, D. J., Lykouris, T., Dudik, M., and Schapire, R. E. Bayesian decision-making under misspecified priors with applications to meta-learning. Advances in Neural Information Processing Systems, 34: 0 26382--26394, 2021

    2021

  75. [83]

    A feynman-kac approach for logarithmic sobolev inequalities, 2021

    Steiner, C. A feynman-kac approach for logarithmic sobolev inequalities, 2021

    2021

  76. [84]

    Stroock, D. W. and Zegarlinski, B. The equivalence of the logarithmic sobolev inequality and the dobrushin-shlosman mixing condition. Communications in mathematical physics, 144: 0 303--323, 1992

    1992

  77. [85]

    On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of two Samples

    Thompson, W. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of two Samples . Biometrika, 25 0 (3-4): 0 285--294, 1933

    1933

  78. [86]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  79. [87]

    U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al

    Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  80. [88]

    and Wibisono, A

    Vempala, S. and Wibisono, A. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curr...

    2019

  81. [89]

    Wang, C., Chen, Y., and Murphy, K. P. Model-based policy optimization under approximate bayesian inference. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

    2023

  82. [90]

    Logarithmic sobolev inequalities: conditions and counterexamples

    Wang, F.-Y. Logarithmic sobolev inequalities: conditions and counterexamples. Journal of Operator Theory, pp.\ 183--197, 2001

    2001

  83. [91]

    and Ghosal, S

    Wang, K. and Ghosal, S. Posterior contraction and testing for multivariate isotonic regression. Electronic Journal of Statistics, 17 0 (1): 0 798--822, 2023

    2023

  84. [92]

    V., Azizzadenesheli, K., and Anandkumar, A

    Xu, P., Zheng, H., Mazumdar, E. V., Azizzadenesheli, K., and Anandkumar, A. L angevin M onte C arlo for contextual bandits. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Lear...

    2022

  85. [93]

    Mean field langevin actor-critic: Faster convergence and global optimality beyond lazy learning

    Yamamoto, K., Oko, K., Yang, Z., and Suzuki, T. Mean field langevin actor-critic: Faster convergence and global optimality beyond lazy learning. In Forty-first International Conference on Machine Learning, 2024

    2024

  86. [94]

    Accelerating approximate T hompson sampling with underdamped L angevin M onte C arlo

    Zheng, H., Deng, W., Moya, C., and Lin, G. Accelerating approximate T hompson sampling with underdamped L angevin M onte C arlo. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 23...

    2024

  87. [95]

    Varibad: variational bayes-adaptive deep rl via meta-learning

    Zintgraf, L., Schulze, S., Lu, C., Feng, L., Igl, M., Shiarlis, K., Gal, Y., Hofmann, K., and Whiteson, S. Varibad: variational bayes-adaptive deep rl via meta-learning. The Journal of Machine Learning Research, 22 0 (1): 0 13198--13236, 2021

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.