Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Fully Offline Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Bayesian posterior's information loss bounds the regret of an offline-trained policy, letting hyperparameters and deployment timing be decided entirely offline.

desk verdict Promising offline-RL framework with a genuine PIL regret bound, but the deployment decision rests on an untested heuristic and Theorem 2 has a fixable typo. read the letter →

arxiv 2505.22442 v3 pith:MVUATXYV submitted 2025-05-28 cs.LG cs.AI

Mattie Fellows , Clarisse Wibault , Uljad Berdica , Johannes Forkel , Maike Osborne , Jakob N. Foerster This is my paper
classification cs.LGcs.AI
keywords offlinereinforcementlearningBayesianposteriorinformationlossregretboundsfullyhyperparametertuningmodel-basedRLBayes-adaptiveMDPsafedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning promises policies trained without further environment interaction, but current methods tune hyperparameters online and cannot estimate how badly a trained policy will do when first deployed. This paper claims both problems can be solved inside a Bayesian framework: a posterior over dynamics, learned only from offline data, yields a bound on the deployed policy's true regret before any online rollout. The bound is $\mathrm{Regret}(M^\star,D_N)\le 2R_{\max}\sqrt{1-\exp(-I^\pi_N/(1-\gamma))}$, where $I^\pi_N$ is the posterior information loss, an expected KL divergence between true and model dynamics; for parametric Gaussian world models this gives the parametric minimax regret rate $O(\sqrt{d/N})$. The paper turns this theory into two algorithms: SOReL, which estimates regret offline from the posterior predictive median of returns and tunes all hyperparameters offline, and TOReL, which applies the same information-rate-based tuning to general offline RL methods. In standard continuous-control benchmarks, TOReL's offline tuning matches online tuning and can save hundreds of thousands of online samples.

What carries the argument

The load-bearing object is the posterior information loss (PIL), $I^\pi_N=\mathbb{E}_{\theta\sim P_\Theta(D_N)}\mathbb{E}_{(s,a)\sim\rho^\star_\pi}\mathrm{KL}(P^\star_{R,S}(s,a)\|P_{R,S}(s,a,\theta))$, an expected KL divergence between true and model dynamics weighted by the arithmetico-geometric ergodic state-action distribution $\rho^\star_\pi$, which downweights state-action pairs reached only after many steps. Theorem 1 converts PIL into a regret bound through the Bretagnolle-Huber inequality, so the information rate, the rate at which PIL falls with more data, becomes the object that governs regret decay. For Gaussian world models, PIL decomposes into a mean-squared-error term plus a predictive-variance term, both estimable from offline data, and Theorem 2 derives the $\sqrt{d/N}$ rate from this decomposition under local asymptotic normality. The second mechanism is the posterior predictive median of returns in Eq. (10), which serves as the offline proxy for regret by rolling out the Bayes-optimal policy under posterior-sampled dynamics; SOReL tunes the BAMDP solver against this proxy and checks the MSE/variance balance as a reliability signal.

What would settle it

Train SOReL on an offline dataset with deliberately limited coverage, such as one that contains no transitions in the state-action region the optimal policy visits, using a Gaussian world model that is known to be misspecified; compute approximate regret from Eq. (10) and then deploy to measure true regret. If approximate regret falls below $R_{\mathrm{Deploy}}$ while true regret is above it, the proxy is falsified, and the paper's own 'unreliable' shaded region for MSE/variance misalignment is exactly the regime where this should be tested.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that Bayesian offline RL has a clean frequentist regret theory: the true regret of the Bayes-optimal policy trained on the posterior from offline data is controlled by how much information the posterior still lacks about the true dynamics. Theorem 1 bounds regret by $2R_{\max}\sup_\pi\sqrt{1-\exp(-I^\pi_N/(1-\gamma))}$, with $I^\pi_N$ defined as the expected posterior KL divergence between the true transition-reward distribution and the model, weighted by an arithmetico-geometric ergodic state-action distribution. Theorem 2 shows that, under standard local asymptotic normality assumptions, the expected regret of parametric Gaussian world models decays at the parametric minimax rate $O(\sqrt{d/N})$, the same rate as frequentist parametric density estimation. The paper then operationalizes this: SOReL estimates regret before deployment as $\hat{R}_{\max}$ minus the median predictive return from posterior rollouts, monitors the PIL and the alignment between mean-squared error and predictive variance to tune model and inference hyperparameters offline, and deploys only when approximate regret falls below a threshold $R_{\mathrm{Deploy}}$. The empirical claim is that this approximate regret tracks true regret in the tested control environments and that TOReL's fully offline hyperparameter selection is competitive with online tuning while sparing large numbers of online samples.

Load-bearing premise

The deployment certificate rests on the unproven hypothesis, introduced in Section 6.1 rather than derived, that the posterior predictive median of returns in Eq. (10) approximates true regret closely enough that the safety threshold $R_{\mathrm{Deploy}}$ is meaningful; that proxy is validated on only five control environments.

Editorial extensions

If this is right

  • A practitioner can set a safety threshold $R_{\mathrm{Deploy}}$ and use only offline data to decide whether the Bayes-optimal policy is ready to deploy, breaking the train-deploy-fail-retune loop.
  • The information rate becomes a fully offline diagnostic: a flat PIL curve signals that more data, a different model class, or different approximate-inference settings are needed before deployment.
  • For parametric Gaussian world models, the $\sqrt{d/N}$ regret rate means the number of offline transitions needed for a target regret grows roughly linearly with the effective dimension of the dynamics model.
  • TOReL's PIL-based selection can replace online hyperparameter sweeps for both model-free and model-based offline RL algorithms, with no measured sacrifice in final regret.
  • Because the safety decision is made through posterior predictive rollouts, the guarantee is only as strong as the posterior's coverage of the true MDP; outside that support the bound still holds, but the proxy in Eq. (10) may not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the regret bound in Theorem 1 applies to an exact Bayes-optimal policy and exact posterior, while SOReL's experiments use approximate inference and an approximate BAMDP solver; a natural sharper check is to measure how approximation error enters the gap between the Eq. (10) proxy and true regret.
  • Editorial inference: if the posterior-predictive-median proxy is taken as calibrated, the same construction could estimate other deployment metrics of interest, such as the probability of constraint violation in safety-critical control, not just discounted return.
  • Editorial inference: the $\sqrt{d/N}$ rate suggests that lowering effective model dimension, for example by a structured prior or by model reduction, should directly reduce the data needed to reach a given regret level; the paper's plateau curves imply this but do not test it.
  • Editorial inference: the current empirical support for the proxy covers five control environments with deliberately diverse full-replay datasets; testing on misspecified model classes or datasets with missing coverage in the optimal-policy region would clarify whether the proxy remains conservative where the posterior is least trustworthy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SOReL, a Bayesian model-based offline RL method that learns a posterior over environment dynamics and uses the posterior information loss (PIL) to bound true regret and to tune hyperparameters entirely offline, and TOReL, a tuning framework that applies the same offline tuning principles to general model-free and model-based ORL algorithms. The theoretical contribution is a regret bound in terms of the PIL (Theorem 1) and a claimed parametric minimax rate for the expected regret of a Bayes-optimal policy in Gaussian world models (Theorem 2). The empirical sections evaluate SOReL's approximate regret proxy and TOReL's offline hyperparameter selection on brax and D4RL MuJoCo benchmarks, reporting near-oracle performance for ReBRAC+TOReL and large savings in online samples relative to UCB bandit tuning.

Significance. If the central claims hold, the paper makes a valuable contribution: it gives a principled information-theoretic handle on offline RL regret and shows, on standard benchmarks, that hyperparameter tuning can be moved fully offline. The manuscript ships public code, includes detailed proofs, and reports concrete sample-complexity savings in the TOReL experiments. However, the printed statement of Theorem 2 is internally inconsistent with its own proof, and the deployment safety gate in SOReL rests on a heuristic approximation that is not validated in the suboptimal-data regime. The theoretical and empirical ideas are significant, but the main safety claims are not yet supported in their current form.

major comments (3)
  1. [Section 5.2, Eq. (5) and Section D.3] The displayed expected-regret bound in Eq. (5) is 2Rmax * exp(1 - sqrt(Cd/((1-gamma)N))), which tends to 2e Rmax as N grows and never decays to zero. The proof at the end of Section D.3 instead derives 2Rmax * sqrt(1 - exp(-C'd/((1-gamma)N))), which decays at the claimed O(sqrt(d/N)) rate. The theorem statement and Figure 2 must be aligned with the proof; as printed, Theorem 2 does not establish the claimed parametric minimax rate.
  2. [Section 6.1, Eq. (10) and Algorithm 1] The deploy/no-deploy decision in Algorithm 1 is gated by the approximate regret Rhat_max minus the median posterior predictive return, where Rhat_max is estimated from the maximum return in the offline dataset. When the offline data is collected by a suboptimal or narrow policy, Rhat_max can be much smaller than J(pi*), so the proxy can be near zero even when the deployed policy is far from optimal. The theoretical bound in Theorem 1 is not evaluated for the deployed policy, so Eq. (10) is the only safety certificate. Appendix C.2 attempts to justify a variance-based proxy by replacing J^pi(M*) with J^pi_Bayes, which assumes the model posterior is unbiased, and Eq. (10) is not even that bound. The empirical validation in Section 7.1 covers only five environments with random or full-replay datasets that include poor, medium and expert trajectories; it does not cover the suboptimal-data regimes where the proxy is most dangerous. The authors should either derive a conservative proxy with coverage corrections or validate Eq. (10) across dataset qualities, and should state explicitly that the safety guarantee is conditional on data diversity and on the true MDP lying in the posterior support.
  3. [Section 6.1 and Proposition 1] The PIL in Eq. (3) is defined as an expectation under the true dynamics' state-action distribution rho^star_pi induced by the policy of interest. In the practical estimation described just below Eq. (9), the term E(DN,M*) is computed as the empirical MSE on the offline dataset, i.e., under the data-collection distribution. If the behavioral policy does not cover the state-action regions visited by the deployed policy, the estimated PIL can be systematically smaller than the true PIL appearing in Theorem 1, and the tuning constraint E(DN,M*) approx V(DN) does not detect this coverage mismatch. This gap matters because the paper claims in Section 1 that the PIL can be tracked offline to gauge online performance; as it stands, that claim is not supported by the theory unless coverage assumptions are made explicit and justified.
minor comments (5)
  1. [Section 5.3 vs. Section D.2] The Gaussian-world-model identity is called Proposition 1 in the main text but is labelled Proposition 2 in the proof appendix; please renumber for consistency.
  2. [Section 5.2, Figure 2] The caption says 'Normalised Regret Curves for C=1', but the plotted curve resembles the corrected bound sqrt(1 - exp(-d/((1-gamma)N))) rather than the expression in Eq. (5); after fixing the theorem, please make the figure consistent with the corrected formula.
  3. [Section 2.1] The sentence 'Let X be a X subset of R^n-valued random variable' contains a typo; it should read 'Let X be an R^n-valued random variable'.
  4. [Section D.3, Lemma 3] The integrand in the lemma statement is missing a closing parenthesis: it should be '(ell^star - ell(theta)) exp(N ell_N(theta))' rather than '(ell^star - ell(theta) exp(N ell_N(theta)))'.
  5. [Table 3] The brax-walker2d-full-replay row for ReBRAC reports a p-value of 'nan'; please clarify how the correlation analysis handled this case, for example by stating that the metric was constant or the correlation was undefined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regret bounds are derived from external inequalities and standard asymptotic assumptions, and the main practical proxy is an explicitly labelled heuristic validated against held-out deployment rather than a fitted restatement of the target.

full rationale

The theoretical chain is self-contained. Theorem 1 is proved from the total-variation representation of rewards, the Bretagnolle–Huber inequality, and Jensen's inequality; the PIL is not defined in terms of the regret it bounds, so Eq. (4) is a genuine contraction inequality rather than an identity. Theorem 2 follows from the stated local asymptotic normality conditions in Assumption 1 and a standard Laplace-style posterior expansion, resting on external asymptotic references; no constant is fitted to the regret being predicted. The deployment proxy in Eq. (10) is explicitly introduced as a hypothesis ('We hypothesise that the sample median offers a good compromise') and the empirical validation compares it against true regret obtained by actually deploying the policy, so it is held-out evidence. A biased or unvalidated Eq. (10) would be a correctness and safety limitation, not a circular reduction: the dataset-maximum return is not definitionally equal to the optimal return, and the median posterior return is not the true regret. The paper's self-citations (notably to the authors' own Unifloral framework and to normalization references) concern implementation and benchmarking choices, not the load-bearing theoretical derivations, and therefore do not make the argument circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on standard asymptotic statistics assumptions plus a heuristic regret proxy. The free parameters are threshold and normalization choices, not fitted constants in the derived bound. No new physical or ontological entities are introduced; PIL is a new name for a posterior expectation of KL divergence, not a new entity.

free parameters (3)
  • E-vs-V balance threshold = 0.25
    Section 7.1 and Figure 3 shade approximate regret as unreliable when normalized MSE and predictive variance differ by more than 0.25; this hand-chosen threshold determines when SOReL trusts its regret estimate.
  • Rhat_max estimator = maximum return in the offline dataset
    Eq. (10) defines approximate regret as Rhat_max minus median predictive return; which statistic estimates Rhat_max (max vs percentile) is hand-selected and shifts the proxy.
  • Normalization percentiles = 2.5th and 97.5th percentiles
    Section F.5 suggests estimating rmin and rmax from offline data percentiles; the choice affects absolute approximate regret, although the main experiments use known true minimum and maximum returns.
assumptions (5)
  • domain assumption The true MDP has a parameterization in the model class and the prior puts mass near it (Assumption 1i and 1iv)
    Theorem 2's rate relies on well-specification; with misspecification the Bernstein-von Mises and LAN analysis does not apply, as the paper acknowledges in D.3.
  • domain assumption Local asymptotic normality, C2 likelihood, unique local maximizers, and SLLN/CLT for score and Hessian (Assumption 1)
    Invoked in D.3 to prove Theorem 2; standard in asymptotic statistics but unverifiable for neural network world models.
  • ad hoc to paper The predictive median of returns is a reliable proxy for true regret (Eq. 10)
    No theorem supports Eq. (10); Section 6.1 calls it a hypothesis. SOReL's safety claim depends on this proxy.
  • ad hoc to paper When predictive variance and MSE are balanced, the approximate posterior spans the true dynamics
    Algorithm 1 tunes model and inference hyperparameters subject to E(D_N,M*) approximately equal to V(D_N), but no guarantee links this balance to coverage of the true MDP.
  • domain assumption The offline dataset has diverse coverage across poor, medium, and expert regions of performance
    Section 7.1 and F.1 state that without a model prior the offline data must be diverse; this is not generally available in standard offline RL benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fully Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/MVUATXYV

@misc{pith2026250522442,
  author       = {Pith},
  title        = {Pith review of: Fully Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVUATXYV}},
  note         = {Machine review of arXiv:2505.22442}
}
read the original abstract

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Figures

Figures reproduced from arXiv: 2505.22442 by the authors.

Figure 1
Figure 1. Existing model-based offline approaches rely on online interactions for hyperparameter tuning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Normalised Regret Curves for C = 1 Theorem 2 applies to the Gaussian world model introduced in Section 5.3 with neural network mean functions with C 2 -continuous activations (tanh, identity, sigmoid, softplus, SiLU, SELU, GELU...) using a Gaussian or uniform prior truncated to a compact parameter space and similarly well-behaved parametric models. The O (d/N) information rate coincides with the op￾timal ‘minimax’ c… view at source ↗
Figure 3
Figure 3. A simplified version of SOReL applied to brax-halfcheetah-full-replay to identify when the policy [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: TOReL-selected hyperparameter regret versus mean hyperparameter regret (lower is better). [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: TOReL compared to UCB bandit-based online hyperparameter selection. The x-axis shows the [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: SOReL BAMDP hyperparameter sweeps (tuning set [PITH_FULL_IMAGE:figures/full_fig_p035_6.png]
Figure 7
Figure 7. Figure 7: Simplified version of SOReL on brax-halfcheetah-full-replay. The plot showing only the ensemble [PITH_FULL_IMAGE:figures/full_fig_p035_7.png]
Figure 8
Figure 8. Figure 8: Simplified version of SOReL applied to various tasks. For the gymnax environments, [PITH_FULL_IMAGE:figures/full_fig_p036_8.png]
Figure 9
Figure 9. Figure 9: Simplified version of SOReL applied to various tasks. Shaded purple shows the approximate regret [PITH_FULL_IMAGE:figures/full_fig_p037_9.png]
Figure 10
Figure 10. Figure 10: Scatter plots to visualise the positive correlation between the TOReL regret metric and the true regret. [PITH_FULL_IMAGE:figures/full_fig_p039_10.png]
Figure 11
Figure 11. Figure 11: Training curves while collecting the brax full-replay offline datasets. We ensure that the agent spend [PITH_FULL_IMAGE:figures/full_fig_p040_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

    cs.AI 2026-08 conditional novelty 5.0 of 10

    For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.

Reference graph

Works this paper leans on

83 extracted references · 65 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aitchison

    J. Aitchison. Goodness of prediction fit.Biometrika, 62(3):547–554, 1975. ISSN 00063444, 14643510. URLhttp://www.jstor.org/stable/2335509. 5.1

    work page arXiv 1975

  2. [2]

    Alaa and Mihaela van der Schaar

    Ahmed M. Alaa and Mihaela van der Schaar. Bayesian nonparametric causal inference: Information rates and learning algorithms.IEEE Journal of Selected Topics in Signal Processing, 12(5):1031–1046, 2018. doi: 10.1109/JSTSP.2018.2848230. 5.1

    work page arXiv 2018

  3. [3]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In M. Ranzato, A. Beygelz- imer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Infor- mation Processing Systems, volume 34, pages 7436–7447. Curran Associates, Inc., 2021. URL https://proceed...

    2021

  4. [4]

    Asymptotically minimax bayes predictive densities.The Annals of Statistics, 34(6):2921–2938, 2006

    Mihaela Aslan. Asymptotically minimax bayes predictive densities.The Annals of Statistics, 34(6):2921–2938, 2006. ISSN 00905364. URL http://www.jstor.org/stable/ 25463538. 5.1, D.3

    2006

  5. [5]

    Augmented world models facilitate zero-shot dynamics generalization from a single offline environment

    Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 619–629. PMLR, 18–...

    2021

  6. [6]

    Information-theoretic characterization of bayes performance and the choice of priors in parametric and nonparametric problems

    Andrew R Barron. Information-theoretic characterization of bayes performance and the choice of priors in parametric and nonparametric problems. InBayesian Statistics 6: Proceedings of the Sixth Valencia International Meeting June 6-10, 1998. Oxford University Press, 08

    work page 1998

  7. [7]

    Barron.The Exponential Convergence of Posterior Probabilities with Implications for Bayes Estimators of Density Functions

    A.R. Barron.The Exponential Convergence of Posterior Probabilities with Implications for Bayes Estimators of Density Functions. Department of Statistics, University of Illinois, 1988. URLhttps://books.google.co.uk/books?id=8raEnQAACAAJ. 5.1, D.3

    work page 1988

  8. [8]

    Bass.Real Analysis for Graduate Students, chapter 21

    R.F. Bass.Real Analysis for Graduate Students, chapter 21. Createspace Ind Pub,

    work page

Show all 83 references
  1. [9]

    A survey of meta-reinforcement learning, 2024

    Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning, 2024. URL https://arxiv. org/abs/2301.08028. 4.2

    2024 arXiv

  2. [10]

    A problem in the sequential design of experiments.Sankhy ¯a: The Indian Journal of Statistics (1933-1960), 16(3/4):221–229, 1956

    Richard Bellman. A problem in the sequential design of experiments.Sankhy ¯a: The Indian Journal of Statistics (1933-1960), 16(3/4):221–229, 1956. ISSN 00364452. URL http: //www.jstor.org/stable/25048278. 2.2

    1933

  3. [11]

    Dynamic programming and stochastic control processes.Information and Control, 1(3):228–239, 1958

    Richard Bellman. Dynamic programming and stochastic control processes.Information and Control, 1(3):228–239, 1958. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019- 9958(58)80003-0. URL https://www.sciencedirect.com/science/article/ pii/S0019995858800030. 2.2

    1958 doi

  4. [12]

    Foster, and Daniel M

    Blair Bilodeau, Dylan J. Foster, and Daniel M. Roy. Minimax rates for conditional density estimation via empirical entropy.The Annals of Statistics, 2021. URL https://api. semanticscholar.org/CorpusID:237592759. 5.1, 5.2

    2021

  5. [13]

    On the foundations of statistical inference.Journal of the American Statistical Association, 57(298):269–306, 1962

    Allan Birnbaum. On the foundations of statistical inference.Journal of the American Statistical Association, 57(298):269–306, 1962. doi: 10.1080/01621459.1962.10480660. URL https: //www.tandfonline.com/doi/abs/10.1080/01621459.1962.10480660. 6

    1962

  6. [14]

    J. L. Bretagnolle and Catherine Huber. Estimation des densités: risque minimax.Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 47:119–137, 1978. URL https: //api.semanticscholar.org/CorpusID:122597694. D.1, 1

    1978

  7. [15]

    Bayes adaptive monte carlo tree search for of- fline model-based reinforcement learning, 2024

    Jiayu Chen, Wentse Chen, and Jeff Schneider. Bayes adaptive monte carlo tree search for of- fline model-based reinforcement learning, 2024. URL https://arxiv.org/abs/2410. 11234. 3

    2024

  8. [16]

    Conser- vative uncertainty estimation by fitting prior networks

    Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conser- vative uncertainty estimation by fitting prior networks. InEighth International Conference on Learning Representations, 04 2020. 4.2, 7.1, C.1

    2020

  9. [17]

    Clarke and A.R

    B.S. Clarke and A.R. Barron. Information-theoretic asymptotics of bayes methods.IEEE transactions on information theory, 36(3):453–471, 1990. ISSN 0018-9448. 5.1, D.3

    1990

  10. [18]

    J. L. Doob. Application of the theory of martingales. InLe calcul des probabilités et ses appli- cations [The calculus of probabilities and its applications], number 13 in CNRS International Colloquia, pages 23–27. Centre National de la Recherche Scientifique, Paris, 1949. (Ly...

    1949

  11. [19]

    Observation of a markov process through a noisy channel.PhD Thesis, 1962

    Alvin Drake. Observation of a markov process through a noisy channel.PhD Thesis, 1962. B

    1962

  12. [20]

    Fast reinforcement learning via slow reinforcement learning

    Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Peter Abbeel. Fast reinforcement learning via slow reinforcement learning. 1987. 4.2

    1987

  13. [21]

    PhD thesis, 2002

    Michael O’Gordon Duff.Optimal Learning: Computational Procedures for Bayes-Adaptive Markov Decision Processes. PhD thesis, 2002. AAI3039353. 1, 4.1, 4.2, 4.2, B

    2002

  14. [22]

    Bayesian exploration networks

    Mattie Fellows, Brandon Kaplowitz, Christian Schroeder de Witt, and Shimon Whiteson. Bayesian exploration networks. InICML, 2024. 4.2

    2024

  15. [23]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020. 7.2

    2020

  16. [24]

    A minimalist approach to offline reinforcement learn- ing

    Scott Fujimoto and Shixiang (Shane) Gu. A minimalist approach to offline reinforcement learn- ing. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20132–20145. Curran A...

    2021

  17. [25]

    A new proof of the likelihood principle.The British Journal for the Philosophy of Science, 66(3):475–503, 2015

    Greg Gandenberger. A new proof of the likelihood principle.The British Journal for the Philosophy of Science, 66(3):475–503, 2015. doi: 10.1093/bjps/axt039. URL https://doi. org/10.1093/bjps/axt039. 6 13

    2015 doi

  18. [26]

    Efficient bayes-adaptive reinforcement learn- ing using sample-based search

    Arthur Guez, David Silver, and Peter Dayan. Efficient bayes-adaptive reinforcement learn- ing using sample-based search. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger, editors,Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates...

    2012

  19. [27]

    Scalable and efficient bayes-adaptive reinforcement learning based on monte-carlo tree search.Journal of Artificial Intelligence Research, 48:841– 883, 10 2013

    Arthur Guez, David Silver, and Peter Dayan. Scalable and efficient bayes-adaptive reinforcement learning based on monte-carlo tree search.Journal of Artificial Intelligence Research, 48:841– 883, 10 2013. doi: 10.1613/jair.4117. 4.2

    2013 doi

  20. [28]

    Bayes-adaptive simulation-based search with value function approximation

    Arthur Guez, Nicolas Heess, David Silver, and Peter Dayan. Bayes-adaptive simulation-based search with value function approximation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, volume 27....

    2014

  21. [29]

    J. A. Hartigan. The maximum likelihood prior.The Annals of statistics, 26(6):2083–2103, 1998. ISSN 0090-5364. 5.1, D.3

    1998

  22. [30]

    A clean slate for offline reinforcement learning.arXiv preprint arXiv:2504.11453,

    Matthew Thomas Jackson, Uljad Berdica, Jarek Liesen, Shimon Whiteson, and Jakob Nicolaus Foerster. A clean slate for offline reinforcement learning.arXiv preprint arXiv:2504.11453,

    arXiv

  23. [31]

    Relu to the rescue: Improve your on-policy actor-critic with positive advantages

    A Jesson, C Lu, N Beltran-Velez, A Filos, J Foerster, and Y Gal. Relu to the rescue: Improve your on-policy actor-critic with positive advantages. 2024. 8

    2024

  24. [32]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains.Artif. Intell., 101(1–2):99–134, may 1998. ISSN 0004-3702. B

    1998

  25. [33]

    The validity of posterior expansions based on laplace’s method.Bayesian and Likelihood Methods in Statistics and Economics, pages 473–488, 1990

    Robert E Kass, Luke Thierney, and Joeseph B Kadane. The validity of posterior expansions based on laplace’s method.Bayesian and Likelihood Methods in Statistics and Economics, pages 473–488, 1990. URL https://www.stat.cmu.edu/~kass/papers/validity. pdf. D.3

    1990

  26. [34]

    Morel: Model-based offline reinforcement learning

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 21...

    2020

  27. [35]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014. F.2

    2014 arXiv

  28. [36]

    Kleijn and A.W

    B.J.K. Kleijn and A.W. van der Vaart. The Bernstein-V on-Mises theorem under misspecification. Electronic Journal of Statistics, 6(none):354 – 381, 2012. doi: 10.1214/12-EJS675. URL https://doi.org/10.1214/12-EJS675. D.3

    2012 doi

  29. [37]

    On asymptotic properties of predictive distributions.Biometrika, 83(2):299–313, 06 1996

    FUMIYASU KOMAKI. On asymptotic properties of predictive distributions.Biometrika, 83(2):299–313, 06 1996. ISSN 0006-3444. doi: 10.1093/biomet/83.2.299. URL https: //doi.org/10.1093/biomet/83.2.299. 5.1, D.3

    1996 doi

  30. [38]

    Offline reinforcement learning with implicit q- learning.CoRR, abs/2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning.CoRR, abs/2110.06169, 2021. URL https://arxiv.org/abs/2110.06169. 1, 1, 3, 7.2

    2021 arXiv

  31. [39]

    Danie G. Krige. A statistical approach to some basic mine valuation problems on the witwater- sand.Journal of the Chemical, Metallurgical and Mining Society of South Africa, 52:119–139,

  32. [40]

    Conserva- tive q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conserva- tive q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 1179–1191. ...

    2020

  33. [41]

    Springer Berlin Heidelberg, Berlin, Heidelberg, 2012

    Sascha Lange, Thomas Gabel, and Martin Riedmiller.Batch Reinforcement Learning, pages 45–73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27645-3. doi: 10.1007/978-3-642-27645-3_2. URL https://doi.org/10.1007/978-3-642- 27645-3_2. 1, 2.2

    2012 doi

  34. [42]

    On some asymptotic properties of maximum likelihood estimates and related bayes’ estimates

    Lucien Le Cam. On some asymptotic properties of maximum likelihood estimates and related bayes’ estimates. volume 1, pages 277–300, 1953. 5.1, D.3

    1953

  35. [43]

    Efficient backprop

    Yann LeCun, Leon Bottou, Genevieve B Orr, and Klaus-Robert Mueller. Efficient backprop. In Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998. F.2

    1998

  36. [44]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL https://arxiv.org/ abs/2005.01643. 1, 2.2

    2020 arXiv

  37. [45]

    D. V . Lindley. Approximate bayesian methods.Trabajos de Estadistica Y de Investigacion Operativa, 31(1):223–245, 1980. D.3

    1980

  38. [46]

    Discovered policy optimisation.Advances in Neural Information Processing Systems, 35:16455–16468, 2022

    Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation.Advances in Neural Information Processing Systems, 35:16455–16468, 2022. F.3

    2022

  39. [47]

    Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen J. Roberts. Re- visiting design choices in offline model based reinforcement learning. InInternational Con- ference on Learning Representations, 2022. URL https://openreview.net/forum? id=zz9hXVhf40. 1, 3

    2022

  40. [48]

    J. J. Martin.Bayesian decision problems and Markov chains [by] J. J. Martin. Wiley New York,

  41. [49]

    Reinforcement learning: An overview, 2024

    Kevin Murphy. Reinforcement learning: An overview, 2024. URLhttps://arxiv.org/ abs/2412.05265. 1, 2.2

    2024

  42. [50]

    Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pag...

  43. [51]

    Randomized prior functions for deep reinforcement learning

    Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems 31, pages 8617–

  44. [52]

    Hyperparameter selection for offline reinforcement learning.CoRR, abs/2007.09055, 2020

    Tom Le Paine, Cosmin Paduraru, Andrea Michi, Çaglar Gülçehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning.CoRR, abs/2007.09055, 2020. URL https://arxiv.org/abs/2007.09055. 3

    2007 arXiv

  45. [53]

    K. B. Petersen and M. S. Pedersen. The matrix cookbook, nov 2012. URL http: //localhost/pubdb/p.php?3274. Version 20121115. 3

    2012

  46. [54]

    Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming

    Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779. 2.2 15

    1994

  47. [55]

    Carl Edward Rasmussen and Christopher K. I. Williams.Gaussian Processes for Machine Learning. The MIT Press, 2006. URL https://gaussianprocess.org/gpml/. 4.1, B

    2006

  48. [56]

    Roberts and Jeffrey S

    Gareth O. Roberts and Jeffrey S. Rosenthal. General state space Markov chains and MCMC algorithms.Probability Surveys, 1(none):20 – 71, 2004. doi: 10.1214/154957804100000024. URLhttps://doi.org/10.1214/154957804100000024. D.3

    2004 doi

  49. [57]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. 3, 4.2, 7.1

    2017 arXiv

  50. [58]

    The edge-of-reach problem in offline model-based reinforcement learning

    Anya Sims, Cong Lu, Jakob Nicolaus Foerster, and Yee Whye Teh. The edge-of-reach problem in offline model-based reinforcement learning. InThe Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https://openreview.net/forum? id=3dn1hINA6o. 1, 3

    2024

  51. [59]

    Smallwood and Edward J

    Richard D. Smallwood and Edward J. Sondik. The optimal control of partially observable markov processes over a finite horizon.Operations Research, 21(5):1071–1088, 1973. ISSN 0030364X, 15265463. URLhttp://www.jstor.org/stable/168926. B

    1973

  52. [60]

    A strong baseline for batch imitation learning, 2023

    Matthew Smith, Lucas Maystre, Zhenwen Dai, and Kamil Ciosek. A strong baseline for batch imitation learning, 2023. URLhttps://arxiv.org/abs/2302.02788. 3

    2023 arXiv

  53. [61]

    Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Scholkopf, and Gert R

    Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Scholkopf, and Gert R. G. Lanckriet. On integral probability metrics, ϕ-divergences and binary classification.arXiv: Information Theory, 2009. URL https://api.semanticscholar.org/CorpusID: 14114329. D.1

    2009

  54. [62]

    Model-Bellman inconsistency for model-based offline reinforcement learning

    Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-Bellman inconsistency for model-based offline reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceeding...

    2023

  55. [63]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the- book-2nd.html. 2.2

    2018

  56. [64]

    Algorithms for Reinforcement Learning.Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1):1–103, 2010

    Csaba Szepesvári. Algorithms for Reinforcement Learning.Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1):1–103, 2010. ISSN 1939-4608. doi: 10.2200/ S00268ED1V01Y201005AIM009. URL http://www.morganclaypool.com/doi/ abs/10.2200/S00268ED1V01Y201005AIM009. 2.2

    2010 doi

  57. [65]

    Revisiting the minimalist approach to offline reinforcement learning

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id= vqGWslLeEw. 1, 1, 3, 7.2

    2023

  58. [66]

    Luke Tierney and Joseph B. Kadane. Accurate approximations for posterior moments and marginal densities.Journal of the American Statistical Association, 81(393):82–86, 1986. ISSN 0162-1459. D.3

    1986

  59. [67]

    Kass, and Joseph B

    Luke Tierney, Robert E. Kass, and Joseph B. Kadane. Fully exponential laplace approximations to expectations and variances of nonpositive functions.Journal of the American Statistical Association, 84(407):710–716, 1989. ISSN 0162-1459. D.3

    1989

  60. [68]

    A. W. van der Vaart.Bayes Procedures, page 138–152. Cambridge Series in Statistical and Proba- bilistic Mathematics. Cambridge University Press, 1998. doi: 10.1017/CBO9780511802256.011. 5.1, D.3

    1998 doi

  61. [69]

    Information rates of nonparametric gaussian process methods.J

    Aad van der Vaart and Harry van Zanten. Information rates of nonparametric gaussian process methods.J. Mach. Learn. Res., 12(null):2095–2119, July 2011. ISSN 1532-4435. 5.1, 5.2 16

    2011

  62. [70]

    No more pesky hyperparameters: Offline hyperparameter tuning for RL.Transactions on Machine Learning Research, 2022

    Han Wang, Archit Sakhadeo, Adam M White, James M Bell, Vincent Liu, Xutong Zhao, Puer Liu, Tadashi Kozuno, Alona Fyshe, and Martha White. No more pesky hyperparameters: Offline hyperparameter tuning for RL.Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URLhtt...

    2022

  63. [71]

    Foster, and Sham M

    Ruosong Wang, Dean P. Foster, and Sham M. Kakade. What are the statistical limits of offline rl with linear function approximation?, 2020. URL https://arxiv.org/abs/2010. 11895. 3

    2020

  64. [72]

    Differential-space.Journal of Mathematics and Physics, 2(1-4):131–174, 1923

    Norbert Wiener. Differential-space.Journal of Mathematics and Physics, 2(1-4):131–174, 1923. doi: https://doi.org/10.1002/sapm192321131. URL https://onlinelibrary.wiley. com/doi/abs/10.1002/sapm192321131. 4.1, B

    1923 doi

  65. [73]

    Information-theoretic determination of minimax rates of convergence.The Annals of Statistics, 27(5):1564 – 1599, 1999

    Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence.The Annals of Statistics, 27(5):1564 – 1599, 1999. doi: 10.1214/aos/1017939142. URLhttps://doi.org/10.1214/aos/1017939142. 5.1, 5.2

    1999

  66. [74]

    Mopo: Model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Informa- tion Processing S...

    2020

  67. [75]

    Combo: Conservative offline model-based policy optimization

    Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In M. Ranzato, A. Beygelz- imer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Infor- matio...

    2021

  68. [76]

    On the importance of hyperparameter optimization for model-based reinforcement learning

    Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of ...

    2021

  69. [77]

    Varibad: A very good method for bayes-adaptive deep rl via meta- learning

    Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. InInternational Conference on Learning Representations, 2020. URL https: //openreview...

    2020

  70. [78]

    N−1X i=0 1 N D 2 log(2π) + 1 2 D−1X d=0 logσ 2 θd (xi) + (yid −µ θd (xi))2 σ2 θd (xi) !!# , =Ei∼UN

    Åström, Karl Johan. Optimal Control of Markov Processes with Incomplete State Information I. 10:174–205, 1965. ISSN 0022-247X. doi: {10.1016/0022-247X(65)90154-X}. URL https://lup.lub.lu.se/search/files/5323668/8867085.pdf. B 17 A Broader Impact This paper presents work whose ...

    1965

  71. [83]

    Eθ∼PΘ(DN )

    Noting that B(θ ⋆ j , r 2 )⊂Θ\ ¯Θ and 0≤exp (N ℓN (θ)), this allows us to lower bound the integral: Z Θ\ ¯Θ exp (N ℓN (θ))p(θ)dθ≥ Z B(θ ⋆ j , r 2 ) exp (N ℓN (θ))p(θ)dθ, ≥exp Nmin θ′′∈B(θ ⋆ j , r 2 ) ℓN (θ′′) ! Z B(θ ⋆ j , r 2 ) p(θ)dθ, = exp Nmin θ′′∈B(θ ⋆ j , r 2 ) ℓN (θ′′) ...

  72. [1999]

    doi: 10.1093/oso/9780198504856.003.0002

    ISBN 9780198504856. doi: 10.1093/oso/9780198504856.003.0002. URL https: //doi.org/10.1093/oso/9780198504856.003.0002. 5.1, D.3

  73. [2013]

    URL https://books.google.co.uk/books?id= s6mVlgEACAAJ

    ISBN 9781481869140. URL https://books.google.co.uk/books?id= s6mVlgEACAAJ. D.3, 2 12

  74. [2025]

    1, 1, 2.2, 3, 7.2, 7.2, F.2, F.4

  75. [8629]

    URL http://papers.nips.cc/paper/8080- randomized-prior-functions-for-deep-reinforcement-learning.pdf

    Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/8080- randomized-prior-functions-for-deep-reinforcement-learning.pdf . 4.2, 7.1, C.1

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.